Enforcing Retention Policies on AWS S3
With the recent rush for GDPR compliance, services are becoming more aware of the amount of data they hold and if it's really necessary to have all of it.
Application logs contain a variety of historical data coming from both users and third-parties, making it extremely useful when running reports and to monitor production behaviour. However, after a certain period, the burden of responsibility will begin to outweigh the usefulness of the data. Once that point is reached, it's best to shed that responsibility.
A common way to store logs is to put them on AWS S3. But, without the proper configuration, those logs will remain there indefinitely. You could manually delete objects or set an expiry when they're uploaded but there's an even more convenient solution built into S3: Lifecycle Rules.
At Unruly we use Terraform to provision our AWS resources. So, I'll be showing how you can do the same to enforce your retention policies. Before continuing, you'll need to familiarise yourself with Terraform's basics.