Jahed Ahmed

Long-Term Open Source Maintainance

At some point on GitHub, I switched off "automatically watch repos that you have push access to". This was back when I was in a team that had hundreds of repos and I didn't want to auto-watch all of them. Little did I know that also included my personal repos... so I wasn't getting notifications on any of my new ones.

Back in August, someone asked me to publish a new version of node-terraform to coincide with a new version of Terraform. But since I wasn't watching the repo, it went completely under my radar. I barely use Terraform personally, other than for provisioning some of my personal infra -- which barely changes -- so I had no idea.

Automate Robotic Work

For some background, node-terraform is a NodeJS wrapper I made to automatically download and link up separate versions of Terraform per-project. Each wrapper version matches a version of Terraform, so when a new Terraform version comes out, I need to release a new version of the wrapper. This way, the Terraform version is maintained through NodeJS's package manager with every other dependency. Why NodeJS? Most of my projects use it so it integrates nicely with my workflow.

After catching up with new versions, I decided that, if I was going to maintain the project long-term, I needed to automate the process sooner rather than later. I originally planned to use CircleCI, but with GitHub Actions, a continuous integration (CI) service built into GitHub, I could avoid introducing another third-party dependency.

Open Source and Infinite CIs

I really dislike how there's so many CI services, each with their own way of doing things. It means if you want to contribute beyond a certain amount to a project, chances are you'll need to figure out a new CI service, how it structures jobs, provisions infra, and understand its dreadful YAML configuration. None of it is meaningfully different. They all do the same stuff. Just slightly differently.

GitHub Actions initially tried something a bit more fresh, with HashiCorp Configuration Language (HCL), the same language Terraform uses. A new CI that tries to remove YAML is more than welcome in my eyes. However, they dropped HCL and moved to YAML recently so... yeah. More of the same. Actually worse, because most of the existing articles on GitHub Actions are just useless now. The service is still in beta so I'm not too fussed.

I really hope we get some sort of "OpenCI" initiative so we can end this cycle.

Conclusion

I've now got automated processes for:

I don't know if the scheduled checks work as they haven't picked up any changes yet. Only time will tell.

Automating the dependency check seems a bit risky as I'm not manually auditing the third-party changes being pulled in before they're pushed. However, GitHub and NPM provide security notifications so that should be covered retroactively.

It's pretty much impossible for me to check all the external code every time they publish a new package. It's mainly down to minimising dependencies as much as possible, and making sure the ones that are needed are alive and well.

At the end of all this, I now have a pretty generic set of actions which I can copy over to my other projects to make maintenance less tedious.

Thanks for reading.

Enforcing Retention Policies on AWS S3

With the recent rush for GDPR compliance, services are becoming more aware of the amount of data they hold and if it's really necessary to have all of it.

Application logs contain a variety of historical data coming from both users and third-parties, making it extremely useful when running reports and to monitor production behaviour. However, after a certain period, the burden of responsibility will begin to outweigh the usefulness of the data. Once that point is reached, it's best to shed that responsibility.

A common way to store logs is to put them on AWS S3. But, without the proper configuration, those logs will remain there indefinitely. You could manually delete objects or set an expiry when they're uploaded but there's an even more convenient solution built into S3: Lifecycle Rules.

At Unruly we use Terraform to provision our AWS resources. So, I'll be showing how you can do the same to enforce your retention policies. Before continuing, you'll need to familiarise yourself with Terraform's basics.

Read more