Jahed Ahmed

Weekly Report: 7th October 2019

FrontierNav Terminology

Over the years FrontierNav's data model has changed a lot. This meant various parts of the codebase and web app used a range of words to describe the same thing. As I start introducing more terminology to the public, there's really no room for confusion. So I went through the project, migrated all the data and reduced the number of terms.

As an example, most data models have terms where these mean the same thing:

FrontierNav probably used at least 4 of these terms in the same contexts but now it only uses two: Entity and Relationships.

FrontierNav Performance

It's really hard for me to tell how well FrontierNav performs on lower-end devices as I don't own one. So as a general rule, after finishing a substantial feature, I'll put in some time to improve its performance or at least look into it.

This week, I optimised the Data Tables which uses artificial viewport scrolling to avoid rendering thousands of rows at once. Any lag in rendering is easily noticable while scrolling, even on high-end devices so it's important to optimise it as much as possible. In the React world, that usually means caching function calls (memoizing) and reducing re-renders in general. There's not much else to say about it. React's DevTools are good enough as they provide rendering times and let you know what triggered it to understand where the costs comes from.

Other Stuff

Thanks for reading.

Long-Term Open Source Maintainance

At some point on GitHub, I switched off "automatically watch repos that you have push access to". This was back when I was in a team that had hundreds of repos and I didn't want to auto-watch all of them. Little did I know that also included my personal repos... so I wasn't getting notifications on any of my new ones.

Back in August, someone asked me to publish a new version of node-terraform to coincide with a new version of Terraform. But since I wasn't watching the repo, it went completely under my radar. I barely use Terraform personally, other than for provisioning some of my personal infra -- which barely changes -- so I had no idea.

Automate Robotic Work

For some background, node-terraform is a NodeJS wrapper I made to automatically download and link up separate versions of Terraform per-project. Each wrapper version matches a version of Terraform, so when a new Terraform version comes out, I need to release a new version of the wrapper. This way, the Terraform version is maintained through NodeJS's package manager with every other dependency. Why NodeJS? Most of my projects use it so it integrates nicely with my workflow.

After catching up with new versions, I decided that, if I was going to maintain the project long-term, I needed to automate the process sooner rather than later. I originally planned to use CircleCI, but with GitHub Actions, a continuous integration (CI) service built into GitHub, I could avoid introducing another third-party dependency.

Open Source and Infinite CIs

I really dislike how there's so many CI services, each with their own way of doing things. It means if you want to contribute beyond a certain amount to a project, chances are you'll need to figure out a new CI service, how it structures jobs, provisions infra, and understand its dreadful YAML configuration. None of it is meaningfully different. They all do the same stuff. Just slightly differently.

GitHub Actions initially tried something a bit more fresh, with HashiCorp Configuration Language (HCL), the same language Terraform uses. A new CI that tries to remove YAML is more than welcome in my eyes. However, they dropped HCL and moved to YAML recently so... yeah. More of the same. Actually worse, because most of the existing articles on GitHub Actions are just useless now. The service is still in beta so I'm not too fussed.

I really hope we get some sort of "OpenCI" initiative so we can end this cycle.

Conclusion

I've now got automated processes for:

I don't know if the scheduled checks work as they haven't picked up any changes yet. Only time will tell.

Automating the dependency check seems a bit risky as I'm not manually auditing the third-party changes being pulled in before they're pushed. However, GitHub and NPM provide security notifications so that should be covered retroactively.

It's pretty much impossible for me to check all the external code every time they publish a new package. It's mainly down to minimising dependencies as much as possible, and making sure the ones that are needed are alive and well.

At the end of all this, I now have a pretty generic set of actions which I can copy over to my other projects to make maintenance less tedious.

Thanks for reading.

FrontierNav Report: September 2019

Progress Report

Changes in September

I missed updates over August so I'll try filling in the blanks for this report.

Guides for Astral Chain

On 30th August, Astral Chain released for the Nintendo Switch. So I spent the first few days after its release playing it to completion. After that I started working on an interactive guide.

Implementing the in-game "File Select" menu for the Web was a lot of fun. Getting it to flex and fit various screen sizes was the most challenging part. Game consoles have the benefit of rendering at a fixed resolution, the Web doesn't really have that luxury (unless if you're willing to compromise user experience).

I don't use the vh unit much in CSS, but it was perfect for this use case to scale up the size on larger displays. Dealing with the various behaviours of nested flexboxes was probably the least enjoyable part, but that's more to do with the tedium of setting up the necessary constraints rather than issues with CSS Flexbox itself.

Interactive Guides for Astral Chain

Adding Features as Needed

While I was adding data for Astral Chain, I made the effort to use the editing features built into FrontierNav rather than writing single-use scripts. If anything I needed was missing or too tedious I made the effort to improve that workflow so that future edits can benefit from it.

This is probably the best way to work on FrontierNav going forward as it provides a clear acceptance criteria for each feature. i.e. did it actually solve the problem I was having and did it speed up data entry.

Entity Dropdown
Entity Dropdown

Real-time Features

I can probably start adding more dynamic features into FrontierNav now. Things like real-time collaboration, user-suggested games, image uploads etc. The only thing stopping me is server costs. The current income from Patreon won't be enough to cover it. However, it's a bit of a "chicken or the egg" situation. If I don't add these features, users will have less incentive to contribute in the first place. So I'll try adding them and hopefully it gets people to help cover the costs.

The minimum cost for most of this is around $50 per month, so I've set a new goal on the Patreon page. Please consider contributing if you haven't already on Patreon.

Thanks for reading.

FrontierNav Report: July 2019

Progress Report

Changes in July

Now that I'm spending more time on FrontierNav, I did a lot in July so looking back and writing it all up is draining.

I'm also writing the weekly reports making it difficult for me to filter out stuff I've said in the previous month's report and stuff I've said in the previous weekly reports.

So, I may decide to stop writing monthly reports in the future. Instead, I'll link to the weekly reports and related articles as a summary.

Let me know what you think.

User Interface

The user interface for FrontierNav has been overhauled. Apologies in advance if the changelist lacks context. The main thing to takeaway is that a lot has changed. To give things more context I'll have to go through the old user interface and take before-and-after screenshots which is a lot of work.

The aim of the overhaul is to provider faster and consistent navigation and in the future support multiple visualisations on the same page.

New Navigation and Universal Sidebar
New Navigation and Universal Sidebar

Universal Search

The universal search is now using a "fuzzy" search algorithm again. This means text doesn't need to exactly match the thing you're looking for and results appear faster.

Xenoblade Chronicles X

Treasure Map
Treasure Map

FrontierNav Editor

The editor is now in a working state. It is not "feature complete" as the number of possible features is endless. The main focus now will be to use it as the sole way to add data to FrontierNav, adding any new features to it when needed.

The list of changes here will be too long so I've recorded some simple examples below. Try it out yourself if you want.

Editing an entity
Editing an entity.

Editing a relationship
Editing a relationship.

Next Up

I'm going to use the Editor now to add more data into FrontierNav for Xenoblade X, Xenoblade 2 and some of the emptier games. The main goal being to add more immediately obvious functionality to it.

Thanks for reading.

Weekly Report: 22nd July 2019

Desktop UIs

This week I've been doing a lot of UI experiments. A lot of it was fueled by yet another poor desktop experience being introduced by a major web company. This time it was Twitter. Luckily they have Tweetdeck which provides a much better desktop interface anyway.

FrontierNav

FrontierNav Editor Initial Release

The FrontierNav editor is pretty much complete. Of course, there's a lot of features and improvements I'll be adding a long the way as with any product. But in terms of data-entry, the foundations have all been laid.

The biggest issue now I guess is figuring out the best ways for users to apply their changes. FrontierNav isn't active enough to allow any public change to get applied without moderation. So chances are, I'll go for a request-based approach. Users can make changes and send them over for approval when they're done using the import/export feature.

As a first step, I'll be using the editor to fill in the missing data for Xenoblade 2 and X. Adding functionality to the editor as I need it for others to use in the future. That's a much better approach than hacking scripts together and throwing them away.

Automated Change Submissions

I'll probably remove the need to import/export changes in the near future. If changes are stored on Firebase, there's no need for it outside of offline usage. However there's a risk of flooding the database so it'll need to be restricted to trusted "Editors" or rate-limited.

Video Games

I finished "Donkey Kong Country: Tropical Freeze" on the Switch finally. It's a good game. Astounding music and level design. Some flaws. Very different from Mario's more nimble and acrobatic platforming. Donkey Kong's a lot more weighted and slow; as you'd expect from a gorilla.

I'll probably pick up "Starlink: Battle for Atlas" from my backlog next. Not expecting much from it.

Thanks for reading

Weekly Report: 15th July 2019

FrontierNav

Agility

Misc.

So Many Security Updates

It seems every week there's a new vulnerability found in an npm package. While this is great progress for the community, it's a pain to deal with when maintaining multiple projects. Automation is a solution, but then you need to maintain the automation. Removing dependencies and writing your own just means you'll probably have vulnerabilities that you don't know about since no one's looking. Eventually dependencies rot and security issues take over. At that point you need to abandon ship and find a new one.

It's an unsolvable problem as far as I can tell.

FrontierNav Static Generation

FrontierNav is a JavaScript heavy application. So much so that loading it up without JavaScript will just show a warning. This is fine, since without JavaScript, FrontierNav becomes a bare bones wiki and there are plenty of better wikis around.

However, it does cause issues with search engines, social network sharing and others where they expect relevant information in the HTML on request. Google's search engine does use JavaScript, but others like Bing don't. Share a link to FrontierNav on Twitter and you'll just see a generic embed. This is a problem as it means FrontierNav isn't interacting properly with the rest of the web.

The proper solution to this is "Server-Side Rendering" (SSR). I've tried SSR before, but it doesn't work. A lot of the code wasn't made for it in mind so it just fails. To fix it would mean going through a lot of code and also making sure any future code works in both cases. It's a big investment.

An alternative I've been thinking of is to go through every page in a web browser and dump the HTML. I've tried it this week using Puppeteer to automate the process and... it works.

There are a few issues though: cache invalidation (as always) and deciding where to run it. It can't run server-side since it's using a whole web browser which is slow and eats up a lot of resources, so it'll need to run in parallel. Which in turn means it'll need to know which pages to cache.

Right now, the experiment is using the sitemap.xml to scrape FrontierNav. That works for the more simple pages. But pages like the Maps have potentially hundreds of elements, and dumping all of that into HTML will be ridiculous. These pages are the most likely to be shared on social media. I could strip the excess post-render, but then the solution becomes more complex, at which point SSR becomes more appealing again.

Running in parallel is also a lot of waste since the vast majority of pages won't be directly navigated to. I could make it more intelligent and have it use access logs to find the pages it should cache, but then it's caching after the fact so it'll always be one step behind.

Decisions. Decisions.

FrontierNav Search Performance

I've become more comfortable using Web Workers now. My first implementation of it for the Universal Search had a single global Worker searching across multiple indexes. Now, there's one for each index, created only when Search is activated and discarded after. I really can't tell what difference it's made since there aren't any Dev Tools that show individual performance metrics, but from an implementation point of view, it makes a lot more sense. There's no point having Workers waiting around, hogging resources.

Agility Bug Fixes and Maintenance

Agility, the mobbing timer I made a while back, had a bug report. Since it's just a single web page, I recently stripped it down from a Middleman static generated site, to a simple static site with no build step. It's a lot easier to maintain now. I'm still a bit reluctant to be maintaining it though since it's not something I use anymore and I'm not getting anything out of it. But it's not a big deal.