I've been working on making Image Uploads public this week which will allow users to add images to the data tables. Eventually, it will also allow other attachments like audio and video.
Most of the work involved moving away from path strings that reference manually uploaded files to dynamically generated file names which may either be:
Locally stored, while the user is editing data.
Temporarily stored remotely, for open merge requests.
Permanently stored remotely, when it's merge into FrontierNav.
I took this approach to avoid having abritrary files being uploaded by anyone. Moderating content on the internet is a pain so I'd prefer to avoid it by making things ephemeral by default. I'd also prefer not to pay for bandwidth on data that isn't useful, so only uploading when a merge request is created does that.
I implemented the process a few months back, but my main fear holding it back is the ephemeral nature of browser storage. As far as I know, browsers only really guarantee a few megabytes of storage, if that. But I decided to make it public this week anyway instead of sitting on it. Sadly, it didn't take long for me to start seeing the first QuotaExceededError.
I mentioned it before so I guess it's time for me to take my own advice: stop using local storage. It just isn't reliable for anything. It's a shame since there's clearly a lot of work being put into it by browser vendors, but at the end of the day it's a glorified cache.
I'll be removing the local step now and jump straight to temporary remote storage. Hopefully bandwidth usage remains below any limits.
Offline Support
Following on from my problems with browser storage, I don't think the recommended approach of implementing offline support for web apps is going to work with FrontierNav. That is, using Service Workers and the Cache API. FrontierNav has a ton of data, and the chances of it fitting in whatever space a random user's browser provides me isn't reliable enough.
So, chances are, the only way to get offline support is to essentially ship an Electron app which downloads 'game bundles' and stores it on disk without relying on the browser. A pretty large effort which isn't going to get done any time soon. Especially if I want to add mobile support. Unless if there's a sudden large demand for it, it's not going to happen.
Discord Integration
In my on-going research into finding ways to increase FrontierNav's exposure, I figured a great way is to introduce a Discord bot. One that can be added to popular Discord servers for users to query.
The downside is, unlike Slack, Discord does not support outgoing webhooks. I'll need a server connected to Discord 24/7 to process any queries. That's extremely wasteful and maintenance heavy compared to a serverless solution driven by webhooks. To make the most of it, I could also start introducing features for FrontierNav's Discord server to better integrate it with the web app and keep users informed.
I have some fears around overly relying on a third-party to keep in touch with FrontierNav's userbase. I did implement a simple real-time chat feature into FrontierNav for fun, but it's a lot of maintenance. Email is barely used and comes with its own burdens too. In comparison, Discord is already an extremely popular platform with solid features that comes with no financial burden. So, much like Firebase, it's too good to avoid.
Names are now visible when hovering over or selecting a marker.
Introduced toggles to choose which markers are visible.
Greatly improved querying capabilities to automatically suggest maps and markers based on search and custom filters.
Xenoblade X is excluded from these improvements as it uses its own bespoke map logic.
Sidebars are now entirely data-driven and no longer hard-coded. (2019-12-01)
Entities and Entity Types are now all colour-coded. (2019-12-01)
Tags and list items now use a common layout. (2019-12-17)
Migrated all existing games to use data-driven logic instead of hard-coded logic to allow users to easily edit and change data and presentation. (2019-12-17)
Various features and improvements to data tables and editors throughout the month.
Next Up
I was hoping for a Nintendo Direct this month to confirm the date of Xenoblade Chronicles: Definitive Edition. That way I'd have a clear deadline to start adding data for it from the Wii game. I doubt a lot has changed between the two outside of additional content. Sadly, we got a measly Pokemon Direct so maybe next month.
I'll focus this month on getting the Image uploads process production ready. It's been in limbo for a while now and I have other features that can use a similar process.
I'll also be improving the Merge Request process to be more streamlined with less need to wait for me to approve and merge changes.
I didn't publish a weekly for the week before since it was a holiday week. So this weekly will be a combination of the two, that is the 16th and 23rd.
All of the updates are related to FrontierNav. I haven't really blogged much about specific topics for the last few months. Maybe I should.
FrontierNav
Improved Map Visuals
Maps are now visually consistent across games. This was needed as FrontierNav moves towards using the same maps for all future games.
Icons are now outlined in the colour that represents them across FrontierNav
When selected, icons change to a "map pin" so that they stand out and avoid obstructing the location they're indicating.
Names are shown next to icons when they're hovered over or selected.
Icons move to the front when hovered over or selected to make them more visible.
Showcasing the improved visuals.
Introduced Map Feature Toggles
Some map features are hidden by default to reduce performance issues when there's hundreds of features on the screen. They are only visible when it's relevant to your search. However, this made manual discovery difficult as users didn't know what sorts of information were available. So now, you can now unhide features.
In the future, maps can hide/show features based on how zoomed in the user is. However, since there's no culling for features outside the viewport, it will still have a performance impact. Which leads onto...
Showcasing feature toggles.
Performance Issues
The new map visuals use CSS filters extensively. This seems fine for Chromium-based browsers, but Firefox struggles with it. It pretty much freezes on mobile.
I'm considering moving to a canvas-based mapping library like OpenLayers as it's clear the DOM can't handle so many elements.
Improved Map Feature Discovery
Previously, map features were discovered and selected through hard-coded queries for each game. Searching for a Collectible in Xenoblade 2 would trigger a specific query to find the related Collection Points. This meant if a new type of entity was added, the maps wouldn't know how to find features relevant to it.
Now, instead of a custom query for every type of entity, FrontierNav discovers these relationships by itself. So when you search for a Collectible, FrontierNav will look for the various ways it's connected to a map location and show you those.
Improved Map Discovery
In a similar manner to feature discovery, map lists were also hard-coded. Maps had to be under a specific Region, Area, Map hierarchy. This structure came from Xenoblade 2's in-game map, and even there it often didn't make sense. The idea of Regions, Areas and Locations are vague and far from strict. They are context sensitive depending on your viewpoint.
This structure also doesn't work for general use cases such as in Astral Chain, where locations are re-used but have different features based on the current mission.
So instead of a strict hierarchy, Maps are now discoverable based on what they represent and what their features represent. This generic capability is the same as how map features are discovered.
Finding Secret Areas in Xenoblade 2, and filtering by Region to further refine the search.
Revamped Sidebars
I always had a nagging feeling whenever I opened the sidebar. The font sizes never looked correct and the alignments seemed off. So I went ahead and improved it.
Headings are more muted to give focus on data.
Migrations Mostly Complete
The only game left to migrate now is Xenoblade X. As I mentioned in the previous weekly, Xenoblade X is the oldest game in FrontierNav and has a lot of custom features specific to it. Moving that over to a generic featureset is going to take some time.
If a new release of Xenoblade X is announced for the Switch, I'll have more incentive to move it over, but as it is right now, there's not much reason to put the effort in.
Other Changes
Greatly improved the speed at which local changes re-apply.
Fixed a bug which causes partially re-applied changes to persist, causing perceived data loss.
Introduced forms to create properties, replacing the temporary alert-driven approach.
Moved all search logic to web workers.
This might improve performance as a side effect, but was more done for refactoring.
Fixed layout on Mobile Chromium where navigation bar was cut-off.
Isolated remaining Xenoblade X code to only execute under Xenoblade X.
Since last week's migration to data-driven features, FrontierNav is starting to feel a lot more powerful. I've spent most of this week ensuring more of the in-code features are covered by data-driven features. It's been a week of deleting lots and lots of code.
Last week I mentioned focusing on Lufia II's data entry, but I couldn't fight off the temptation of removing so much technical debt. I was constantly hitting it whenever I searched the codebase. So instead of putting up with it, I went ahead migrated the last 2 games: Xenoblade 2 and Xenoblade X. Xenoblade X is still a work in progress since it's the oldest and has the most edge cases.
Formula Properties Never?
One of the most obvious feature requests for the data tables are formulas. Since the tables are kind of like spreadsheets, having a way to use formulas seems inevitable. Whenever I need a new feature, the idea of using formulas always comes up but I always look for an alternative.
Formulas are too general purpose. It's code. And for a data-driven approach, introducing that will exponentially increase complexity when it comes to data retrieval, processing and migration.
Time will tell however as there may come a point where existing features become simpler when migrated to formulas.
Reaping the Rewards
While I was migrating Xenoblade 2, everything just started falling into place automatically. That's the great thing about having a data-driven approach to problems. Add an icon in one place, and everything connected to it will use the same icon. Change a relationship to go elsewhere and everything else follows it immediately. It really does "just work". I haven't hit a data-related bug for the entire week.
Other Changes
Migrated Xenoblade 2 to be data-driven.
Migrated most of Xenoblade X to be data-driven.
Entities can delegate their colour and thumbnail to a relationship.
e.g. an Item can say that its colour is based on its Rarity.
Introduced "Fill Cells" to populate cells with the same value as another cell.
This reduces a lot of repetitive data entry.
Introduced experimental "Tunnel Properties" to allow Entities to delegate a property to a relationship.
e.g. a Location has a Region and Area. Instead of setting the Region in both Area and Location, Location can say its Region is the same its Area's Region. So changing the Region in Area, changes it for Location automatically.
Tunnel Properties are recursive, so a Tunnel can point to another Tunnel and so on until a value is reached.
Introduced experimental "Bulk Edit" to generate text values based on other values.
In the future, this may change to a "Formula Property" to avoid data bloat where values can be generated on the fly, but as mentioned, that comes with downsides.
Introduced icons for Entity tags to better visualise them.
Migrated some maps to use icons from data rather than hardcoding.
Migrated all Entity list items to use the same layout.
Previously FrontierNav's sidebars were hard-coded React components that grabbed data and displayed them in various ways. Over time, the layout of these sidebars gradually converged into some simple components as patterns and similarities emerged across the various datasets.
I briefly mentioned the data migration I did last week, I didn't have much to say about it but it enabled me to finally take the steps to generate layouts using just the data. No coding necessary.
Astral Chain was the first game I migrated to this approach since it's a fairly recent addition. Pokemon Sun/Moon was the next since there wasn't a lot of data.
The two hard ones are Xenoblade 2 and Xenoblade X. There's a lot more data, but more importantly, they make use of some custom components that are difficult to standardise. For example, Xenoblade X lets you choose Probes for its Probe Simulation using the Sidebar. Both have a range of custom styling to better match their games.
Of course, all of these edge cases can wait as they're not needed for most games. This comes back to my main goal: to fully document Lufia II purely through the web client to prove that FrontierNav's data editing features are viable.
Other Changes
Support reordering properties
Support renaming properties
Support renaming entity types.
Colours. Lots of colours.
Each Entity Type now has a random color assigned which can be changed.
Introduced Universal Entity Type indicator to know when something is an Entity Type.
Introduced Universal Entity indicator to know when something is an Entity.
Table column headings now have context menus for quick edits.
Data validation on export.
This is mainly to ensure I didn't miss any edge cases when changing things.
Redesigned modal dialogs to be less... in your face.
Browsers & Local Storage
Browsers come with ways to persist data locally using Local Storage, IndexedDB and other APIs. I've always been reluctant to use these features as most browsers tend to treat them as disposable. I can't have users making hundreds of changes stored locally and expect it to stay there. Things can go wrong.
Firefox 71 for example has broken local storage, at least the Fedora build of it. FrontierNav's authentication details are stored locally to persist logins. That doesn't work now on that browser, so users get logged out whenever they load the page. Even LastPass doesn't let me login, so I'm stuck using my phone to get passwords. I could rollback, but then I lose security updates which are more critical. I just have to wait for the patch to be release. The situation sucks.
Added more placeholder games: Lufia, Lufia II, Golden Sun, Golden Sun: The Lost Age (2019-11-14)
Various Data Table improvements. (throughout the month)
Next Up
I'll be using Lufia II as a way to finish off the remaining work needed to allow anyone to contribute. At the end of this, someone should be able to contribute most of the data for an entirely new game without much input from me.
Why Lufia II? It's a reasonably small game with enough variety to match modern games. Also, it provides nostalgic motivation for me.
I released merge requests around the start of last week and it already had its first use before I even properly showed it off. I wasn't expecting it and only noticed after seeing a sustained increase in events around that feature. The first request was waiting for 3 days which isn't great. I initially contacted the contributor by email, then by Twitter after I noticed a recent follower with the same name.
Just by having this one contribution, I was able to gather a lot of data and fix a few issues. People obviously don't view things like I do, so having others even just try things helps a lot.
The lack of communication channels directly on the website is a problem but not an urgent one until more people start contributing. Merge requests currently don't allow comments. Introducing them shouldn't be difficult but ideally, I want to integrate it with the existing Community Forums to reduce duplication and maintenance.
Other FrontierNav Changes
Added incoming relationship columns to the tables.
This required a lot of work migrating the data model to support bi-directional look ups.
Listed entity types for each relationship column.
Added data validation when exporting to avoid invalid data.
Introduced documentation to help users discover more advanced features.
Marketing vs Sharing
I personally view "marketing" as a dirty word. I know it isn't, but with commercialised tracking, privacy breaches and all the lot going on in the Web, I can't help feel that way. It is the default. Whatever my feelings, I need to market FrontierNav a lot more than I currently do, otherwise no one will even know it exists.
I recently watched a GDC Talk by David Wehle where he went through how he marketed his budget indie game side project. A lot of his points reminded me of when I first shared FrontierNav. At that point, I was just sharing. I didn't view it as marketing. But David deliberately went on forums like Reddit, and posted there weekly in order to market his game. He shared other things just to avoid the Self-Promotion Rules. To me, it's a bit disingenuous, but it worked. At the end of the day, people got what they want, the forums got more activity (as he posted other things to avoid getting banned for spam), and the game was a success.
Overcoming my distate for marketing is going to take a while, but no doubt I have to do it. So I'm going to try dedicate up to an hour or so every week to get FrontierNav out there. For example, I'll be sharing gaming-related news via FrontierNav's Twitter account. I already started since today marks 2 years since Xenoblade 2's release.
I said "sharing" again without realising. I guess "sharing" is the tactical term, where as "marketing" is the strategic term.
I finally bought a new mechnical keyboard. Previously, I was using the Logitech G710+ which is a full-size keyboard with Cherry MX Brown switches. It's big, but comfortable. The media keys and volume rocker are especially convenient. The thick chunky cable is especially inconvenient.
At my previous workplace, I used a Razer Black Widow Tournament Edition which was a tenkeyless (TKL) with Razer Green switches, similar to Cherry MX Blue. Overall, I much preferred the clicky-ness of the Razer Greens over the Cherry Browns and I've been unsatisfied with it ever since. I would've bought a Razer Green keyboard if they weren't so expensive and prone to issues. The one in my workplace used to randomly shoot out its Left CTRL key...
I tried the Das Keyboard which was nice, but I prefer a TKL layout which they only manufacture with Cherry Browns. Filco was also on my list, but they they're a bit old so I was risking spending a lot of money for something that might get a new model soon (even their website is straight out of Flash-era).
This sort of whac-a-mole went on for months. I even went down the rabbit hole of customisable do-it-yourself keyboards with hot-swapping switch slots. I blame YouTube's algorithms for that. Luckily, most of the choices aren't available in the UK so I dodged a bullet. At the same time, it kind of sucks how far behind the UK is.
Anyways, this week one of the keyboards I was eyeing on dropped in price: the HyperX Alloy FPS Pro. TKL and Cherry Blue. ~60 GBP. The board is pretty much edge-to-edge just the keys, no bezels, solid steel with no flex. No spacebar rattle. Perfect. Yet, whereever I look, it doesn't get much recommendations. Maybe I'm missing something, but I've got it now I and I'm happy. So much more desk space, and when I'm typing, it feels like a machine gun.
Cons? The red lights are garish, but I switch off backlighting anyway. The red-striped Mini USB cable isn't great either, but it's not a big deal. Neither is the logo on the spacebar. US layout only? I can get used to it.
In terms of personal issues, I have noticed some wrist pain but a wrist rest should sort that out. My key misses due to the US layout should fix itself over time. I'll probably use acronyms for currencies more often now with the GBP symbol key. I can't use my right thumb to hit Enter quickly without the numpad (the only time I ever use one).
Lastly, probably my biggest issue: the actuation force is really high. I don't recall needing so much force when I was using other Cherry Blues, but it has its pros and cons. I'm now hardly ever bottoming out the keys so I'm typing a lot faster. But if I'm pressing more keys which require more force, my fingers get tired even faster. The cold weather doesn't help.
All of these personal issues I feel will lessen over time as I get used to it. Like with most niche computer accessories, the keyboard is "gaming" branded, but it definitely is not for gaming. Overall, I'm very happy with it. After typing this post, my fingers say otherwise.
I wasn't planning to write a monthly report since I'm already writing weekly ones. But I realised weekly reports are a bit varied and it's nice to have a monthly update just around FrontierNav. This report only covers October. I'll be sharing what I did in November in a future report.
Changes in October
Optimised Data Tables for smoother scrolling (2019-10-13)
Introduced Staging Environment for more accurate automated tests (2019-10-29)
More features around Data Tables.
Windows in FrontierNav
Pop-out Windows
Pop-out Windows are the biggest feature this month. It's a huge convenience on desktop and saves a lot of clicking around.
They are a bit limited. Windows can't be resized and their content is static. However the ground work has been laid for more advanced features using the "Window Manager", such as...
Sidebar Behaviour
Currently the Sidebar is tied to the URL. The Main Window, where the Maps and other visualisations are rendered, also relies on the URL.
Previously, FrontierNav only really had one context so sharing a single state, the URL, was never an issue. But the limitations are starting to show as new features start conflicting with existing ones.
For example, on mobile viewports, the Sidebar covers the Main Window. At the same time, closing the Sidebar causes the Main Window to change too; making certain pages inaccessible. I've been working around this by essentially having permutations of state for each page: one for the Sidebar, one for the Main Window, and one for both. This obviously is a major headache to manage.
Ideally, the behaviour of the Sidebar should depend on the context. So having the contexts drive that behaviour makes the most sense. Things like "Show the Sidebar when the user selects a search result", "Show the Sidebar when the user expands a table row", and so on.
Now with a Window Manager implemented, this sort of behaviour should be easier to implemented. The Sidebar pretty much is a Window, except it's docked to the left side rather than freely floating.
I haven't release this change yet, but it's one example of what the "Window Manager" enables.
Data Tables
As always, Data Tables have been increasing in features as-needed. They're not that major to individually list. I've also added more data for Astral Chain such as Enemy Spawns and tidied up existing data from previous games.
There are still certain processes that I need to migrate over. Image upload is probably the more obvious one but it carries a huge security and cost risk compared to everything else. There's also templating to properly render the data in the Sidebar which is currently done in code.
I'm going to also have to start thinking about on-boarding processes to get others to use the data editing tools. Things like documentation, user guides, integrated merge processes and so on. There's a lot to do.
For some reason, Cloudflare has started to report an ever-increasing number of "Unique Visitors". Currently, it stands at 4 times the usual levels. It'd be great if that was true but I'm doubtful.
My access logs, which avoid Cloudflare's cache, say it's more or less the same as before. Cloudflare's other metrics like "Total Requests" are also the same as before. Nothing else is following this new trend. So there's no reason to believe it.
GitHub Actions
I noticed node-terraform's automated publish workflow wasn't get triggered when new version tags were pushed. I use the exact same trigger for FrontierNav and it works fine. The only difference is that I manually push tags for FrontierNav, whereas node-terraform's is pushed by another workflow.
I'm kind of burnt out from debugging GitHub Actions so I'm giving it a break. It's probably an issue on their end or yet another caveat like a lot of the previous issues I had.
Google Search is Trash
I've been using DuckDuckGo as my default search engine for over a year now. Everything's been good, and having the !g command to fallback to Google has helped ease the transition to a less forgiving service.
However, I noticed something: Google is become worse at being a search engine as time goes on. It's full of "SEO" trash websites. The results are useless without basically telling it what website to search through using the site: keyword. The top half of the first page is always full of auto-generated junk too.
I don't know how long this trend will last, but I'm becoming more and more reliant on my bookmarks nowadays to find specific sites and run searches through them.
