Progress has been slow for a bit over a week due to personal reasons. This weekly also covers the week before as I didn't get a chance to write it up for that same reason.
Merge Request Flow
Currently, Merge Requests need to go through me and it requires a full application deployment to apply. That kind of sucks, especially when I have to wait 10 minutes. So, I've been working on making data updates separate from the application.
This will also allow chosen "Editors" to update and moderate games in the future so that I'm not the bottleneck. To add to this, anyone will be able to add games to FrontierNav and with enough valuable Merge Requests, those contributors can become "Editors".
This work is almost done. Annoyingly I didn't get much time to work on it even though it should only take a few hours to implement and would unlock a lot of potential.
Activity Timeline
There's been enough updates now since I introduced the activity timeline to make it public. It's a good way for people to discover FrontierNav's range of features and I've been using it personally so see how alive the website is. So, it's a priority after improving the Merge Request flow.
GitHub Actions
I've been using GitHub Actions to run deployments for a few months now, but recently it's been really acting up. Deployments hang forever, ChromeDriver fails to launch, and so on. I've just turned it off and went back to manual deployments. It's pretty easy to re-enable again when I need it so I'm not too fussed.
To progress the "Wiki", I've introduced text attachments. Text attachments, like image attachments, represent text files that you can upload. In the future, the text can be entered and uploaded through FrontierNav rather than requiring you to upload files.
Right now, you can manually add an overview Text Attachment Property to any Entity Type and it will render it as basic markdown on the Entity Overview page.
An Alternative to Text Attachments
After implementing text attachments, I didn't really like it. As mentioned before, it's too arbritary and prone to editorialisation. Something I want to avoid polluting the main database with.
Instead, I've been thinking about a different solution. Rather than having a single "Wiki" page, individuals should be able to create their own guides and link them to specific Entities. These guides would not be in the main database, they'll be in the user database like the forums and completion tracking are.
This would allow anyone to contribute without having to go through the stricter Merge Request progress. To add to this, the features required to make these guides would be useful across FrontierNav, as they're essentially powered up forum posts.
Email Link Login
Over time, FrontierNav will be moving away from social network logins. The main reason is because social networks are constantly changing their privacy policies and I'm constantly having to keep up with them. It's tedious to managed so many integrations and it just keeps growing. There's always the risk of people losing access to their accounts if any of the integrations stops working. This is especially worse for Twitter as they don't even provide an email to use for account recovery.
So, you can now login by providing just your email. A login link is sent to it to login. Pretty simple. Not as convenient as a one-click OAuth login, but since FrontierNav keeps you logged in, it's not a constant annoyance.
I'm avoiding Email/Password logins because it's a huge responsibility to store salted and hashed passwords and deal with hacked accounts.
It's Not A Back Button
I noticed a lot of people use the FrontierNav logo on the top-left as a back button. However, it's a home button. People would press it, realise they went to the wrong place, go back and press something else. To save RAM, FrontierNav cleans up data when going back to the main homepage so this user flow is not only wrong, but it also wastes resources.
So now, when navigating a game, the top-left navigation item is the game's homepage rather than FrontierNav's. When on the game's homepage, it changes back to the FrontierNav logo and takes you to the main homepage.
The consequence of this is a loss of branding, but I can always add branding elsewhere in the future (like the bottom-left).
User Library
To progress user libraries and allow users to add their own games, I've made some minor improvements to how libraries are stored. Libraries can now be sorted and filtered by status (i.e. 'playing', 'completed', etc.).
Activity Timeline
Almost forgot to mention, there's now an activity timeline to see what people are up to on FrontierNav. Activities include things like adding games to their library, posting on the forum and creating merge requests.
I have not made it public yet as the timeline does not include any activity before it was implemented so it's a bit empty.
This will be intergrated into Discord somewhere down the line for easier access and notifications.
I've mostly been looking into which feature to focus on next and trying out various ideas.
Notebooks
Kind of like Wikis, Notebooks can help people complete specific goals like completing a quest or finding a set of collectibles. This would reduce the need to click around FrontierNav as Notebooks can embed entity data, maps and other visualisations in a single page for convenience.
There are some pitfalls I want to avoid with this feature. A lot of them related to the problems with Wikis. Duplication, inconsistencies, etc. However, I think it's best to get it started and see where it goes than assume the worst.
Discord Integration
Before implementing any integrations with Discord, I'll need to better display activity on FrontierNav. So when users take certain actions, like marking things as complete and adding new games, those are visible for others to see and engage with. Then, I can easily bring that data into Discord for convenience.
That's the main approach I'll be taking with third-party integrations. They shouldn't really replace any interfaces, as it compromises FrontierNav's activity, but instead provide convenient access to them.
Once that basic integration is done, I can look into more complicated integrations like quick search.
Other Features
While looking at the potential of the above two features, I couldn't help exploring other areas.
Completion and progress tracking could be greatly improved by provided clear 'achievements', either from the games or user-generated. This would allow users to clearly record what they start and complete. Combined with activity sharing, it would allow others to see what people are up to and what they've achieved.
A good way to let the community grow organically is to allow them to add any games to their library without having me police which games are supported. Progress tracking would complement this as you'd essentially have a hierarchy: To complete the game, complete these objectives.
Choosing One
I've been going on about giving FrontierNav a visible 'heartbeat' for a while so I'll be doing the Discord Integration next. Specifically the focus is on making user activity more visible on FrontierNav. Setting up a Discord Bot comes with additional financial cost, so I might not actually integrate it into Discord depending on how useful it ends up being.
GitHub Actions Problems
I spent more time than I wanted to dealing with failing test cases on GitHub Actions (their remote automation service). To keep it short: tests were only failing in GitHub's environment, I tried to figure out what was going wrong, eventually gave up, reverted everything, and the tests were passing again.
My development process wasn't blocked by any of this since, knowing these issues would come up, I'm able to run everything locally. But at the end of the day, it was a wild goose chase. I hope it doesn't become a regular thing.
There's a bit of work left to polish up image uploads. After that, I'll be improving the merge request process as mentioned in last month's report.
There's a bunch of smaller tasks surrounding existing features which I'll probably do on the way. But the next major feature is likely either Discord integration or the groundwork for introducing Notebook guides.
Image uploads are more or less done. There's still work left to migrate all the existing images over to the new process but it's not urgent.
The browser storage issue from last week is kind of mitigated now by uploading and clearing space on a merge request. So if users do run out of storage, they can create a merge request and carry on.
Map Uploads
With image uploads implemented, users can now upload their own maps. Along with the ability to add their own markers, FrontierNav now supports fully user-contributed maps.
Map Editing in FrontierNav
Xenoblade X of course is using its own mapping logic so it doesn't benefit from this. As mentioned previously, migrating it to the shared logic is a lot of work which I'll likely do if and when a re-release or sequel is announced.
GIFs on the Web
When recording previews for FrontierNav's features, I always bounce around between using GIFs and other video formats. It's definitely true that GIFs, archaic as they are, still drive a lot more engagement, even on platforms like Twitter which converts them to more suitable formats. I wish we moved away from GIFs by now, but it seems to be the Internet Explorer of video formats.
The preview I recorded above is over 10MB as a GIF. It's less than 3MB as an MP4. Probably a lot less as a WebM. It's a lot easier to convert a GIF to something else than the other way round. ffmpeg's GIF output isn't great whereas screen recorders like Peek do an excellent job at optimising GIFs.
Platforms like Patreon also don't allow uploading videos directly; prefering a third-party like Vimeo. However, they do allow image uploads (up to 512MB!), so GIFs work well there.
Taking all of this into account, I'll probably continue recording as GIF and exporting to other formats. Having said that, the MP4 of the GIF I uploaded to Tumblr ended up as a green, glitchy mess after it when through Tumblr's encoding pipeline. And they have a small size limit on image uploads. So... whatever.
Back on Schedule
As a rule, I'll be posting weekly updates every Friday now. It's a good way to end the week and have a record for Monday. Monthly updates will be on the Friday closest to the last day of the month.
I've been working on making Image Uploads public this week which will allow users to add images to the data tables. Eventually, it will also allow other attachments like audio and video.
Most of the work involved moving away from path strings that reference manually uploaded files to dynamically generated file names which may either be:
Locally stored, while the user is editing data.
Temporarily stored remotely, for open merge requests.
Permanently stored remotely, when it's merge into FrontierNav.
I took this approach to avoid having abritrary files being uploaded by anyone. Moderating content on the internet is a pain so I'd prefer to avoid it by making things ephemeral by default. I'd also prefer not to pay for bandwidth on data that isn't useful, so only uploading when a merge request is created does that.
I implemented the process a few months back, but my main fear holding it back is the ephemeral nature of browser storage. As far as I know, browsers only really guarantee a few megabytes of storage, if that. But I decided to make it public this week anyway instead of sitting on it. Sadly, it didn't take long for me to start seeing the first QuotaExceededError.
I mentioned it before so I guess it's time for me to take my own advice: stop using local storage. It just isn't reliable for anything. It's a shame since there's clearly a lot of work being put into it by browser vendors, but at the end of the day it's a glorified cache.
I'll be removing the local step now and jump straight to temporary remote storage. Hopefully bandwidth usage remains below any limits.
Offline Support
Following on from my problems with browser storage, I don't think the recommended approach of implementing offline support for web apps is going to work with FrontierNav. That is, using Service Workers and the Cache API. FrontierNav has a ton of data, and the chances of it fitting in whatever space a random user's browser provides me isn't reliable enough.
So, chances are, the only way to get offline support is to essentially ship an Electron app which downloads 'game bundles' and stores it on disk without relying on the browser. A pretty large effort which isn't going to get done any time soon. Especially if I want to add mobile support. Unless if there's a sudden large demand for it, it's not going to happen.
Discord Integration
In my on-going research into finding ways to increase FrontierNav's exposure, I figured a great way is to introduce a Discord bot. One that can be added to popular Discord servers for users to query.
The downside is, unlike Slack, Discord does not support outgoing webhooks. I'll need a server connected to Discord 24/7 to process any queries. That's extremely wasteful and maintenance heavy compared to a serverless solution driven by webhooks. To make the most of it, I could also start introducing features for FrontierNav's Discord server to better integrate it with the web app and keep users informed.
I have some fears around overly relying on a third-party to keep in touch with FrontierNav's userbase. I did implement a simple real-time chat feature into FrontierNav for fun, but it's a lot of maintenance. Email is barely used and comes with its own burdens too. In comparison, Discord is already an extremely popular platform with solid features that comes with no financial burden. So, much like Firebase, it's too good to avoid.
Names are now visible when hovering over or selecting a marker.
Introduced toggles to choose which markers are visible.
Greatly improved querying capabilities to automatically suggest maps and markers based on search and custom filters.
Xenoblade X is excluded from these improvements as it uses its own bespoke map logic.
Sidebars are now entirely data-driven and no longer hard-coded. (2019-12-01)
Entities and Entity Types are now all colour-coded. (2019-12-01)
Tags and list items now use a common layout. (2019-12-17)
Migrated all existing games to use data-driven logic instead of hard-coded logic to allow users to easily edit and change data and presentation. (2019-12-17)
Various features and improvements to data tables and editors throughout the month.
Next Up
I was hoping for a Nintendo Direct this month to confirm the date of Xenoblade Chronicles: Definitive Edition. That way I'd have a clear deadline to start adding data for it from the Wii game. I doubt a lot has changed between the two outside of additional content. Sadly, we got a measly Pokemon Direct so maybe next month.
I'll focus this month on getting the Image uploads process production ready. It's been in limbo for a while now and I have other features that can use a similar process.
I'll also be improving the Merge Request process to be more streamlined with less need to wait for me to approve and merge changes.
Since last week's migration to data-driven features, FrontierNav is starting to feel a lot more powerful. I've spent most of this week ensuring more of the in-code features are covered by data-driven features. It's been a week of deleting lots and lots of code.
Last week I mentioned focusing on Lufia II's data entry, but I couldn't fight off the temptation of removing so much technical debt. I was constantly hitting it whenever I searched the codebase. So instead of putting up with it, I went ahead migrated the last 2 games: Xenoblade 2 and Xenoblade X. Xenoblade X is still a work in progress since it's the oldest and has the most edge cases.
Formula Properties Never?
One of the most obvious feature requests for the data tables are formulas. Since the tables are kind of like spreadsheets, having a way to use formulas seems inevitable. Whenever I need a new feature, the idea of using formulas always comes up but I always look for an alternative.
Formulas are too general purpose. It's code. And for a data-driven approach, introducing that will exponentially increase complexity when it comes to data retrieval, processing and migration.
Time will tell however as there may come a point where existing features become simpler when migrated to formulas.
Reaping the Rewards
While I was migrating Xenoblade 2, everything just started falling into place automatically. That's the great thing about having a data-driven approach to problems. Add an icon in one place, and everything connected to it will use the same icon. Change a relationship to go elsewhere and everything else follows it immediately. It really does "just work". I haven't hit a data-related bug for the entire week.
Other Changes
Migrated Xenoblade 2 to be data-driven.
Migrated most of Xenoblade X to be data-driven.
Entities can delegate their colour and thumbnail to a relationship.
e.g. an Item can say that its colour is based on its Rarity.
Introduced "Fill Cells" to populate cells with the same value as another cell.
This reduces a lot of repetitive data entry.
Introduced experimental "Tunnel Properties" to allow Entities to delegate a property to a relationship.
e.g. a Location has a Region and Area. Instead of setting the Region in both Area and Location, Location can say its Region is the same its Area's Region. So changing the Region in Area, changes it for Location automatically.
Tunnel Properties are recursive, so a Tunnel can point to another Tunnel and so on until a value is reached.
Introduced experimental "Bulk Edit" to generate text values based on other values.
In the future, this may change to a "Formula Property" to avoid data bloat where values can be generated on the fly, but as mentioned, that comes with downsides.
Introduced icons for Entity tags to better visualise them.
Migrated some maps to use icons from data rather than hardcoding.
Migrated all Entity list items to use the same layout.
Previously FrontierNav's sidebars were hard-coded React components that grabbed data and displayed them in various ways. Over time, the layout of these sidebars gradually converged into some simple components as patterns and similarities emerged across the various datasets.
I briefly mentioned the data migration I did last week, I didn't have much to say about it but it enabled me to finally take the steps to generate layouts using just the data. No coding necessary.
Astral Chain was the first game I migrated to this approach since it's a fairly recent addition. Pokemon Sun/Moon was the next since there wasn't a lot of data.
The two hard ones are Xenoblade 2 and Xenoblade X. There's a lot more data, but more importantly, they make use of some custom components that are difficult to standardise. For example, Xenoblade X lets you choose Probes for its Probe Simulation using the Sidebar. Both have a range of custom styling to better match their games.
Of course, all of these edge cases can wait as they're not needed for most games. This comes back to my main goal: to fully document Lufia II purely through the web client to prove that FrontierNav's data editing features are viable.
Other Changes
Support reordering properties
Support renaming properties
Support renaming entity types.
Colours. Lots of colours.
Each Entity Type now has a random color assigned which can be changed.
Introduced Universal Entity Type indicator to know when something is an Entity Type.
Introduced Universal Entity indicator to know when something is an Entity.
Table column headings now have context menus for quick edits.
Data validation on export.
This is mainly to ensure I didn't miss any edge cases when changing things.
Redesigned modal dialogs to be less... in your face.
Browsers & Local Storage
Browsers come with ways to persist data locally using Local Storage, IndexedDB and other APIs. I've always been reluctant to use these features as most browsers tend to treat them as disposable. I can't have users making hundreds of changes stored locally and expect it to stay there. Things can go wrong.
Firefox 71 for example has broken local storage, at least the Fedora build of it. FrontierNav's authentication details are stored locally to persist logins. That doesn't work now on that browser, so users get logged out whenever they load the page. Even LastPass doesn't let me login, so I'm stuck using my phone to get passwords. I could rollback, but then I lose security updates which are more critical. I just have to wait for the patch to be release. The situation sucks.
