I've mostly been looking into which feature to focus on next and trying out various ideas.
Kind of like Wikis, Notebooks can help people complete specific goals like completing a quest or finding a set of collectibles. This would reduce the need to click around FrontierNav as Notebooks can embed entity data, maps and other visualisations in a single page for convenience.
There are some pitfalls I want to avoid with this feature. A lot of them related to the problems with Wikis. Duplication, inconsistencies, etc. However, I think it's best to get it started and see where it goes than assume the worst.
Before implementing any integrations with Discord, I'll need to better display activity on FrontierNav. So when users take certain actions, like marking things as complete and adding new games, those are visible for others to see and engage with. Then, I can easily bring that data into Discord for convenience.
That's the main approach I'll be taking with third-party integrations. They shouldn't really replace any interfaces, as it compromises FrontierNav's activity, but instead provide convenient access to them.
Once that basic integration is done, I can look into more complicated integrations like quick search.
While looking at the potential of the above two features, I couldn't help exploring other areas.
Completion and progress tracking could be greatly improved by provided clear 'achievements', either from the games or user-generated. This would allow users to clearly record what they start and complete. Combined with activity sharing, it would allow others to see what people are up to and what they've achieved.
A good way to let the community grow organically is to allow them to add any games to their library without having me police which games are supported. Progress tracking would complement this as you'd essentially have a hierarchy: To complete the game, complete these objectives.
I've been going on about giving FrontierNav a visible 'heartbeat' for a while so I'll be doing the Discord Integration next. Specifically the focus is on making user activity more visible on FrontierNav. Setting up a Discord Bot comes with additional financial cost, so I might not actually integrate it into Discord depending on how useful it ends up being.
GitHub Actions Problems
I spent more time than I wanted to dealing with failing test cases on GitHub Actions (their remote automation service). To keep it short: tests were only failing in GitHub's environment, I tried to figure out what was going wrong, eventually gave up, reverted everything, and the tests were passing again.
My development process wasn't blocked by any of this since, knowing these issues would come up, I'm able to run everything locally. But at the end of the day, it was a wild goose chase. I hope it doesn't become a regular thing.
There's a bit of work left to polish up image uploads. After that, I'll be improving the merge request process as mentioned in last month's report.
There's a bunch of smaller tasks surrounding existing features which I'll probably do on the way. But the next major feature is likely either Discord integration or the groundwork for introducing Notebook guides.
Image uploads are more or less done. There's still work left to migrate all the existing images over to the new process but it's not urgent.
The browser storage issue from last week is kind of mitigated now by uploading and clearing space on a merge request. So if users do run out of storage, they can create a merge request and carry on.
With image uploads implemented, users can now upload their own maps. Along with the ability to add their own markers, FrontierNav now supports fully user-contributed maps.
Xenoblade X of course is using its own mapping logic so it doesn't benefit from this. As mentioned previously, migrating it to the shared logic is a lot of work which I'll likely do if and when a re-release or sequel is announced.
GIFs on the Web
When recording previews for FrontierNav's features, I always bounce around between using GIFs and other video formats. It's definitely true that GIFs, archaic as they are, still drive a lot more engagement, even on platforms like Twitter which converts them to more suitable formats. I wish we moved away from GIFs by now, but it seems to be the Internet Explorer of video formats.
The preview I recorded above is over 10MB as a GIF. It's less than 3MB as an MP4. Probably a lot less as a WebM. It's a lot easier to convert a GIF to something else than the other way round. ffmpeg's GIF output isn't great whereas screen recorders like Peek do an excellent job at optimising GIFs.
Platforms like Patreon also don't allow uploading videos directly; prefering a third-party like Vimeo. However, they do allow image uploads (up to 512MB!), so GIFs work well there.
Taking all of this into account, I'll probably continue recording as GIF and exporting to other formats. Having said that, the MP4 of the GIF I uploaded to Tumblr ended up as a green, glitchy mess after it when through Tumblr's encoding pipeline. And they have a small size limit on image uploads. So... whatever.
Back on Schedule
As a rule, I'll be posting weekly updates every Friday now. It's a good way to end the week and have a record for Monday. Monthly updates will be on the Friday closest to the last day of the month.
I've been working on making Image Uploads public this week which will allow users to add images to the data tables. Eventually, it will also allow other attachments like audio and video.
Most of the work involved moving away from path strings that reference manually uploaded files to dynamically generated file names which may either be:
Locally stored, while the user is editing data.
Temporarily stored remotely, for open merge requests.
Permanently stored remotely, when it's merge into FrontierNav.
I took this approach to avoid having abritrary files being uploaded by anyone. Moderating content on the internet is a pain so I'd prefer to avoid it by making things ephemeral by default. I'd also prefer not to pay for bandwidth on data that isn't useful, so only uploading when a merge request is created does that.
I implemented the process a few months back, but my main fear holding it back is the ephemeral nature of browser storage. As far as I know, browsers only really guarantee a few megabytes of storage, if that. But I decided to make it public this week anyway instead of sitting on it. Sadly, it didn't take long for me to start seeing the first QuotaExceededError.
I mentioned it before so I guess it's time for me to take my own advice: stop using local storage. It just isn't reliable for anything. It's a shame since there's clearly a lot of work being put into it by browser vendors, but at the end of the day it's a glorified cache.
I'll be removing the local step now and jump straight to temporary remote storage. Hopefully bandwidth usage remains below any limits.
Following on from my problems with browser storage, I don't think the recommended approach of implementing offline support for web apps is going to work with FrontierNav. That is, using Service Workers and the Cache API. FrontierNav has a ton of data, and the chances of it fitting in whatever space a random user's browser provides me isn't reliable enough.
So, chances are, the only way to get offline support is to essentially ship an Electron app which downloads 'game bundles' and stores it on disk without relying on the browser. A pretty large effort which isn't going to get done any time soon. Especially if I want to add mobile support. Unless if there's a sudden large demand for it, it's not going to happen.
In my on-going research into finding ways to increase FrontierNav's exposure, I figured a great way is to introduce a Discord bot. One that can be added to popular Discord servers for users to query.
The downside is, unlike Slack, Discord does not support outgoing webhooks. I'll need a server connected to Discord 24/7 to process any queries. That's extremely wasteful and maintenance heavy compared to a serverless solution driven by webhooks. To make the most of it, I could also start introducing features for FrontierNav's Discord server to better integrate it with the web app and keep users informed.
I have some fears around overly relying on a third-party to keep in touch with FrontierNav's userbase. I did implement a simple real-time chat feature into FrontierNav for fun, but it's a lot of maintenance. Email is barely used and comes with its own burdens too. In comparison, Discord is already an extremely popular platform with solid features that comes with no financial burden. So, much like Firebase, it's too good to avoid.
Names are now visible when hovering over or selecting a marker.
Introduced toggles to choose which markers are visible.
Greatly improved querying capabilities to automatically suggest maps and markers based on search and custom filters.
Xenoblade X is excluded from these improvements as it uses its own bespoke map logic.
Sidebars are now entirely data-driven and no longer hard-coded. (2019-12-01)
Entities and Entity Types are now all colour-coded. (2019-12-01)
Tags and list items now use a common layout. (2019-12-17)
Migrated all existing games to use data-driven logic instead of hard-coded logic to allow users to easily edit and change data and presentation. (2019-12-17)
Various features and improvements to data tables and editors throughout the month.
I was hoping for a Nintendo Direct this month to confirm the date of Xenoblade Chronicles: Definitive Edition. That way I'd have a clear deadline to start adding data for it from the Wii game. I doubt a lot has changed between the two outside of additional content. Sadly, we got a measly Pokemon Direct so maybe next month.
I'll focus this month on getting the Image uploads process production ready. It's been in limbo for a while now and I have other features that can use a similar process.
I'll also be improving the Merge Request process to be more streamlined with less need to wait for me to approve and merge changes.
Since last week's migration to data-driven features, FrontierNav is starting to feel a lot more powerful. I've spent most of this week ensuring more of the in-code features are covered by data-driven features. It's been a week of deleting lots and lots of code.
Last week I mentioned focusing on Lufia II's data entry, but I couldn't fight off the temptation of removing so much technical debt. I was constantly hitting it whenever I searched the codebase. So instead of putting up with it, I went ahead migrated the last 2 games: Xenoblade 2 and Xenoblade X. Xenoblade X is still a work in progress since it's the oldest and has the most edge cases.
Formula Properties Never?
One of the most obvious feature requests for the data tables are formulas. Since the tables are kind of like spreadsheets, having a way to use formulas seems inevitable. Whenever I need a new feature, the idea of using formulas always comes up but I always look for an alternative.
Formulas are too general purpose. It's code. And for a data-driven approach, introducing that will exponentially increase complexity when it comes to data retrieval, processing and migration.
Time will tell however as there may come a point where existing features become simpler when migrated to formulas.
Reaping the Rewards
While I was migrating Xenoblade 2, everything just started falling into place automatically. That's the great thing about having a data-driven approach to problems. Add an icon in one place, and everything connected to it will use the same icon. Change a relationship to go elsewhere and everything else follows it immediately. It really does "just work". I haven't hit a data-related bug for the entire week.
Migrated Xenoblade 2 to be data-driven.
Migrated most of Xenoblade X to be data-driven.
Entities can delegate their colour and thumbnail to a relationship.
e.g. an Item can say that its colour is based on its Rarity.
Introduced "Fill Cells" to populate cells with the same value as another cell.
This reduces a lot of repetitive data entry.
Introduced experimental "Tunnel Properties" to allow Entities to delegate a property to a relationship.
e.g. a Location has a Region and Area. Instead of setting the Region in both Area and Location, Location can say its Region is the same its Area's Region. So changing the Region in Area, changes it for Location automatically.
Tunnel Properties are recursive, so a Tunnel can point to another Tunnel and so on until a value is reached.
Introduced experimental "Bulk Edit" to generate text values based on other values.
In the future, this may change to a "Formula Property" to avoid data bloat where values can be generated on the fly, but as mentioned, that comes with downsides.
Introduced icons for Entity tags to better visualise them.
Migrated some maps to use icons from data rather than hardcoding.
Migrated all Entity list items to use the same layout.
Previously FrontierNav's sidebars were hard-coded React components that grabbed data and displayed them in various ways. Over time, the layout of these sidebars gradually converged into some simple components as patterns and similarities emerged across the various datasets.
I briefly mentioned the data migration I did last week, I didn't have much to say about it but it enabled me to finally take the steps to generate layouts using just the data. No coding necessary.
Astral Chain was the first game I migrated to this approach since it's a fairly recent addition. Pokemon Sun/Moon was the next since there wasn't a lot of data.
The two hard ones are Xenoblade 2 and Xenoblade X. There's a lot more data, but more importantly, they make use of some custom components that are difficult to standardise. For example, Xenoblade X lets you choose Probes for its Probe Simulation using the Sidebar. Both have a range of custom styling to better match their games.
Of course, all of these edge cases can wait as they're not needed for most games. This comes back to my main goal: to fully document Lufia II purely through the web client to prove that FrontierNav's data editing features are viable.
Support reordering properties
Support renaming properties
Support renaming entity types.
Colours. Lots of colours.
Each Entity Type now has a random color assigned which can be changed.
Introduced Universal Entity Type indicator to know when something is an Entity Type.
Introduced Universal Entity indicator to know when something is an Entity.
Table column headings now have context menus for quick edits.
Data validation on export.
This is mainly to ensure I didn't miss any edge cases when changing things.
Redesigned modal dialogs to be less... in your face.
Browsers & Local Storage
Browsers come with ways to persist data locally using Local Storage, IndexedDB and other APIs. I've always been reluctant to use these features as most browsers tend to treat them as disposable. I can't have users making hundreds of changes stored locally and expect it to stay there. Things can go wrong.
Firefox 71 for example has broken local storage, at least the Fedora build of it. FrontierNav's authentication details are stored locally to persist logins. That doesn't work now on that browser, so users get logged out whenever they load the page. Even LastPass doesn't let me login, so I'm stuck using my phone to get passwords. I could rollback, but then I lose security updates which are more critical. I just have to wait for the patch to be release. The situation sucks.
Added more placeholder games: Lufia, Lufia II, Golden Sun, Golden Sun: The Lost Age (2019-11-14)
Various Data Table improvements. (throughout the month)
I'll be using Lufia II as a way to finish off the remaining work needed to allow anyone to contribute. At the end of this, someone should be able to contribute most of the data for an entirely new game without much input from me.
Why Lufia II? It's a reasonably small game with enough variety to match modern games. Also, it provides nostalgic motivation for me.
I released merge requests around the start of last week and it already had its first use before I even properly showed it off. I wasn't expecting it and only noticed after seeing a sustained increase in events around that feature. The first request was waiting for 3 days which isn't great. I initially contacted the contributor by email, then by Twitter after I noticed a recent follower with the same name.
Just by having this one contribution, I was able to gather a lot of data and fix a few issues. People obviously don't view things like I do, so having others even just try things helps a lot.
The lack of communication channels directly on the website is a problem but not an urgent one until more people start contributing. Merge requests currently don't allow comments. Introducing them shouldn't be difficult but ideally, I want to integrate it with the existing Community Forums to reduce duplication and maintenance.
Other FrontierNav Changes
Added incoming relationship columns to the tables.
This required a lot of work migrating the data model to support bi-directional look ups.
Listed entity types for each relationship column.
Added data validation when exporting to avoid invalid data.
Introduced documentation to help users discover more advanced features.
Marketing vs Sharing
I personally view "marketing" as a dirty word. I know it isn't, but with commercialised tracking, privacy breaches and all the lot going on in the Web, I can't help feel that way. It is the default. Whatever my feelings, I need to market FrontierNav a lot more than I currently do, otherwise no one will even know it exists.
I recently watched a GDC Talk by David Wehle where he went through how he marketed his budget indie game side project. A lot of his points reminded me of when I first shared FrontierNav. At that point, I was just sharing. I didn't view it as marketing. But David deliberately went on forums like Reddit, and posted there weekly in order to market his game. He shared other things just to avoid the Self-Promotion Rules. To me, it's a bit disingenuous, but it worked. At the end of the day, people got what they want, the forums got more activity (as he posted other things to avoid getting banned for spam), and the game was a success.
Overcoming my distate for marketing is going to take a while, but no doubt I have to do it. So I'm going to try dedicate up to an hour or so every week to get FrontierNav out there. For example, I'll be sharing gaming-related news via FrontierNav's Twitter account. I already started since today marks 2 years since Xenoblade 2's release.
I said "sharing" again without realising. I guess "sharing" is the tactical term, where as "marketing" is the strategic term.