FrontierNav Report: June 2018
Changes in June
For Xenoblade 2's map, legend toggles have been implemented, allowing you to choose which map markers to show. However as the data bundle now contains substantially more data -- roughly doubled by adding all enemy spawn points -- the current approach to data management may not be feasible. So I've spent some time looking into alternative approaches.
Current approach
Right now, FrontierNav stores game data in-memory. All of it. Including the stuff not on the screen. I chose this approach because it:
- Reduces bandwidth. Data is bundled, optimised and versioned. So unless if the data changes, you only need to download it once.
- Reduces server maintenance. Since the bundle is pre-calculated, FrontierNav can handle usage spikes since it's serving the same data without any additional processing.
- Unrestricted queries. Since data queries are done client-side, there's no restrictions on the user interface. It's easy to create a one-off widget, search and visualisation entirely client-side without worrying about server load, caching and so on.
- Reduces page load times. Since the data is downloaded once when a game is chosen, successive page loads already has the data and only needs to download page templates.
- Allows offline usage. One of the goals of FrontierNav is to work offline. By having the data in a single bundle, the entire app can be used without having to download more data.
- Guarantees client-data compatibility. Since the bundles are tied to the client they're deployed with, updating the data will not break existing clients and only updated clients will fetch the new data.
There are clear downsides to this. The main one being memory usage. However, for the most part, the benefits greatly outweigh the negatives. It's just that there's a limit to how bad the negatives can get.
Solutions
The broad solution is to reduce the amount of in-memory data. There's a number of ways to go about this.
1. Client-side Database
Web Browsers now have databases like IndexedDB. It's possible to stream all the data into IndexedDB and load it into memory when needed. The downside is that FrontierNav is run off graph queries and without a graph database to run queries on, it'll take a while to re-implement.
Also, there's no guarantee that the user has a client-side database, and if they do, how much disk usage they allow. So it's quite a big can of worms.
2. Pre-calculated Responses
Much like static-site generation, it's possible to pre-calculate every individual query and generate responses for them to serve up. So instead of one big bundle, you can have lots of smaller files. A side benefit to this approach is improved search engine results as pages can be pre-generated without requiring JavaScript.
This will prevent the server load issue but it'll still couple the client to a single server. Also, to figure out all the queries to pre-calculate will require a lot of code changes.
3. Server-side Queries
I really want FrontierNav to not require a server to function and eventually move towards peer-to-peer without any central servers. But if push comes to shove, it's still an option. On the bright-side, it's traditional and I know what to do. It'll just be a lot of maintenance for a one-man side project.
4. Don't worry about it for now.
While it's a good idea to reduce memory usage, especially on mobile, 200MB may not be that bad nowadays compared to a couple of years ago.
Next Up
I'll try out various approaches to solve this issue. Every solution has its problems so it's a matter of finding out which has the best balance for FrontierNav's current needs.
It's Summer!
I haven't been able to progress FrontierNav that much these past few weeks, and it'll likely continue through Summer. This is typical during Summer as work, events and personal responsibilities ramp up, leaving less time for side projects. So a heads up as progress will be slow for the next month or so.