Database and Security Blog
Posted by: Echo
Hey Titan Slayers,
Some of you may have spoken with me before, and some of you might not have. I'm Spyda, and I'm one of the programmers on the TitanReach team. While I may not make the game go bleep bloop, I am the brains behind all things web, API and database, and today I wanted to take some time to walk you all through the weird and wacky behind-the-scenes work we've got happening under the hood to handle all of TitanReach's data.
This post will discuss both the technologies powering TitanReach as it is today, as well as walking you through an almost 12-month-long journey to get to where we are today.
TitanReach runs on a predominantly PHP-based stack. Despite the fact PHP is an aged language and gets a bad rap among most developers, the reality is it's a versatile language, has thorough documentation, a bunch of StackOverflow posts for when we get stuck, and most importantly... was the easiest thing we were able to get off the ground as quickly as possible in the early days while things were changing almost daily.
The Journey so Far
Back when I joined the team in September of 2020, the TitanReach team had yet to set up any web infrastructure, let alone a database backing (many of you may still remember the Kickstarter demo using unencrypted passwords!). We were racing against the clock to get content out to our community as quickly as possible. Thanks to the awesome UI work that had been done by our old Project / Community Manager Shuu, we were able to get an initial site up and running within a few days. This site was built using Bootstrap 4 (if you haven't checked it out, get onto it! Responsive web design has never been so easy), and PHP v8 (including a modern "routing" system, rather than an old "page" system). This was all run via Ubuntu on a Vultr VPS, with a MySQL database back end. Very "High School IT Project", I know, but it did the job, and more importantly, it was cheap! We were alive!
After a while, managing the build of the website took a back seat, and I started working on constructing a back-end to power the game. This involved migrating the existing database from a boring MySQL database to a more modern cloud platform - in this case, I am referring to an Azure SQLServer database. Working with Azure was a new experience for me, so there were a lot of mistakes throughout this process, but eventually, after a few weeks, the website, including a full account system, was being powered by the cloud. Fast forward a few months, and our web server is now also hosted within the Azure ecosystem.
The final piece of the puzzle is our Mirror Database and API Gateway. These components work together to provide a near-live place our users are able to access TitanReach data via a secure endpoint. I'll go on to explain the different kinds of endpoints that have been developed for TitanReach in further points below, but essentially, there are times when we need to interact with live data (for example, when we save your character's profile), but then there are times when we don't (for example, showing high scores). This is where the mirror database comes in. The data here is hosted in a separate instance of SQLServer and is updated every ~5-10 minutes, depending on latency. The benefit of this is that as we continue to grow and consume more data, we'll be able to continue providing access to it for all of our players without the fear of impacting our live game.
As far as the actual APIs are concerned, all internal APIs are delivered via a similar PHP stack to the website, while all public APIs are delivered via Node.js / Express. As we speak, I'm currently working through converting our internal APIs over to use Node.js as well. This will give us a whole host of benefits; least of all, we should see a huge improvement in terms of the processing time it takes to complete specific actions. During initial testing, comparing the PHP login API to the Node.js login API, we've been able to reduce the processing time from 1.5 seconds to 0.7 seconds - a 50% reduction! While this doesn't sound like much, just consider the benefit when we have 100 people logging in at once!
Throughout the whole journey, I've been lucky to have members of the community working with me to test each of my new features to make sure nothing has slipped through the cracks. A few people worth mentioning by name are two of our mods, Chippy and Dynny, as well as our newest team member, Echo (or as some of you may remember him, Bamboozler / Spookzilla) - without some of the combined efforts of these people, plus some other community members, we wouldn't be as secure as we are today! To give you an idea of some of the methods of security we're using (without going into incriminating detail), we are currently using SHA256 encryption as part of securing your account passwords. For those of you who aren't familiar, this is the kind of encryption used by banks, the military and many password managers. Providing it's implemented correctly, SHA256 encryption boasts the feat that it has not been cracked in its almost 20-year existence.
Next and probably not quite as interesting, we have a range of configuration options set up on our servers to prevent unauthorised snooping - one example of this is ensuring our server directories are hidden from web crawlers so they're unable to view our source code. We also ensure no credentials are ever stored in our source code - this means that if I ever forget to have my coffee in the morning and share our code repo, nothing other than some untidy code would be leaked, no personal information for our players and more importantly, no information for a hacker to directly access our database.
When it comes to front-end authentication, we have also implemented a "User Authentication and Authorisation schema" into our app. What this means is that every single user on our website is assigned "permissions" that allow them to do certain things. At a basic level, having an account gives you permission to log into the site. For our team, and some special admin, we have different permission levels which allow them to do things like create news posts, or generate item keys. All of these permissions are governed with the same kind of level of security your account information is, which means it's theoretically impossible for someone to perform an "Admin" action unless they manage to steal someone's password (friendly reminder to burn all of those post-it notes you've got on your desk with your passwords and upgrade to a password manager ASAP! It's the best thing you'll ever do!).
Finally, on the note of security, we come to our APIs. Each of our APIs are governed by specific rules. Some of these APIs are public and have been created for the purpose of providing the TitanReach community with information (think "High Scores"). These kinds of APIs have been made open, and are linked to a "Mirror Database" so it will not impact gameplay performance. The next level are internal APIs which have been created specifically for use on the website, but have not been "locked down". If you've ever snooped on the TitanReach news page, you'll know what I mean. Next, we have our "Permission Level" APIs which have been designed explicitly for use on the website. These are restricted behind the same front-end authentication that I mentioned above. Unless you have been assigned specific permissions, it's impossible to run these APIs, even if you know where to go looking for them. And lastly, we have our restricted APIs. Some of these are available for public consumption (for example, Dynny's Discord Bot) while others are restricted for use by our team to transmit information between the server and the database. These APIs are restricted via an OAuth2 flow, powered by Okta, and their authentication servers. Each time we need to use one of these APIs, we first make a "call" to Okta to prove who we are. Once this has been done, we have a short window in which we are able to make 1 or more API call. This is the most extreme level of restriction we use in our APIs and for good reason - this is where we handle information such as our login.
Information is Power
I am a strong believer in providing people with as much visibility as possible. That's something I strive for in my main job and something I want to carry on into TitanReach. That's why I try and stay active in the Discord community and am often asking people what they'd like to see. One example of this is kill tracking, which has been implemented in-game. This information is calculated by the server and sent through to my database, after which I'm able to provide this via a public API to our players... because who doesn't want to know how many chickens we've killed since Early Access went live?
As time goes on, I'd love to track more information. Some of it might be obviously useful like the number of players to complete a specific task. While others might be useful to us down the track, such as looking at what the most populated areas of the game world are or whether people prefer melee combat to ranged combat. Getting all of this information sent through to me takes our game devs time so we may not have it soon, but it is something we're always in discussions about.
If there's a specific data point you'd like to see from the game, jump into chat and talk to us about it! We can't promise we'll release every single piece of information (due to potential security implications), but we'll always review every request.
We also track some other arbitrary data such as how often someone visits a page, how people interact with a page and all login attempts. This kind of data helps us to improve our website design or to make sure we keep your accounts secure. Most importantly, we don't believe in cramming your PC full of cookies like so many websites seem to do these days. Other than your PHP Session ID (to link your session to your specific device) and your email address (if you click "remember me" when logging in), we don't store any information on your computer.
Other Bits and Pieces
This section will speak about a few other pieces to do with my work that don't really deserve an entire section on their own. To start, let's discuss payments. Much like many of the other elements of the setup, our ability to take payments was something that needed to be stood up as quickly as possible and was something I hadn't specifically had experience with in the past. We needed a way to be able to integrate some kind of secure payment portal to accept payments, not only from our own country (Australia) but across the world. This presented a few concerns about currency conversions, or what we would do if we needed to refund a player, not to mention that we were conscious our players would want the peace of mind knowing that we wouldn't just take their money and run. All of these factors ended up leading us to PayPal. Through the use of their API, we accept payments from our players, and upon receiving a confirmation message from PayPal (normally within seconds), we then credit a player's account. Overall, this has been a smashing success! A couple of transactions have proven troublesome, but we've been able to manually resolve these, and in most cases, it's within the hour of being contacted.
We are always trying to think of how we can integrate our systems with those of others to make our lives and the lives of our players easier. The best example of this is Discord. For those of you who have been supportive enough to purchase one of our Discord roles, you will already be familiar with this functionality, but for those of you who may not be aware, it is possible to tie your TitanReach and Discord accounts together. Once linked, a user who has purchased a Discord role will be able to toggle their titles off or on instantly from their account. This is all powered through Discord's secure and public API and is made possible by granting the TitanReach bot temporary access to add you to our Discord Server. No more waiting for Spyda to finish sleeping in the "Down Under" before he can turn roles on / off for you!
Posted at: 09 April 2021, 03:45:46 AM (GMT +0)