Near the end of 2015, I wanted to get back into working on my own games. Shortly after finishing my master's program I was just mentally and psychologically exhausted. I had spent a significant amount of time playing games with a few friends and started to get an itch for making my own game. I was originally thinking to make a mobile game that I would want to play while taking transit. Something I could easily pause but also be complicated and requiring a good amount of problem-solving. At the time I had spent many hours playing Factorio, and so I was wanting to create a tower defense which required you to collect resources, manage power/communications, and fend off attacks from a moderately intelligent adversary.
I got a very basic prototype done after a few weeks, but I started to envision a much bigger project. I was no longer taking transit simply because driving took about 1/5 the time. Together both of those factors made me change my mind. I began looking into technologies and libraries I could use to make it happen. Most of my development had focused on 2D but I had done some basic 3D work. In the end, I decided it would be worth my time to look into Unreal 4 vs Unity 5 as my experience with both engines had shown they had extremely robust rendering engines that would easily outdo anything I could create. They also had the nice feature of being able to support multiple platforms. The end result was my decision to go with Unreal 4, as the performance as my rendering engine exceeded what Unity 5 could deliver.
After a few weeks of hacking together the general concept for what I wanted to create I through together a basic prototype in April 2016. Which you can take a look at below:
It was very simple but gave me the foundation for what I was going to build. As time went on I slowly started to build up a robust engine which delegated the rendering to Unreal. I still use the approach today and surprisingly it allows for rapid iterations and excellent utilization of the hardware.
I had a short and bitter fight with Unreal's GUI frameworks in May of 2016 and decided to not bother fighting with them. Instead, I decided to stick a web overlay on top of the game canvas and use web technologies for my interface. On the surface, it seems more complicated, but it turns out to be significantly simpler and more productive. It also means I can create really good and solid looking user interfaces.
Everything in my game is built to be networked and scalable. Over the summer I worked on pushing towards planetary sized worlds. Optimizing and reworking terrain generation, and pushing the limits of what can be done with today's technology. Eventually, I got terrain generation to a point where I could very quickly create terrain on the fly and travel at high speeds with fairly minimal lag. This is when I started to notice the limitations of my physics engine, more on that later.
I then started working on making the terrain modifiable in September of 2016. I was originally planning to make use of an embedded database engine but I quickly came to realize that a database just isn't fast enough. That isn't to say the databases I tested were not high quality, or that they couldn't scale. It was mostly the fact I wanted to make them store and retrieve data with extremely low latency. I also didn't need to do a significant amount of querying so most of the advantages of a database were not helpful. I ended up writing my own storage system which allowed me to handle high levels of concurrency with low latency by keeping fragments of data in memory and using an asynchronous event-based approach. It worked really well and was straightforward when it came to implementing replication across clients. Something I only partially implemented, enough to prove it would work but then something else caught my attention.
The physics engine was causing most of my headaches. Again, not the fault of the physics engine. It was doing a great job for what it was designed for. It just wasn't scaling well and caused a massive amount of headaches and complexity in my engine because I was having to delegate the game structures to something the physics engine could work with. I was also suffering from issues with high speeds and the occasional terrible collision response handling. I also had to write my own 'fix' for issues with tunneling because they solutions provided by the physics engine only worked most of the time. I could consistently break it.
I had written my own physics engine for a 2D game many years ago. I was able to handle 17000 objects colliding at 60 fps using a few strategies I developed. I discovered the names of some of the strategies a few years later. It sort of hit me, there are no 2D physics engines that could handle that many objects and I have doubts any will be able to in the near future. Part of the reason is they are generalized solutions and can't make certain classes of optimizations in the problem space. Their generalizability makes them inefficient, which isn't a bad thing, it just means they are only limited to micro-optimizations. Such as using SIMD instructions, GPU acceleration, newer hierarchy structures, or newer algorithms. Whereas if I wrote my own I could tailor it to my application and only make the physics engine do the work that it actually needs to. I would also be able to deal with high-speed collisions and tunneling without having to fight against the physics engine.
After reading a number of books on the subject, a few research papers, a thesis, and looking at various open source physics engines, I started thinking about how I could integrate a custom physics engine into my game engine. It was now December of 2016, but I wasn't ready to hook it up and swap out my existing physics engine interface. I wanted to start building a prototype physics engine. Very quickly I also realized that I would need a visual debugger to help me develop the engine and come up with test cases which I could use to verify correctness as I switch from naive implementations to heavily optimized solutions. I slapped together a quick visual debugger and was able to see my simulations. I also made sure I could easily step forward through time to make sure everything was behaving correctly.
It sort of bothers me. I originally wanted to use a bunch of frameworks and libraries to make my life easier and allow me to develop the game faster. Instead, I kept having to deal with limitations and find workarounds for them and noticed my productivity decreased. I do have the fortunate benefit of not having a deadline and so I can take the hit of writing my own solutions where appropriate.
This year has been a huge learning process, a year ago I would have never dreamed of writing my own physics engine or writing something instead of using a database for storage. I certainly would have thought by this point I'd have something playable instead of a sort of early tech demo. I guess that is the problem when you have to build your own technologies to solve problems others have yet to solve.
Well enough about the past, where am I going next?
Short-term (Q1 2017)
- [Physics] Expand capabilities of the physics engine, such as supporting more shapes
- [Physics] Deal with high-speed interactions and tunnelling
- [Engine] Switch from existing physics engine to custom physics engine
- [Graphics] Finish implementing client-side terrain updates
Medium-term (Q2-Q3 2017)
- [Graphics] Determine what type of art style to use.
- [Physics] Optimize custom physics engine including special handling for game
- [Gameplay] Start working on core gameplay
- [Interface] Revisit UI with updated technologies
Long-term (Q4 2017+)
- [Gameplay] Get playable demo
- [Graphics] Polish rough areas
