White Shining Rock Logo
White Shining Rock Logo

Save and Load

January 13, 2021 | Dukus | 31 Comments

Does anyone want to know how save games work in video games? Yeah! of course you do! I've spent the last two weeks working on it. In a simulation game you can't get very far testing things without being able to save. I suppose you can, but you'd spend a lot of time recreating the conditions you need to test a new feature or fix a bug.

The basics of saving are pretty easy. You write out all of an objects properties. What model and materials is the object using? Does it have collision and pathing information? Does it store inventory? What animation is playing? What plan has the AI made? When you read it back in, you just take that data, and setup the object again based on what was stored.

Then it gets harder. Let's say there are several hundred objects. And some of them reference each other. An character might have a target that it is following. When the game is running, this is easy - that character stores the physical (well, technically virtual) address in memory of the item it needs to reference. But when the game goes to write that out, I can't write the physical address. It's meaningless. Because the object is probably going to be in a different place in memory when it's recreated on load.

So I have a system where each object is assigned an identifier. It's just a unique number. The first time an object is written out to a save game, the full object is written, and then its identifier. After that, if the object is referenced again, just the identifier is written. When the save is restored, the object is read in, and while the object gets a new unique identifier, the previous identifier is stored in a dictionary with the objects new address. If references are encountered, the identifier can be looked up and exchanged for the new memory address of the object.

Alright, so now save game works! Well, that was it for my last game. But this one is slightly more complicated, and consists of the bulk of my recent work.

Since the game is going to be open world-ish, saving is harder. The first thing to consider is that when the player is far away from some regions, they don't need to be simulated. So when they go out of range, they get removed. But whatever the player has done in the area needs to be saved. So the same save process happens, but just for that area.

A bit of care needs to be taken here, because to save a group of items together, they can reference each other, but they can't reference things outside the area. So the region is disconnected from the rest of the world. I have some safety code that ensures the disconnection - if something goes to write to the save that isn't in the region, that's a fatal error and needs to be fixed before I can proceed.

The game also can't pause to do these types of saves. So after being disconnected, the save process happens in a background thread. This save data is just kept in memory, rather than being written to disc.

The loading process for this is similar. A background process loads the needed resources, all objects and references are restored, and then the region is connected to the rest of the world and the player can then interact with it. This happens hopefully before the player gets near the area so the world appears continuous. There's some safety code there as well - if the player somehow gets too close to the area before the load and restore is complete, a message appears that loading is occurring and the player has to wait.

So when a normal save comes along, all active game play regions undergo the same process as above, anything remaining goes through the normal save process, and all the chunks of memory that were saved for each region are written to disc.

This whole process took a while to get right, and to get all the bugs worked out. One single piece of information not saved, and really strange things happen when a save game loads. For example, I had a bug where a save game would load and all characters would face east and quickly turn to the direction they were supposed to be facing. I had just failed to save out the current direction they were facing. Another example - I was dealing with objects being duplicated, since multiple areas were referencing the same object. That was before I added the safety checks for objects not being written if they were in different areas.

Dealing with references was tricky too. Since objects depend on each other, loading has to be done in three passes. The first pass restores values for each object. That's generally easy. However, there are internal dependencies in an object. Graphics are totally separate from animation which is separate from audio, but a logic controller might reference all three. Since the graphics component might not be loaded before the controller, anything to configure that uses those dependencies has to be delayed. That's what the second pass is for. Finally a third pass is made once all objects are setup so that external references can be dealt with. Like making sure that an animation of a bow pointing at a specific target can be setup to point that way and the initial animation is in the right pose.

Once save game was working properly, I then wanted to deal with space requirements. The image below has 9363 objects placed on it. You can't see all of them, since small ones disappear when they are far away, but they are there.
Picture of island with many trees
Saving all of them requires 3.7 megabytes! That's around 400 bytes per object! yikes! It seems like a lot, but each object is storing a lot of gameplay data you don't visually see.

Each object has some overhead making them future proof. Each section of an object, (graphics, audio, collision, animation, ai, etc) has a header and footer, version number, and size. This allows me to make sure when the data is read back in everything is read correctly. It allows me to make additions in future versions, read the old data, and setup any new data. It allows me to skip over any sections that aren't needed anymore. It gives me safety when I change code and forget to update the save function - i'll get a fatal error that the code doesn't match the data written to disc.

This large size doesn't seem a huge deal, until I consider that a player might visit 100 such places. I dont want saves to be 370 megabytes. A player might not go that many places or the save might not total that large, but I need to deal with worst case.

My first fix was to apply a quick LZ compression. Since the trees and rocks save data are self similar, there's a lot of data repetition, so that takes it down to 731 kilobytes. That's 5 times smaller, but still not great, and real game data will have much more variety. I've got some Huffman/Arithmatic encoding I can do as well that brings it down another 50% or so, but I had a better idea.

Items placed on the map are known ahead of time. Even if they are procedural, they can be computed. But a player can interact with every one of them, removing them or changing them, requiring them to be saved. So what if instead, I track if a player has modified one of them. Say cut down one tree. Then I can just procedurally generate all but one of them, skippng over the one that was modified. I can store a single bit per object, which tells if the object is placed as it was on map creation, or it's been modified or removed.

If the tree is in the middle of being cut down it's saved as a full object. If it's gone it costs nothing but 1 bit. If it's there it costs 1 bit. All the sudden 3.7 megabytes becomes just over 1 kilobyte, plus a little for any object that is being interacted with.

As development continues I plan on extending this system a little further. I can precompute objects that aren't there initially. For example if you can plant a forest of trees on a plain, I can have the positions precomputed, but the objects not added. When a tree is planted later, it uses one of the available positions. For a while, the tree grows and is stored as a full object. And when it's reached full growth, the bit is changed to have the tree be there on load, and it costs nearly nothing in save space.

This way the majority of the simple objects in an area cost nothing to save. The save game size will be determined as just what the player builds and by the characters that are walking around. Certainly if a player builds 1000 buildings save games will be bigger, but it's unlikely they will do so across many different places.

Saving is one of those things that's so important, and no one ever notices if it's working like its supposed to. So now saving is out of the way, except for nice menus and interface. Hopefully next time I'll be talking about the beginnings of my new AI simulation.

Leave a Reply

31 comments on “Save and Load”

  1. Always thought about my large save files in factorio, esp as I like keep multiple copies.
    Thanks for your progress thoughts, it's fun to read this, even if I'm not a programmer.
    Looking forward to what this finally ends up as which game.

  2. My current save in No Man's Sky is 1.7 Gig. I guess that's not bad for an entire universe - lots of regions missing, etc.

  3. Wow. I've learned a lot reading this. I've always wondered why some games have humongous save files, now turns out they are just not optimised enough.

    I know that we're only talking about the terrain for now. Im curious how massive it'll be when items, buildings, animals, history/statistics were to be included.

  4. Thanks for the insight. I am also starting development of a game with procedural vegetation and player harvesting and construction so a lot of this very relevant!

  5. My GF plays Banished for hours on end and even ends up messing up her sleep schedule because of how much she plays it. Happy to see you working on something new. I love telling her about these updates.
    Thanks for being active.

  6. This was very interesting. I was a programmer many years ago and now I’m a nurse. Programming is still in my blood - miss it!

    Love that you’re still working on this game. I have thousands of hours in Banished. It’s a little embarrassing how much I enjoy my time there. Haha! In fact, think I’ll go play it now!

  7. Wow that does sound challenging. Since you have to procedurally generate everything on load, does that take substantially longer?

  8. Very interesting read, thanks for posting. I was thinking about how a save system could efficiently be implemented (I am a starting programmer).

    So this came at the perfect time! Thanks for sharing, my thought experiment can now be finished very satisfyingly.

  9. Supreme Commander devs wrote some good diaries about this sort of thing. They made their game deterministic (easier said than done) so a save is actually a log file of inputs and loading it is playing back those inputs in fast forward to arrive at the same game state you left off.

  10. Really enjoy seeing these updates over the years pop up in my email. I’m sure there are many who silently watch and read, so keep it up!

  11. Paradox interective games (HoI 3,4 Victoria 2, EU4, Stllaris, CrusaderKings 2,3) can make saves about 3-8 mb, also they use "outdated" engine. Sorry for my bed england

  12. As @CharleyDeallus says, I'm one of surely many who silently watch and read for many years. It's a great pleasure to read how you tackle dev challenges. I found this post on saving data particularly interesting. I hope you'll keep your excellent work up for many more years. And I'm eager to play this new game. A happy new year to you and to this faithful community!

  13. Fantastic post, I actually used a lot of similar techniques in a chunk based game I am working on. Keep up the great work and keeping posting more we love it!!

  14. Hey there, I'm the creator of the game Prosperity, also a city building game. I've come across similar problems you've faced, although I think 3 mb is fine for save game files.

    One thing I prioritize is that saves must be fast to load and write, and doing things like Hoffman and LZ takes valuable CPU time, even if it runs in separate thread. A technique I use which may not apply to Banished is that for the most part I didn't use OOP, and instead kept mappings of objects in memory. This enforces reference by ID, and ensures that we can write to files exactly the maps data in order.

  15. Ahh, the joys of saving and loading a state.
    I don't really have much experience in game dev area, but having to work on various apps that allowed saving preferences + windows layout + report state for each window (query, filters, sorting etc) I can tell you the most problematic thing is keeping it all backwards compatible. Try ensuring a save file from 2 months ago still works when you've added new types of windows, new buttons, new columns, modified some existing features is quite often harder than adding all those new things:)

  16. "Does anyone want to know how save games work in video games? Yeah! of course you do!" ..Actually yes I do want to know!

  17. If you've taken the time to write it, I'll want to know about it. 🙂 I have always wondered about the finer details of saving information and how small save games often are (when you think about it, most games have HEAPS of information they need to store). After reading this I might find a few articles on procedural generation as that is also something I've always been curious to know about.

    All the best for the continued development of your next game and keep up these dev logs where you can. 🙂

  18. I am very curious how much you learned since you developed Banished. Is this a technique you found later?
    Still a bit confused on how you can compute where an object is placed, when you don't know what the player is going to do.
    But very cool that it's just a couple of kb, though I think that in these times it isn't really important. What's most important is load speed!

  19. I am always happy to read about your progress and I am looking forward to the article about AI.

  20. A really interesting explanation to read Luke, thanks for sharing! I planned to read this item when I had time, i just did and I wasn't disappointed. 😀

  21. Thank you for writing some fantastic articles. I really enjoy reading your open, well thought out posts.

  22. Hey there,

    my Banished-time was long ago (but really intense :D) and I recently stumbled on a very similar game on steam and checked if there is anything new to the Banished universe. This is how I found your blog article here and even if I'm not a developer, I'm really looking forward to news from your studio and I'm very curious about what's possibly coming in the future. Keep up the good work and thanks for the wonderful game you gifted us.

  23. you are a visionary, you know how it should go and the pathway, but you're a teamleader without a team. you need code-monkeys my man. Banished was a better game than all the polished AAA studios copies of your brand and ideas, but only because of the work put in from the few guys at colonial charter who basically were your free help. imagine if you'd have gotten them on board and made a real version without losing weeks of time to this bug or that, when an employee could do it for you in half the time. im still pissed you never finished the osx/linux versions too - i mean just pay a guy a flat rate for a port, sell it on steam for 2x. just stop burning yourself out.

    anyways your new ideas and direction sound good, screenshots are great. less rigidness of the AI townfolk seems like a great idea, i think everyone that got into banished liked the macro "Ant Farm" aspects of it, but also being able to spy at the micro when it fancied you with the follow tool.

  24. Hey man, i hope you will update Banished which is the best simulation town builder game in steam (stats really verify it).Or recreate a game like banished! Keep up the good work!

  25. So glad you're still at it. Please release something soon. I'm old now. I don't have a lot of waiting time left...

More Posts

UI Redo: Part 3

October 25, 2021
1 2 3 47
Back to devlog
Back to devlog
© Copyright 2021 Shining Rock Software
Website Design & Branding by Carrboro Creative
menu-circlecross-circle linkedin facebook pinterest youtube rss twitter instagram facebook-blank rss-blank linkedin-blank pinterest youtube twitter instagram