Save Files

Sorry, no new binary release. The last one can be found here.

Sorry, no new source release. The last one can be found here.

Save files are easy to implement, but quickly becomes more difficult the longer you think about it.

Directory

The simplest solution is to create save files somewhere in the same directory as where the executable resides. Unfortunately that would require admin privileges when installed in the Program Files directory, not to mention the annoyance this would cause on multi-user PC's. When our game is installed, the best location would be somewhere in "C:\Users\ [username] \AppData\Local\", unless we create a self-contained "extract-and-run" kind of game of course.

Our application also allows for multiple versions running side-by-side on the same computer and it would be bad if they all used the same save-directory, in case we are changing stuff in the file-layout (which should always be assumed).

If self contained:

• [ExecutableDirectory] \Saves\

And if installed:

• [LocalAppData] \ [CompanyName] \ [ProductName] \ [Version] \

Version Management on Disk

Even though we are assuming that we are continuously modifying the used file structure for saving data, there is no reason to throw away save files every time we're creating a new version.

Backward compatibility

So this is what we are going to do:

• When using a new version, we can still open old save files.
  I want to achieve this by creating shortcut-like files.
  • [SaveFileDir] \v2.7.2.0.0\MySaveFile.dat (2 MB, the actual save file)
  • [SaveFileDir] \v2.7.2.1.0\MySaveFile.ref (1 KB, the shortcut to the actual save file in the v2.7.2.0.0-directory)
• When a save file has been opened from an older version and saved, we can't just overwrite the save file in the directory of the old version. The old version might still be installed and maybe can't open the new save file.
  • [SaveFileDir] \v2.7.2.0.0\MySaveFile.dat (2 MB, the original save file)
  • [SaveFileDir] \v2.7.2.1.0\MySaveFile.dat (2.1 MB, the new save file with new file layout)
• Deleting a save should be the same, we may not modify files from older versions.
  • [SaveFileDir] \v2.7.2.0.0\MySaveFile.dat (2 MB, the original save file)
  • DELETE: [SaveFileDir] \v2.7.2.1.0\MySaveFile.dat (2.1 MB, the new save file, deleted)
  • [SaveFileDir] \v2.7.2.1.0\MySaveFile.nil (1 KB, a file indicating that the save file does not exist and we don't need to look in other directories)

Accessing the save files in game-code should still be straightforward and one-dimensional process, but all this complexity can be abstracted away without too much effort. The interface to the save-files can just pretend there is but one save-file directory.

Save File Datastructure

We could use .NET's serialization mechanism for storing data, but that creates a lot of overhead and by a lot i mean A LOT. Even binary serialization somehow has an enormous amount of overhead, probably because they store full type-names to make serialization easier with compatibility issues. Since we need a method for both writing data to files and a method for sending data over a socket (for future multiplayer stuff), I'll think up my own method for storing data. Having full control over "structured_data --> bytes" and "bytes --> structured_data" also makes it easier for any future ports.

Not 30 minutes after I implemented the whole shebang, I thought:

Okay nice, I can create and read my own file types. Now let's try to read a file that has been made by some other application. Let's try and read the Dwarf-Fortress map files! Yeah! That would be awesome! Yay!

Unfortunately dwarf fortress is closed-source (I would have loved to take a look under the hood), but luckily the tool DFHack is opensource and contains the function "mapexport". It uses a couple of proto-files like this one, which was new to me. Apparently is uses Google's Protocol Buffers to precompile the proto-files to C++ code. So the file "Map.proto" is precompiled to "Map.pb.cc and Map.pb.h" and gives the programmer access to a class 'Map' that can be serialized over a stream. That's just awesome, it is almost exactly what I wanted. So yeah, I implemented something for which a solution already existed, so I thought:

Okay, my implementation of things is kind of redundant now. Google's protocol buffers is really nice and probably tested thoroughly. Let's use protocol buffers to read a dwarf-fortress file so where is the c# version? Wait only C++ and Java? *********! Ah well, let's just implement protocol-buffers ourselves.

I didn't want to use the precompile-method like Google's Protocol Buffers does, because it gives our project an additional compile-step. In C# it's actually unnecessary because there is a real good alternative, but it does require a bit of work. It requires the use of attributes, reflection and interop-language.

Our version of Protocol Buffers

My implementation of protocol buffers will not work with proto-files, so it will not be a simple port. Instead of a proto-file like Map.proto, I'll create a struct like this:

And to read a Map from a stream:

or to write it to a stream:

We don't want to use reflection every time we are serializing or deserializing an object (because reflection is slow), so we are only going to read the attributes once. We will just 'generate' a function for serialization and deserialization. This can be done by using the interop-language of .NET. I don't want to talk about details of that stuff, because well, it's kind of boring (the real fun is in doing it yourself). I don't mind giving an impression of how that code looks like though:

It's a little bit like assembly so if you played around with masm or tasm, it shouldn't be too difficult to read. Finally I was able to read the dfmap-file by creating a couple of struct's and a very simple test-application:

Yay, I'm ecstatic! However, not 30 minutes after I implemented the whole shebang I found Protocol Buffer for .NET. Hehe, whoops. Something like this already existed. Ah well... at least I have my own implementation which is way cooler by definition. I extended my version of protocol buffers to allow 8 bit values (byte/sbyte) and 16 bit values (ushort/short), something that has been requested as a feature for google's version but has been ignored by the developers.

Backward and Forward Compatibility

Even though Protocol Buffer allows backward compatibility (adding properties to existing struct's) and allows forward compatibility (skipping values from the stream that we do not have a defined property for), we still need conversion-functions. We only need to do this for save files because for network-functionality we will enforce the same version of the game.

For each save file we will need to have a magic-string to identify the file and a file-version. The file-version will be the version when we last modified the structure of the file in such manner that we cannot rely on Protocol Buffer's methods for conversion. To give an example for serializing a highscore-object in our 2.7.2.0.0 version:

Backward Compatibility

Now let's say that for some weird and strange reason we want to change the type 'long' to 'string' in some future version. Yes I know, weird right? But the developer has set his mind to it, so let's see what he's going to do.

• He copies HighScore.cs to 'HighScoreOld_v2_7_2_0_0.cs' and renames "HighScore"  to "HighScoreOld_v2_7_2_0_0" in this new file.
• In HighScore.cs he changes long to string and changes "[ProtocolMessage("highscor", 2, 7, 2, 0, 0)]" to "[ProtocolMessage("highscor", 2, 7, 3, 1, 0)]" (or whatever version he's at).
• Next he makes a conversion function between the two version and registers that function as follows:

When registering a conversion functions our engine will make a graph. The graph for this example is really simple:

When we are currently trying to load a v2.7.3.1.0 version, but instead we finds a v2.7.2.0.0 version the engine will try to find the shortest path in this graph to find all the conversion functions we need to apply to get to a v2.7.3.1.0 version.

Forward Compatibility

Forward compatibility is a little bit more difficult. How does an old version of the game even know the data-structure with the string-value? There is just no way. Ha! Fooled you! There is a way!  Plugins!  

If our engine loads plugins and these plugins contain conversion-functions for  forward compatibility then tadaa. So our software-engineer that decided string are better, creates a forward-compatibility-plugins. He might not be the brightest, but at least he's a nice guy. We just make it so that every DLL in the Plugins directory gets loaded and that every class in that DLL with a "[Plugin]" attribute gets initialized. So to continue the actions that our developer takes:

• Our developer makes a class-library project (for creating a DLL), and copies the new HighScore.cs to that class-library (without the conversion function we created for backward-compatibility).
• He renames the file and class to "HighScoreNew_v2_7_3_1_0".
• He creates a new Class named "PluginEntryPoint" with the attribute "[Plugin]" and creates a forward-compatibility conversion function.

The DLL that this project output's can be used by players with an old version that wants to read a save-file from a newer version. Just by putting the DLL in the Plugins directory makes this happen. Totally awesome!

Next Time

I'm thinking about releasing v2.7.2.0.0 with the next post. I'm not 100% sure yet, but I'm optimistic. Hmmmm, haven't used any pictures this post. Pictures of code aren't real pictures. Lat's just add one big picture to make up for all the text:

Game Libraries

Programming Phase 2