Category Archives: Games

Analysing Binary Files using xxd

When you’re trying to make sense of a binary file format, a good hex viewer or hex editor is an invaluable tool. As shown in “Ultima 1 Reverse Engineering: Decoding Savegame Files“, a typical workflow involves viewing a hex dump of the binary data, making some small change, taking a second hex dump, and comparing the differences. If you’re lucky, you might even be able to observe patterns in the data directly.

Tandy 1000 graphics data for the space fighters in Ultima 1 can be easily observed in a hex editor such as Okteta.

While a good visual hex editor (such as Okteta under Linux or xvi32 under Windows) is essential to view the binary data in hex and also make direct changes to the data files, a command-line hex viewer for Linux called xxd also exists. As with most things in a Linux environment, a lot of power comes from being able to combine command line tools in a way that produces the results we want. As we shall see, one of the benefits is that although most hex editors don’t have tools to compare hex dumps, we can leverage existing command-line diff tools for Linux.

Reverse Engineering the Ultima 1 Savegame File Format

We’ve already seen in “Ultima 1 Reverse Engineering: Decoding Savegame Files” how to analyse how the bytes in the Ulltima 1 savegame files change in response to actions in the game world. Let’s see how we could do this more easily using Linux command-line tools.

First, we start a new game, and take note of a few things such as the player’s position and statistics:

Ultima 1 when starting a new game.

We can use xxd take a hex dump of the savegame, which by default goes to standard output:

A hex dump of an Ultima 1 savegame file

By redirecting the output to a file (which we’re arbitrarily calling before.hex), we can save this for later comparison:

xxd PLAYER1.U1 > before.hex

Next, we move a couple of spaces to the right and save the game. Again, we take note of the situation in the game world (i.e. that we have moved to the right, and food has decreased by 1):

Ultima 1 after moving a couple of spaces to the right.

We can now take a new hex dump:

xxd PLAYER1.U1 > after.hex

Now that we have a record of the savegame before and after having moved, we can compare the two dumps using popular Linux diff tools such as diff or vimdiff:

vimdiff before.hex after.hex
Comparing hex dumps with vimdiff

In this case, simply moving two steps to the right has changed four different things in the savegame file: the player’s position, food, move count, and tile in the overworld map. It takes a bit more patience to reduce the number of variables at play and come to some conclusions about what the bytes actually represent, but you can hopefully appreciate how well these command-line tools play together.

Analysing The Savage Empire Dialogues

The Ultima 1 savegame is particularly easy to analyse and compare because it’s got a fixed size of 820 bytes, and each field in the game state has a fixed place within that file. Not all binary files provide this luxury.

For instance, the dialogues of The Savage Empire are LZW-compressed and are written using a proprietary scripting language. However, we can still use command-line tools to extract some interesting insights.

Using tools from the Nuvie project, you can extract each character’s dialogue into a separate binary file, numbered from 0 to 76 with some gaps. We can thus write a simple loop in bash syntax from 0 to 76 and invoke xxd on each file, using the parameters -l 16 to print out only the first 16 octets:

for i in $(seq -f "%03g" 0 76)
do
    echo -n "$i "; xxd -l 16 "path_to_dialogue_files/$i.dat"
done

The result is that we can identify an increasing NPC number as well as the NPC’s name and a part of their description within those first few bytes, indicating that although the structure of the data may be complex, there is still a deterministic pattern to it:

First few bytes of the first several NPC dialogues in The Savage Empire.

Conclusion

Whether analysing binary files using hex tools is your thing or not, I hope at this stage you can appreciate how much you can get out of combining a few simple command-line tools together.

Quick Mount in DOSBox under Linux

DOSBox is incredibly handy to run old games. In “DOSBox for Dummies“, I covered basic usage, as well as how to write a Windows batch file to automate some of the more repetitive operations (such as mounting). I also explained how to extend this to games requiring a CD in “Running Games Requiring a CD in DOSBox“.

If your games are all under the same folder, you might want to consider automatically mounting your DOS games folder using a dosbox.conf file. Otherwise, you can resort to scripting via batch files (Windows) or shell scripts (Linux).

For those one-off situations where you just want to try out a game quickly without setting anything up, regardless of where it resides on the filesystem, you can run the following (in Linux) from the folder where your game is:

dosbox -c "mount c $(pwd)" -c "C:"

This is the same method I used in previous articles to pass commands to DOSBox. The only difference is that here I’m taking advantage of command substitution in bash (as seen in “Scripting Backups with bash on Linux“) to pass in the current directory via the pwd command. That way, no matter where your game folder is on the filesystem, DOSBox will start in the right location. Then, all you’ll need to do is invoke the right executable.

Happy retro gaming!

The King who Forsook his own Virtues

Bias

As I write this, I can’t help but be conscious about bias. I’ve been a fan of the Ultima series of games since childhood. In 2001, I joined the Ultima Dragons Internet Chapter (UDIC) – an online fanclub dedicated to the series – and I’ve been a part of this community longer than I haven’t. In July 2002, I launched my first website, Dino’s Ultima Page, which was a leading site in the Ultima community for about a decade, and it will turn 16 years old in less than two weeks from now.

Left to right: Dr. Cat, Starr Long, Denis Loubet facepalming, and Richard Garriott at the Ultima Dragons Internet Chapter 25th Anniversary Bash

Last year, that same UDIC fanclub turned 25 years old, and a big party took place in Disneyland, Anaheim. I travelled all the way to California to be part of it, and like the rest of the people there, I was thrilled that several of the people who worked on the game – essentially our childhood heroes – were present to hang out with their fans.

The Kickstarter

There was similar enthusiasm a few years before that party, in March 2013, when Richard Garriott’s latest company, Portalarium, set up a Kickstarter campaign to fund a spiritual successor of Ultima called Shroud of the Avatar: Forsaken Virtues. The fans, starved for years of the creativity and entertainment by Electronic Arts (who currently owns the rights to the Ultima intellectual property), and sick of the failures it produced in an attempt to make money off its existing fanbase, readily poured their coin into a new game that would be made by some of the same people behind Ultima. The Kickstarter alone raised $1.9m, with additional funding secured after that.

Starr Long and Richard Garriott, speaking at the Ultima Dragons Internet Chapter 25th Anniversary Bash

Faced with this exciting prospect, what do you think a long-standing fan such as myself did?

I simply ignored it.

One reason was that I seldom had time to play games any more. But more importantly, it felt like madness to put money into a game even before it had started development, no matter who was involved. Coming from a country where customer service is abysmal, the last thing I’m going to do is give people my money to do whatever they want with it, without even being able to check some reviews first.

The trainwreck

In hindsight, I’m glad I did that. A recent lengthy review by taxalot at RPG Codex (with additional post-mortem insight by the author in the article’s comments) exposes the game as unfinished, buggy, and all round underwhelming in just about every aspect.

Most notable is that Portalarium tried to appeal to both the existing Ultima fanbase by promising a single player experience, while also going the MMORPG direction for those who wanted that.

“And sold they did. The first consequence of this was that if you backed the game for the single player experience… well, you probably gave up hope the moment your bank account was debited. To someone who was looking for a great single player adventure, the monthly emails focused solely on player housing were utterly depressing, an obvious sign that Portalarium had taken your money and were doing whatever the hell they wanted with it. Month after month, the studio unveiled new kinds of houses that you could buy with real money. But why stop at a house? Why not buy a castle? Or a whole town? You could do that too, as a solo player or as a guild to have your own place to regroup. The emphasis on this aspect of the game was truly puzzling. Between that and the monthly dance parties thrown by “DJ Darkstarr” (executive producer Starr Long’s alter ego), one might wonder whether the point was to have exciting adventures or just to create some sort of virtual renaissance fair for everyone to LARP in. In many ways, it felt like Portalarium were increasingly less interested in selling a game than a medieval Second Life service.” — RPG Codex Review: Shroud of the Avatar

Even more maddening is the concept of buying virtual houses with real money, and have to pay regular taxes on them. As if real-life housing weren’t bad enough – all we needed was to have the same problems in our games.

As you can imagine, this enraged several fans who backed the game based on the promise of Richard Garriott going back to his roots. One of these, who pledged $1500 for the game, was permanently banned from Shroud of the Avatar forums for questioning the direction of the project in this regard. He recently published the comments he was banned for, along with all the email correspondence that ensued, exposing what seems to be blatant abuse of power and excessive censorship.

The Future of Portalarium

While this whole mess is still unfolding, Portalarium laid off half their team just a few weeks ago, mainly laying off people in their art and design department. Which is ironic, because seeing that review on RPG Codex, it appears that these are the areas where help is most needed.

Meanwhile, in reaction to same review, Ultima Dragons have been discussing whether the resulting game is the fault of incompetent developers or incompetent management. While this is difficult to ascertain without having inside information, one may take a hint from the single Glassdoor review about the company (to be sure, a single review isn’t a very good sample, but it gives an idea):

The email correspondence about the aforementioned banning incident also rings alarm bells.

“It can also be hard to be confronted with your own misbehavior. In fact it can be so hard that many people, like yourself, cannot even face it and instead choose to focus on everything but your own actions.” — Starr Long, email correspondence

Given that this whole incident was a result of trying to stifle criticism, let’s just say I wouldn’t have been too happy to get this kind of response myself, especially from an Executive Producer.

History Repeats Itself

Shroud of the Avatar: Forsaken Virtues was fully released in March 2018 (even if in the pitiful state that the aforementioned review shows). That means it’s taken five years of development, and a whole lot of money. If you’ve been following the history of Ultima, you’ll find that it’s strangely reminiscent of Ultima 9, the last Ultima game that was released in 1999. Ultima fans generally consider the game to be a disaster, and often blame EA for the turnout.

Another thing EA is blamed for is the general fate of the Ultima intellectual property. After Ultima 9, there was pretty much no activity whatsoever for years. In more recent years, EA decided to reuse the Ultima intellectual property, resulting in a series of failures that were cancelled either even before being launched, or afterwards.

Ultima fans, for instance, generally agree that Lords of Ultima had nothing to do with Ultima other than the name. Ultima Forever: Quest for the Avatar similarly has a few names that fans will remember (including “Lady British”), but little else that feels familiar in terms of story or gameplay. This practice is called name-dropping, and guess what other game does this? That’s right. Shroud of the Avatar: Forsaken Virtues.

One would think that veteran game developers would learn from past blunders (theirs or otherwise), but after all this, the advice to management from that earlier Glassdoor review seems to hit the nail on the head.

Forsaken Virtues Indeed

Ultima 4 received critical acclaim because it brought ethics into an RPG genre that was principally dominated by “kill the bad villain” storylines. The virtues, conceived by Richard Garriott, would be central to all the mainstream Ultima games after that, except for a couple set on different words. Ultima 5, for instance, showed what happens when virtues are taken to the extreme.

“Thou shalt not lie, or thou shalt lose thy tongue.” — Ultima 5

If Shroud of the Avatar got nothing right, it has a great name. Forsaken Virtues very much reflects its overall direction. Honesty, for instance, was thrown out the window along with the Kickstarter promises. Compassion is shot down once you read the aforementioned email correspondence. Sacrifice is done by Portalarium only insofar as other people’s money and their own staff are involved.

As for humility, there are multiple aspects to this. One is that the game tried to be everything (scope creep anyone?), and thus failed to be stand out (or even be decent) in any one department. Another is that the top people behind the game need to get off their pedestal and start listening to their fans.

Simple Ultima-Style Dialogue Engine in C#

The Ultima series is one of the most influential RPG series of all time. It is known for open worlds, intricate plots, ethical choices as opposed to “just kill the bad guy”, and… dialogue. The dialogue of the Ultima series went from being simple one-liners to complex dialogue trees with scripted side-effects.

Ultima 4-6, as well as the two Worlds of Ultima games (which used the Ultima 6 engine), used a simple keyword-based dialogue engine.

In these games, conversing with NPCs (people) involved typing in a number of common keywords such as “name” or “job”, and entering new keywords based on their responses in order to develop the conversation. Only the first four characters were taken into consideration, so “batt” and “battle” would yield the same result. “Bye” or an empty input ends the conversation, and any unrecognised keyword results in a fixed default response.

In Ultima 4, conversations were restricted to “name”, “job”, “health”, as well as two other NPC-specific keywords. For each NPC, one keyword would also trigger a question, to which you had to answer “yes” or “no”, and the NPC would respond differently based on your answer. You can view transcripts for or interact with almost all Ultima 4 dialogues on my oldest website, Dino’s Ultima Page, to get an idea how this works.

Later games improved this dialogue engine by highlighting keywords, adding more NPC-specific keywords, allowing multiple keywords to point to the same response, and causing side effects such as the NPC giving you an item.

If we focus on the core aspects of the dialogue engine, it is really simple to build something similar in just about any programming language you like. In C#, we could use a dictionary to hold the input keywords and the matching responses:

            var dialogue = new Dictionary<string, string>()
            {
                ["name"] = "My name is Tom.",
                ["job"] = "I chase Jerry.",
                ["heal"] = "I am hungry!",
                ["jerr"] = "Jerry the mouse!",
                ["hung"] = "I want to eat Jerry!",
                ["bye"] = "Goodbye!",
                ["default"] = "What do you mean?"
            };

We then loop until the conversation is over:

            string input = null;
            bool done = false;

            while (!done)
            {
                // the rest of the code goes here
            }

We accept input, and then process it to make it really easy to just index the dictionary later:

                Console.Write("You say: ");
                input = Console.ReadLine().Trim().ToLowerInvariant();
                if (input.Length > 4)
                    input = input.Substring(0, 4);

Whitespace around the input is trimmed off, and the input is converted to lowercase to match how we are storing the keywords in the dictionary’s keys. If the input is longer than 4 characters, we truncate it to the first four characters.

                if (input == string.Empty)
                    input = "bye";

                if (input == "bye")
                    done = true;

An empty input or “bye” will break out of the loop, ending the conversation.

                if (dialogue.ContainsKey(input))
                    Console.WriteLine(dialogue[input]);
                else
                    Console.WriteLine(dialogue["default"]);

The above code is the heart of the dialogue engine. It simply checks whether the input matches a known keyword. If it does, it returns the corresponding response. If not, it returns the “default” response. Note that this “default” response could not otherwise be obtained by normal means (for example, typing “default” as input) since the input is always being truncated to a maximum of four characters.

As you can see, it takes very little to add a really simple dialogue engine to your game. This might not have all the features that the Ultima games had, but serves as an illustration on how to get started.

The source code for this article is in the UltimaStyleDialogue folder at the Gigi Labs BitBucket repository.

Announcing Ultima 1 Revenge

I am currently working on an engine port of Ultima 1: The First Age of Darkness, called Ultima 1 Revenge. This means I am reverse engineering the game files and building a new game engine for it, using C++ and SDL2.

Ultima 1: The First Age of Darkness was one of the first open-world Computer Role Playing Games (CRPGs). Originally released in 1981 and remade for the PC in 1986, Ultima 1 was followed by a series of games that lasted almost 30 years, generated a cult following, inspired countless other RPGs, and pushed the boundaries of technology.

Ultima 1 is a fairly weird game, featuring an unusual combination of medieval fantasy and space travel. The world of Sosaria is being ravaged by the monsters of the evil wizard Mondain. Before you can face him in battle, you have to complete dungeoneering quests in the service of the lords of the land, become a space ace, free a princess, and travel back in time using a time machine.

The 1986 PC remake, on which the Ultima 1 Revenge project is based, is very old technology, by today’s standards. Still, it provides a vast array of learning areas. The game’s graphics are made up of three tilesets (CGA, EGA, and Tandy 1000), giving a choice for the differently powered machines of the time. The game world is stored in a small map file, where each four bits is an index into the tileset you’re using. Space travel is a combination of top-down 2D and first-person views. The dungeons are simple 3D-like line drawings, randomly generated based on a seed stored in the savegame file (so they remain consistent for each playthrough, but change if you start a new game). The different parts of the game run in different executables, and a special savegame file is used to pipe the player state from one to the other. Savegames mostly use 16-bit numbers, with the least significant byte stored first. Decoding the game files is an ongoing effort that powers tools such as the online map viewer I built in 2015, and the engine itself.

Today, I have released a demo of the engine. So if you own a copy of Ultima 1 (if you don’t, you can grab a copy from GOG), grab it from the downloads page, set the path to your original Ultima 1 folder in the settings file, and take a tour of Sosaria!