Category Archives: Software

Block Selection and Column Editing

We’re all very much used to selecting text by clicking and dragging the mouse. But by pressing the Alt key while doing that, you can select a rectangular block. This feature has been around since Visual Studio 2010 – part of it even since Visual Studio 2008 – and it’s available in most modern text editors such as Notepad++. However, most people seem not to be aware of this, which is why I’m writing this article.

Let’s say you created a new Console Application in Visual Studio, and added a few variables within the Program class:

    class Program
    {
        int name;
        int age;
        int address;

        static void Main(string[] args)
        {
            
        }
    }

Oops. Main() can’t access them, because it is static, and they are not. We’re going to have to make them static as well.

Now we can add the static keyword to each variable, one by one. Or, we can place the cursor before the first int, press Alt, click and drag downwards to create a sort of blue cursor that spans multiple lines. It’s a bit hard to see, so I’ve zoomed in a bit here:

column-editing-1

With that, we’ve enabled column editing. This means that whatever you type will now be written in multiple lines:

column-editing-2

You can use this to comment lines in bulk (similar to the Ctrl+K+C or Ctrl+E+C shortcuts, depending on your editor settings):

column-editing-3

Now, column editing is actually a special case of block selection with a width of zero. To see how block editing works, let’s change our variable names to the following:

        static int personName;
        static int personAge;
        static int personAddress;

Now, due to changing requirements, we decided that these shouldn’t be called person*, but customer*. Given that the variable names are nicely aligned underneath each other, we can press Alt, click and drag around person on all three lines, and we’ve made a block selection:

block-selection-1

Press Backspace to remove person from all three lines. The block collapses to zero width, so we’re back to column editing, and we can now easily write customer on all three lines:

block-selection-2

So there you go. Block selection and column editing are nothing new, but they’re very handy and good to know about.

A Gentle Introduction to Gulp

We’re at the end of 2015, and web technology has changed quite  a bit since I started in 2002. Nowadays, for the front end stuff, there is a whole family of tools based on the node.js package manager (npm) that you can use to streamline and automate your workflow.

In this article (based on Windows), we’ll learn to use Gulp to do routine tasks such as concatenating and minifying JavaScript tasks. There’s another tool called Grunt with a similar purpose, and you’ll find all sorts of discussions on the internet comparing Grunt vs Gulp. Basically, Grunt is the older of the two and has a bigger community – an important factor considering that these tools are plugin-driven. However, I’m covering Gulp here as I felt it was more intuitive. For this small demonstration it has all the plugins we need, and performance (a common point of comparison) isn’t even a factor.

Setting up Gulp

The first thing we need is to install node.js:

install-nodejs

There’s a chance you might already have node.js, if you installed it with Visual Studio 2015.

Once you have installed node.js, you should have npm in your path. Open a command prompt, and install Gulp using the following command:

npm install gulp -g

-g means globally, and thanks to this, gulp should now be in your path.

Next, we want to create a package.json file. This is a kind of project file for node.js-related stuff. We can use npm for this too:

npm init

npm will ask a bunch of questions in order to set up the package.json file, suggesting possible answers where it makes sense to do so. name and version are required, but you can leave the rest empty if you like:

npm-init

Next, we need to install Gulp locally in our project:

npm install gulp --save-dev

This installs Gulp; –save-dev updates the package.json with a devDependencies field:

{
  "name": "gulptest",
  "version": "1.0.0",
  "description": "Learning to use Gulp.",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "Daniel D'Agostino",
  "license": "ISC",
  "devDependencies": {
    "gulp": "^3.9.0"
  }
}

Plugins and the Gulp file

Gulp itself doesn’t do anything; it is just configured to run tasks. Its capabilities come from the plugins you install, and you configure it to do stuff using a Gulp file. For this simple example, we’re just going to use a few plugins:

npm install gulp-concat gulp-uglify --save-dev

Once again, –save-dev updates your devDependencies in package.json:

  "devDependencies": {
    "gulp": "^3.9.0",
    "gulp-concat": "^2.6.0",
    "gulp-uglify": "^1.5.1"
  }

Next, create a file called gulpfile.js, and put the following code in it:

var gulp = require('gulp'),
    uglify = require('gulp-uglify'),
    concat = require('gulp-concat');
    
gulp.task('default', function() {
  return gulp.src('js/*.js')
    .pipe(concat('all.js'))
    .pipe(gulp.dest('dist/'));
});

To test this out, I downloaded jquery and jquery-ui, and put the uncompressed Javascript files in a “js” folder. Having created the Gulpfile above, all you need is to run Gulp:

gulp

You should find a folder called dist, with a file called all.js in it, containing the contents of the files originally in the js folder:

gulp-concat

Concatenating JavaScript is good for performance because the browser only needs to make a single request, rather than having to retrieve several small files. But we can do even better by minifying the JavaScript (using the gulp-uglify plugin). Just add the following line:

var gulp = require('gulp'),
    uglify = require('gulp-uglify'),
    concat = require('gulp-concat');
    
gulp.task('default', function() {
  return gulp.src('js/*.js')
    .pipe(concat('all.js'))
    .pipe(uglify())
    .pipe(gulp.dest('dist/'));
});

Run Gulp again, and you’ll find that all.js has been updated. In fact, it’s much smaller now, and it’s completely illegible:

gulp-uglify

Conclusion and Further Reading

The purpose of this article was to get you set up with Gulp, and see something working with the least possible hassle. Mark Goodyear’s article (on which this article is partly based) covers a lot of other common operations to carry out with Gulp. If you need to do anything particular – linting your JavaScript files, minifying your CSS, using Less, etc, there’s probably a plugin for it.

Beyond that, all you need to know is how to use Gulp effectively as part of your build process.

  • Running Gulp without arguments makes it look for the “default” task. You can pass the name of a task to run as an argument, allowing you to run a variety of operations.
  • How do you debug your minified JavaScript? You don’t. Use separate tasks for development and for release, and minify only in your release task.
  • Ideally these tasks should be run automatically as part of your continuous integration.
  • An ASP .NET 5 (formerly known as vNext) project in Visual Studio 2015 can easily integrate with npm tools, and you can configure it to run your tasks when you build.
  • Not using Windows? These command line tools are easy to use on other platforms (although installing npm will obviously be different).

Update 8th January 2016: Check out “More Gulp in Practice“, the followup to this article.

Pasting JSON/XML as Classes in Visual Studio

Note: This feature is available as from Visual Studio 2015.

REST and JSON together have revolutionised the way we expose and consume data in recent years. For this reason it has become increasingly common to need to integrate with REST services, and data itself is provided in either XML or JSON format. This typically involves creating a set of classes that match the data provided by the service, so that the application may take care of serialisation and deserialisation, and work with structured data.

Although you can create these data classes manually, Visual Studio provides a shortcut to generate classes from XML or JSON data. For instance, let’s say that we want to work with the data from the GitHub API:

github-api

We can easily generate a class for this by copying that data, and in Visual Studio going into Edit -> Paste Special -> Parse JSON As Classes:

paste-json-as-classes

And as if by magic, the following class gets generated:

        public class Rootobject
        {
            public string current_user_url { get; set; }
            public string current_user_authorizations_html_url { get; set; }
            public string authorizations_url { get; set; }
            public string code_search_url { get; set; }
            public string emails_url { get; set; }
            public string emojis_url { get; set; }
            public string events_url { get; set; }
            public string feeds_url { get; set; }
            public string followers_url { get; set; }
            public string following_url { get; set; }
            public string gists_url { get; set; }
            public string hub_url { get; set; }
            public string issue_search_url { get; set; }
            public string issues_url { get; set; }
            public string keys_url { get; set; }
            public string notifications_url { get; set; }
            public string organization_repositories_url { get; set; }
            public string organization_url { get; set; }
            public string public_gists_url { get; set; }
            public string rate_limit_url { get; set; }
            public string repository_url { get; set; }
            public string repository_search_url { get; set; }
            public string current_user_repositories_url { get; set; }
            public string starred_url { get; set; }
            public string starred_gists_url { get; set; }
            public string team_url { get; set; }
            public string user_url { get; set; }
            public string user_organizations_url { get; set; }
            public string user_repositories_url { get; set; }
            public string user_search_url { get; set; }
        }

Now, the class and property names might not be exactly the way you want them, and you might need to change some things, but this will still save you a lot of typing.

If you’re still not convinced, try it with a more complex example that involves nested data structures. For instance, let’s say we have this JSON data:

{
	"name" : "John Smith",
	"summary" : "Wannabe Superhero",
	"workexperience" : [{
			"title" : "Insurance Guy",
			"company" : "ABC Ltd.",
			"start" : "Feb 2015",
			"description" : "Current job, very bad"
		}, {
			"title" : "Cleaning Guy",
			"company" : "Super Clean Ltd.",
			"start" : "Jan 2013",
			"end" : "Jan 2015",
			"description" : "Dirty work",
			"recommendations" : [{
					"name" : "Boss guy",
					"position" : "Boss"
				}, {
					"name" : "Colleague girl",
					"position" : "Cleaning Girl"
				}
			]
		}
	]
}

With no effort whatsoever, you can get these classes generated for you:

        public class Rootobject
        {
            public string name { get; set; }
            public string summary { get; set; }
            public Workexperience[] workexperience { get; set; }
        }

        public class Workexperience
        {
            public string title { get; set; }
            public string company { get; set; }
            public string start { get; set; }
            public string description { get; set; }
            public string end { get; set; }
            public Recommendation[] recommendations { get; set; }
        }

        public class Recommendation
        {
            public string name { get; set; }
            public string position { get; set; }
        }

And there’s a similar function for XML.

You’re welcome.

Running Games Requiring a CD in DOSBox

Back in 2009, I had written DOSBox for Dummies, a short and simple article explaining how to get old games running in DOSBox, and how to write a batch file so you don’t have to do this every time. A typical batch file might look something like this:

dosbox -c "mount c C:\prophet" -c "C:" -c "prophet"

That will often be good enough. But if your game needs to access resources on CD, such as Ravenloft: Stone Prophet in this case, then that’s not quite going to work:

dosbox-no-cd

To allow the game to access the CD, you’ll need an extra command to mount the CD drive:

-c "mount d D:\ -t cdrom"

The whole thing looks something like this:

dosbox -c "mount c C:\prophet" -c "mount d D:\ -t cdrom" -c "C:" -c "prophet"

That’s much better:

dosbox-ravenloft-stone-prophet

If you don’t want to pop in your CD every time you want to play, just copy the CD onto your hard disk, and mount that folder instead:

dosbox -c "mount c C:\prophet" -c "mount d C:\STONE_V1.0 -t cdrom" -c "C:" -c "prophet"

That works just as well.

On Mystery Meat Navigation and Unusability

This article was originally posted here at Programmer’s Ranch on 12th September 2013. It has been updated since a lot of the original examples are no longer available.

Hello folks! 🙂

Today I’m going to talk about Mystery Meat Navigation (MMN). This term was invented 15 years ago by Vincent Flanders of Web Pages That Suck. It refers to a horrible practice of filling a website’s navigation with meaningless icons. When a user moves his mouse over one of these icons, the icon changes or pops up some text, revealing what it really does.

A classic analogy of mystery meat navigation (which seems to have mostly disappeared from the web) is a road sign that initially looks completely blank, but changes to indicate where you’re going just as you drive past it.

Even now, 15 years later, MMN is still used on the web. Even reputable web design companies here in Malta have fallen in the MMN trap. Alert eBusiness, for instance, used to have the following navigation bar:

alert-mmn

Right, so what do those icons mean? The last one seems pretty clear: a shopping cart. Mousing over it reveals it stands for Alert Payment Gateway, which is close enough. But what about the rest? The first one is a mouse, for instance. Would it ever cross your mind that it actually means “Web Design”?

Another example: Pcionix (now defunct):

pcionix-mmn

The home icon is pretty obvious, so that can be forgiven. But a pie chart that stands for SEO – seriously?

But this, from design.com.mt, is even worse:

design-mmn

This has got to be the worst of them all. Whereas you might be able to somehow guess what the icons in the other sites mean, the navigation here is hidden behind meaningless numbers that you again have to mouse over to understand.

It gets worse: there are videos on YouTube of sites with iconic navigation that actually floats around, so you actually have to find out where that “About Us” cube thingy moved to (examples: Mandarina DuckQualcomm).

So why is MMN bad? In case it isn’t obvious, it is very annoying for users to have to click on stuff to figure out what the page offers. A website should give a clear indication of how it is structured, without the user needing to interact with it just to get an idea. Imagine you’re driving and need to interact with a bunch of direction signs (such as these) one by one to get an idea of the places in the area. Then, after sifting through a dozen, you forget what you saw earlier and have to go back and interact with them again. Sorry, the “just a click away” idea is not an excuse when it comes to navigation, which is really a website’s backbone.

Another great example comes from feedback that Vincent Flanders received, and illustrates how MMN would be if applied to a business’s answering machine:

“You’ve reached XYZ Corporation. To find out what option #1 is, press 1. To find out what option #2 is, press 2. (Etc….) If you’d like to continue doing business with our company after we’ve slapped you around and wasted your valuable time, press 9”

MMN is a slap in the face of usability. It shows meaningless icons in the place of important navigational information. What could possibly worse?

The only thing worse than showing meaningless icons is not showing any icons at all! That’s pretty much the direction taken by Windows 8’s notorious alternate UI, formerly known as Metro. One of its design principles is “Do more with less” which includes “Put content before chrome”. In this case the “chrome” refers to the stuff that makes the application – menus, the ‘X’ button at the top-right, toolbars, etc. So basically you end up with something like this:

ubuntu-pdf-metro

That’s the default PDF viewer on Windows 8 – one full screen Windows 8 Style (the new name for Metro) app with the PDF content and nothing else, not even an ‘X’ to close it. In fact Windows 8 users are somehow expected to know beforehand (“by osmosis”, as this Windows 8.1 review puts it) that to close a Windows 8 Style app you have to grab it from the top and drag downwards with your mouse. Contrast this with the same PDF viewed on Windows 7:

ubuntu-pdf-win7

Needless to say, everything that you can do with a PDF is immediately accessible either from the toolbars or via the menus. There is no hidden stuff, no needing to drag your mouse into a corner to open some Start Screen or Charms Bar. See, the program actually shows you what it can do, and for new users that’s important. The “Content before Chrome” idea is wrong precisely because when you use a program, you want to do stuff, not just see stuff.

So it’s no wonder that Microsoft seems to have made a U-turn on its Windows 8 Style design stuff. If MMN is an example of bad usability, this Windows 8 abomination is an example of… unusability.

Reserving Windows 10 Upgrade

Windows 10 will be released on 29th July, and it’s no secret that it will be a free upgrade from Windows 7 or 8. In fact, a special icon is currently appearing in the system tray, allowing us to reserve our upgrade – possibly to spread out the download rather than have everyone hit the servers on launch day.

When you click on the system tray icon, a small application gives you an overview of what’s coming in Windows 10, and allows you to reserve the upgrade. Below are screenshots of this application (from 3 weeks ago), included here as a historical record. The application is slightly different today, but more or less conveys the same message.

On Redis Desktop Manager and Redis Keys

Update 11th August 2015: As from version 0.8.0 beta 1, Redis Desktop Manager now supports the SCAN command (rather than KEYS) for Redis 2.8 onwards. Although this limits the applicability of this article to older servers, I am more than happy that this shortcoming has been addressed. The original article below remains both for historical purposes and to warn developers of incorrect use of the KEYS command.

There’s a tool called Redis Desktop Manager which can sometimes be useful to inspect the keys in a Redis database. Indeed one of its features is that it presents a treeview showing a structured representation of the keys in the database:

redis-desktop-manager-treeview

But how is this treeview built? That’s easy to find out, by using the Redis MONITOR command to see the incoming commands:

redis-desktop-manager-keys

 

The first two commands are executed when Redis Desktop Manager connects to the Redis server, and the other two are executed when the database is expanded in the treeview, revealing the keys in that database.

You’ll notice that the last command is a KEYS command, which with its wildcard argument (*) is effectively retrieving every key in the database. We can see an example of what this gives us by running the KEYS command ourselves:

redis-desktop-manager-keys-response

Now, in this case I only have a handful of keys in my Redis database, but it’s pretty normal for real Redis databases to have very large numbers of keys. Retrieving all that data places a large burden on the Redis server, which due to its single-threaded nature will not be able to serve other requests while it is stuck retrieving every key in the database.

In fact, the KEYS command documentation particularly warns against its use in production environments:

“Warning: consider KEYS as a command that should only be used in production environments with extreme care. It may ruin performance when it is executed against large databases. This command is intended for debugging and special operations, such as changing your keyspace layout. Don’t use KEYS in your regular application code. If you’re looking for a way to find keys in a subset of your keyspace, consider using SCAN or sets.”

I understand why Redis Desktop Manager uses the KEYS command: it needs to retrieve all the keys in the database in order to determine how the tree structure will be displayed (since each delimited part of the key is rendered as a node). That’s the whole point of having a treeview.

However, what seems to be a useful feature can actually be very dangerous, especially when used on production servers. So do take care when using Redis Desktop Manager in such environments.

My recommendation to developers using Redis is to keep good documentation of your keys, so you won’t need any Redis command to tell you what keys are in the database. That’s not what Redis is for.

But if you do ever need to inspect your Redis keys on the server, at least follow the advice in the documentation and use SCAN instead. While this may still be expensive to retrieve the entire set of keys, it can be done in small batches, thus allowing other requests to be serviced in between iterations.

ImapTalk 2.1 beta released

Today, I released a beta of the next version of ImapTalk, which is 2.1. This new version brings a number of small enhancements and features, but the biggest new feature is that of batch commands. This allows you to send multiple commands at once by entering multiple lines of text – very handy when you prepare a text file full of commands and don’t want to execute them one by one.

Visit the ImapTalk project page to view the full release notes and download the beta.

A Redis Analogy in .NET

In this article, we’re going to create a simple Console Application that feels a little bit like Redis. The intention is simply to illustrate what you can do with Redis, not to create anything serious. For this reason, the code will be greatly simplified: we’ll have only a subset of Redis commands, no multithreading, no client/server code, (almost) no parameter validation, and no architecture whatsoever.

Introduction

Redis is a key-value store. You can think of it as a dictionary. In its simplest sense, it could be represented like this:

var store = new Dictionary<string, string>();

However, values in Redis can be more than just strings. In fact, Redis currently supports five main data types:

  • Strings
  • Lists
  • Hashes
  • Sets
  • Sorted Sets

Since in .NET these data types don’t share a common interface, we will be representing our values as plain objects:

var store = new Dictionary<string, object>();

This will result in some unfortunate type checking code, but this is necessary in this case. Maintaining a separate dictionary per type is not feasible since keys must be globally unique (i.e. you can’t use the same key for a list and a hash).

Note also how the use of a dictionary as an in-memory store means we aren’t persisting anything to disk. This might seem to be an oversimplification, but Redis’ default configuration is actually to store data only in memory. It does support persistence if you need it, but we won’t go there.

Redis supports a simple protocol which feels a bit like issuing commands in a command line if you use something like IMAPTalk. Thus, in our Console Application, we’ll use the following code to simulate Redis’ command processing logic:

            string input = string.Empty;

            while (true)
            {
                input = Console.ReadLine();

                if (!string.IsNullOrEmpty(input))
                {
                    var tokens = input.Split(); // split the input into words or tokens

                    if (tokens.Length >= 1)
                    {
                        string command = tokens.First().ToLowerInvariant();

                        try
                        {
                            switch(command)
                            {
                                    // TODO command handling code goes here
                                default:
                                    Console.WriteLine("Unrecognized command.");
                                    break;
                            }
                        }
                        catch (Exception)
                        {
                            Console.WriteLine("Error!");
                        }
                    }
                }
                else
                    Console.WriteLine("Invalid command.");
            }

The above code just accepts input, checks what the command is, and takes action accordingly. In the next sections, we’ll replace that TODO with code to actually handle some of the commands.

Strings

redis-imaptalk-strings

The simplest data type in Redis is the string. You can easily assign a value to a key using the SET command, and then retrieve that value using the GET command, as shown above. The SET command is just a matter of assigning the value in the dictionary:

                                case "set":
                                    {
                                        string key = tokens[1];
                                        string value = tokens[2];
                                        store[key] = value;
                                        Console.WriteLine("Done.");
                                    }
                                    break;

The GET command similarly involves basic retrieval from the dictionary, but we also need to cater for non-existent keys, and values that are not strings:

                                case "get":
                                    {
                                        string key = tokens[1];
                                        if (store.ContainsKey(key))
                                        {
                                            var value = store[key] as string;
                                            if (value != null)
                                                Console.WriteLine(value);
                                            else
                                                Console.WriteLine("Invalid data type!");
                                        }
                                        else
                                            Console.WriteLine("Key not found!");
                                    }
                                    break;

We can also support key removal using the DEL command. Again, this is a simple operation against the dictionary:

                                case "del":
                                    {
                                        string key = tokens[1];
                                        store.Remove(key);
                                        Console.WriteLine("Done.");
                                    }
                                    break;

We can now test our three basic string commands:

redis-analogy-strings

Lists

redis-imaptalk-lists

Redis also supports Lists. You can use the LPUSH and RPUSH commands to add an item to the beginning or end of a list respectively – allowing you to easily use the list as a stack or a queue as well. These operations also automatically create the list if it doesn’t already exist.

Here’s how we can implement LPUSH:

                                case "lpush":
                                    {
                                        string key = tokens[1];
                                        string value = tokens[2];
                                        List<string> list = null;

                                        // create/get list

                                        if (!store.ContainsKey(key))
                                        {
                                            list = new List<string>();
                                            store[key] = list;
                                        }
                                        else
                                            list = store[key] as List<string>;

                                        // insert new value in list

                                        if (list != null)
                                        {
                                            list.Insert(0, value);
                                            Console.WriteLine("Done");
                                        }
                                        else
                                            Console.WriteLine("Invalid data type!");
                                    }
                                    break;

The first thing we do here is retrieve the list we’re going to work with. If it doesn’t exist, we create it. We can then proceed to insert the new item into the list – in this case using List<T>.Insert() to put the item at the beginning of the list.

Note that in Redis, LPUSH actually supports insertion of multiple items with the same command. I’m not doing that here to keep things as simple as possible.

RPUSH is pretty much the same thing, except that you call List<T>.Add() instead of .Insert(), so that the new item ends up at the end of the list.

To remove items from the list, we’ll implement LPOP and RPOP, which remove items from the beginning and end of the list respectively. Here’s LPOP:

                                case "lpop":
                                    {
                                        string key = tokens[1];
                                        
                                        if (store.ContainsKey(key))
                                        {
                                            var list = store[key] as List<string>;
                                            if (list != null)
                                            {
                                                if (list.Count > 0)
                                                {
                                                    Console.WriteLine(list.First());
                                                    list.RemoveAt(0);
                                                }
                                                else
                                                    Console.WriteLine("Empty!");
                                            }
                                            else
                                                Console.WriteLine("Invalid data type!");
                                        }
                                        else
                                            Console.WriteLine("Key not found!");
                                    }
                                    break;

Aside from all the validity checks, all we’re doing here is returning the first item in the list, and then removing it. If there are no items in the list, we simply return “Empty!”.

For RPOP, the only difference is that we retrieve and remove the last item instead:

                                                    Console.WriteLine(list.Last());
                                                    list.RemoveAt(list.Count - 1);

Finally, we need something that can retrieve items in the list (without removing them). For this we have LRANGE, which retrieves a specified range of items (e.g. item #2 till item #4) from the beginning of the list. Indices may be out of range without causing errors; the indices that actually exist will be returned.

Here is the implementation for LRANGE:

                                case "lrange":
                                    {
                                        string key = tokens[1];
                                        int start = Convert.ToInt32(tokens[2]);
                                        int stop = Convert.ToInt32(tokens[3]);

                                        if (start > stop)
                                            Console.WriteLine("Empty!");
                                        else if (store.ContainsKey(key))
                                        {
                                            var list = store[key] as List<string>;
                                            if (list != null)
                                            {
                                                if (start < 0)
                                                    start = 0;

                                                if (stop > list.Count - 1)
                                                    stop = list.Count - 1;

                                                var items = list.GetRange(start, stop - start + 1);
                                                if (items.Any())
                                                {
                                                    foreach(var item in items)
                                                        Console.WriteLine(item);
                                                }
                                                else
                                                    Console.WriteLine("Empty!");
                                            }
                                            else
                                                Console.WriteLine("Invalid data type!");
                                        }
                                        else
                                            Console.WriteLine("Key not found!");
                                    }
                                    break;

We can now test our list functionality:

redis-analogy-lists

Hashes

redis-imaptalk-hashes

Hashes are like dictionaries in themselves. Rather than mapping a value directly to a key, hashes map values onto a field of a key. This allows you to represent objects with a number of attributes (e.g. a customer having separate values for Name, Age, etc).

With HSET, we can assign a value to a key-field, as shown in the screenshot above. Like with Lists, this operation creates the Hash if it doesn’t already exist. Here’s the HSET implementation:

                                case "hset":
                                    {
                                        string key = tokens[1];
                                        string field = tokens[2];
                                        string value = tokens[3];
                                        Dictionary<string, string> hash = null;

                                        // create/get hash

                                        if (!store.ContainsKey(key))
                                        {
                                            hash = new Dictionary<string, string>();
                                            store[key] = hash;
                                        }
                                        else
                                            hash = store[key] as Dictionary<string, string>;

                                        // set field in hash

                                        if (hash != null)
                                        {
                                            hash[field] = value;
                                            Console.WriteLine("Done");
                                        }
                                        else
                                            Console.WriteLine("Invalid data type!");
                                    }
                                    break;

The code is very similar to that of LPUSH, with the difference that the key now maps to a dictionary (the hash), and the value is assigned to a field on that hash. Think of it as: key -> field -> value.

After using HSET to set a value, we can retrieve it with HGET:

                                case "hget":
                                    {
                                        string key = tokens[1];
                                        string field = tokens[2];

                                        if (store.ContainsKey(key))
                                        {
                                            var hash = store[key] as Dictionary<string, string>;

                                            if (hash != null)
                                            {
                                                if (hash.ContainsKey(field))
                                                    Console.WriteLine(hash[field]);
                                                else
                                                    Console.WriteLine("Field not found!");
                                            }
                                            else
                                                Console.WriteLine("Invalid data type!");
                                        }
                                        else
                                            Console.WriteLine("Key not found!");
                                    }
                                    break;

HKEYS can be used to retrieve all fields of a hash, given the key:

                                case "hkeys":
                                    {
                                        string key = tokens[1];

                                        if (store.ContainsKey(key))
                                        {
                                            var hash = store[key] as Dictionary<string, string>;

                                            if (hash != null)
                                            {
                                                foreach (var field in hash.Keys)
                                                    Console.WriteLine(field);
                                            }
                                            else
                                                Console.WriteLine("Invalid data type!");
                                        }
                                        else
                                            Console.WriteLine("Key not found!");
                                    }
                                    break;

HGETALL, on the other hand, gets all fields of a hash and their corresponding values:

                                case "hgetall":
                                    {
                                        string key = tokens[1];

                                        if (store.ContainsKey(key))
                                        {
                                            var hash = store[key] as Dictionary<string, string>;

                                            if (hash != null)
                                            {
                                                foreach (var kvp in hash)
                                                {
                                                    Console.WriteLine(kvp.Key);
                                                    Console.WriteLine(kvp.Value);
                                                }
                                            }
                                            else
                                                Console.WriteLine("Invalid data type!");
                                        }
                                        else
                                            Console.WriteLine("Key not found!");
                                    }
                                    break;

Let’s test out our hash functionality:

redis-analogy-hashes

Sets

redis-imaptalk-sets

Sets in Redis are the mathematical kind: all items in a set are distinct, and multiple sets may be combined via standard set operations (union, intersection and difference).

Use SADD to add one or more items to a set. If the set does not exist, it will be created automatically. Here’s the SADD implementation:

                                case "sadd":
                                    {
                                        string key = tokens[1];
                                        var members = tokens.Skip(2); // sadd key member [member ...]
                                        HashSet<string> set = null;

                                        // create/get set

                                        if (!store.ContainsKey(key))
                                        {
                                            set = new HashSet<string>();
                                            store[key] = set;
                                        }
                                        else
                                            set = store[key] as HashSet<string>;

                                        // add member to set

                                        if (set != null)
                                        {
                                            foreach (var member in members)
                                                set.Add(member);

                                            Console.WriteLine("Done");
                                        }
                                        else
                                            Console.WriteLine("Invalid data type!");
                                    }
                                    break;

SMEMBERS gives you all the members of the set:

                                case "smembers":
                                    {
                                        string key = tokens[1];

                                        if (store.ContainsKey(key))
                                        {
                                            var set = store[key] as HashSet<string>;

                                            if (set != null)
                                            {
                                                foreach (var member in set)
                                                    Console.WriteLine(member);
                                            }
                                            else
                                                Console.WriteLine("Invalid data type!");
                                        }
                                        else
                                            Console.WriteLine("Key not found!");
                                    }
                                    break;

Standard set operations (union, intersection and difference) require two sets. If either doesn’t exist, these operations will return an empty set rather than giving any kind of error. In Redis, these operations may work on multiple sets at once; but in these examples we’re going to limit the scenario to two sets at a time for the sake of simplicity.

Set union (SUNION) returns all distinct items in the specified sets:

                                case "sunion":
                                    {
                                        List<HashSet<string>> sets = new List<HashSet<string>>();

                                        var key1 = tokens[1];
                                        var key2 = tokens[2];

                                        // let's assume the keys exist and are the correct type

                                        var set1 = store[key1] as HashSet<string>;
                                        var set2 = store[key2] as HashSet<string>;

                                        // get union

                                        var union = set1.Union(set2);
                                        foreach(var member in union)
                                            Console.WriteLine(member);
                                    }
                                    break;

Set intersection (SINTER) returns those items which are common to both sets:

                                case "sinter":
                                    {
                                        List<HashSet<string>> sets = new List<HashSet<string>>();

                                        var key1 = tokens[1];
                                        var key2 = tokens[2];

                                        // let's assume the keys exist and are the correct type

                                        var set1 = store[key1] as HashSet<string>;
                                        var set2 = store[key2] as HashSet<string>;

                                        // get intersection

                                        var union = set1.Intersect(set2);
                                        foreach (var member in union)
                                            Console.WriteLine(member);
                                    }
                                    break;

Finally, set difference (SDIFF) returns those items which are in the first set but which are not in the second:

                                case "sdiff":
                                    {
                                        List<HashSet<string>> sets = new List<HashSet<string>>();

                                        var key1 = tokens[1];
                                        var key2 = tokens[2];

                                        // let's assume the keys exist and are the correct type

                                        var set1 = store[key1] as HashSet<string>;
                                        var set2 = store[key2] as HashSet<string>;

                                        // get intersection

                                        var union = set1.Except(set2);
                                        foreach (var member in union)
                                            Console.WriteLine(member);
                                    }
                                    break;

Let’s test that out:

redis-analogy-sets

Sorted Sets

redis-imaptalk-sortedsets

Sorted Sets in Redis are similar to sets, but their members are sorted by a score. This may be used for any kind of ordering from the latest 5 comments to the top 5 posts with most likes.

We can represent a Sorted Set similarly to a hash, using the scheme key -> score -> value. However, there are two main differences from a hash:

  • For each score, there may be multiple values; these are sorted lexicographically.
  • The scores are ordered (sorted).

The implementation of a Sorted Set in .NET is not as trivial as the other data types, and requires a combination of collections. While a simple representation could be made by combining a SortedDictionary (score -> values) with a SortedSet (distinct and sorted values), this would not allow all Sorted Set commands to be supported (e.g. ZRANK can find the index of a given value, requiring a reverse lookup).

Sorted Sets are thus beyond the scope of this article, which is intended to provide a simple mapping between Redis data types and .NET collections.

Summary

This article explained how the Redis data types work, and showed how some of their operations may be implemented using standard .NET collections. A typical mapping could be:

  • Strings – strings
  • Lists – Lists
  • Hashes – Dictionaries or ConcurrentDictionaries
  • Sets – HashSets
  • Sorted Sets – a custom data structure

Source Code

The source code for this article is available on BitBucket.

IMAPTalk 2 (final) Released

The final version of IMAPTalk 2 has finally been released. This version is mostly the same as the beta 3 release from last week. IMAPTalk 2 had been in beta since October, allowing ample time for testing.

The IMAPTalk page has also been updated to reflect the new version. Some older downloads, instructions and screenshots from the early versions have been removed. Go there to download the latest version!

Remember that since beta 3, IMAPTalk can be used for manual interaction with Redis servers. I am planning some new tools (for upcoming releases) that should be useful when using IMAPTalk for Redis.