All posts by Gigi

Software development

SignalR Scaleout with Redis Backplane

Leave a comment

Introduction

In “Getting Started with SignalR“, I provided a gentle introduction to SignalR with a few simple and practical examples. The overwhelming response showed that I’m not alone in thinking this is an awesome technology enabling real-time push notifications over the web.

Web applications often face the challenge of having to scale to handle large amounts of clients, and SignalR applications are no exception. In this example, we’ll see one way of scaling out SignalR applications using something called a backplane.

Scaleout in SignalR

SignalRScaleout

“Introduction to Scaleout in SignalR” (official documentation) describes how SignalR applications can use several servers to handle increasing numbers of clients. When a server needs to push an update, it first pushes it over a message bus called a backplane. This delivers it to the other servers, which can then forward the update to their respective clients.

According to the official documentation, scaleout is supported using Azure, Redis or SQL Server as backplanes. Third-party packages exist to support other channels, such as SignalR.RabbitMq.

Scaleout Example using Redis

“Introduction to Scaleout in SignalR” (official documentation) describes how to use SignalR as a backplane. To demonstrate this, I’ll build on the Chat Example code from my “Getting Started with SignalR” article.

All we need to scaleout using Redis is install the Microsoft.AspNet.SignalR.Redis NuGet package, and then set it up in the Startup class as follows:

        public void Configuration(IAppBuilder app)
        {
            GlobalHost.DependencyResolver.UseRedis("192.168.1.66", 6379, null, "SignalRChat");
            app.MapSignalR();
        }

In the code above, I am specifying the host and port of the Redis server, the password (in this case null because I don’t have one), and the name of the pub/sub channel that SignalR will use to distribute messages.

To test this, you can get a Redis server from the Redis download page. Redis releases for Windows exist and are great for testing stuff, but remember they aren’t officially supported for production environments.

Now to actually test it, I’ve set up the same scaleout-enhanced chat application on two different machines, and subscribed to the Redis pub/sub channel:

signalr-scaleout-computer1

Watching the pub/sub channel reveals what SignalR is doing under the hood. There are particular messages going through when the application initializes on each machine, and you can also see the actual data messages going through. So when you write a message in the chat, you can also see it in the pub/sub channel.

But even better than that, you’ll also see it on the client (browser) that’s hooked up to the other machine:

signalr-scaleout-computer2

The magic you need to appreciate here is that these aren’t two browsers connected to the same server; they are actually communicating with different servers on different machines. And despite that, the messages manage to reach all clients thanks to the backplane, which in this case is Redis.

Caveats

So I’ve shown how it’s really easy to scale out SignalR to multiple servers: you need to install a NuGet package and add a line of code. And I’ve actually tested it on two machines.

stash-1-244250d58073b0ed1

But that’s not really scaleout. I don’t have the resources to do large-scale testing, and only intended to show how scaleout is implemented with this article. The actual benefits of scaleout depend on the application. As the official documentation warns, the addition of a backplane incurs overhead and can become a bottleneck in some scenarios. You really need to study whether your application is a good fit for this kind of scaleout before going for it.

Software development

Getting Started with SignalR

1 Comment

Introduction

SignalR is an open source .NET library enabling real-time broadcasts of data on the web. Typical example applications include chats, stock tickers, and even games. It does this by leveraging and abstracting a number of different technologies, choosing the most suitable transport mechanism based on what the client and server support (since WebSocket support is not yet widespread).

At the time of writing this article, SignalR v2 is the latest implementation, but SignalR v3 is in beta along with ASP .NET 5. I’ll be using VS2015 for this article.

In this article, we’ll cover some really simple examples that will get you up and running fast.

The source code for this article is available on the Gigi Labs BitBucket repository:

Setting up a SignalR Hub

Create a new ASP .NET Web Application, and choose the Empty template. I’m using the one under ASP .NET 4.5.2 since ASP .NET 5 is still a preview at the time of writing this article:

signalr-empty-application

Next, you’ll need to add a SignalR Hub (screenshot below). A hub is a publish/subscribe implementation, and automatically tracks connections. Additionally, it provides ways of calling methods on clients, and selecting which clients to broadcast to – we’ll see how this works shortly.

signalr-add-hub

Adding the hub not only adds a class for the hub, but also gives you a basic method and also adds scripts needed for SignalR to run:

signalr-added-hub-scripts

Next, add an OWIN Startup Class:

signalr-add-owin-startup-class

In this class, just set up SignalR in the Configuration() method:

        public void Configuration(IAppBuilder app)
        {
            app.MapSignalR();
        }

Good! Now we’re all set to create our examples. We’ll also need some client code, so add an empty HTML page called index.html. Right click it in Solution Explorer, and select “Set as Start Page” so it will be automatically loaded when you hit F5 to debug.

Basic Hello World Example

When we added our Hub, we got some initial code:

        public void Hello()
        {
            Clients.All.hello();
        }

This works as follows: when a client calls this Hello() method on the server (Hub), the hub will call a hello() method on all the connected clients. We can use this example code as-is, but we need to create some client code.

First, add the following structure to index.html:

<!DOCTYPE html>
<html>
<head>
    <title>SignalR Example 1</title>
	<meta charset="utf-8" />
    <script src="Scripts/jquery-1.10.2.min.js"></script>
    <script src="Scripts/jquery.signalR-2.1.2.min.js"></script>
    <script src="signalr/hubs"></script>
    <script type="text/javascript">
        $(function () {
            // TODO client code goes here
        });
    </script>
</head>
<body>
    <div id="log"></div>
</body>
</html>

There are three things to notice here. First, we have a few references to JavaScript files – the versions need to match the ones in the Scripts folder. The signalr/hubs reference is autogenerated, so you don’t have to worry about it. Secondly, there is a placeholder for Javascript client code which we’ll add next. Finally, we have a “log” div which is where we’ll dump any output received from the server.

For the client code, we first need a reference to the hub itself:

var myHub = $.connection.myHub1;

Next, we add the method on the client that the server will call whenever it needs to broadcast something:

            myHub.client.hello = function () {
                $("#log").append("Hello <br />");
            }

Finally, we add code to initialize the hub itself, and when the initialization is complete, we will call the method on the server. This will in turn cause the client method above to be called:

            $.connection.hub.start().done(function () {
                myHub.server.hello();
            });

We can now test this and see that we get a Hello response from the server:

signalr-hello-1

More importantly, if we open a second browser window on the same URL, you’ll notice that both windows will get a new Hello response. That’s because the server/hub is broadcasting to all clients whenever a client calls its Hello() method.

signalr-hello-2

Chat Example

Creating a chat server is not much more complex than this. In fact it’s a typical example of SignalR usage, and you can check out the official tutorial if you like.

For this example, I’ve created a new project with a new Hub:

    public class ChatHub : Hub
    {
        public void Broadcast(string message)
        {
            Clients.All.receive(Context.ConnectionId, message);
        }
    }

Some names have changed, but it’s mostly the same as before. One thing you’ll notice that’s different, though, is that Context.ConnectionId. It’s SignalR’s way of giving you the connectionId of the client who called the server/hub method, and I’m using it to distinguish between clients in the chat (as opposed to the official tutorial which prompts for a name).

On the client side, things have also changed slightly. In the HTML body, I’ve added an input field and a send button:

<body>
    <input id="message" type="text" />
    <button id="send">Send</button>
    <div id="log"></div>
</body>

In the client JavaScript code, as before, we’re first getting a reference to the hub (different name this time):

var chatHub = $.connection.chatHub;

Then we add the method that the server will call on the client. Remember, the signature must match what we declared on the hub. In this case we’re passing in connectionId and message, and then displaying them:

            chatHub.client.receive = function (connectionId, message) {
                $("#log").append("<strong>User " + connectionId
                    + " wrote:</strong>" + message + " <br />");
            }

Then we add the hub initialization code, together with any actions we want to carry out post-initialization. In this case, we set up a handler for when the user sends a message:

            $.connection.hub.start().done(function () {
                $("#send").on("click", function () {
                    var message = $('#message').val();
                    chatHub.server.broadcast(message);
                })
            });

There’s nothing keeping us from adding further logic to show when someone joins the chat, for instance. But let’s keep this super-simple.

We can now test this:

signalr-chat-test

Stock Ticker Example

The previous examples show broadcasts to clients triggered by the clients themselves. There are many scenarios where it makes sense for the server to spontaneously send updates to clients – a stock ticker is a common example (see official tutorial).

For this example, I’ve again created a new project and a new Hub. Since the server will spontaneously send updates to clients, the hub does not need to expose any methods:

    public class StockTickerHub : Hub
    {

    }

Instead, we’ll have a thread that periodically sends updates to the clients of that hub. Since I have no better place to put this, it’s going in the Startup class:

    public class Startup
    {
        private static Thread stockTickerThread = new Thread(() =>
            {
                var stockTickerHub = GlobalHost.ConnectionManager.GetHubContext<StockTickerHub>();

                while (true)
                {
                    stockTickerHub.Clients.All.update(5);
                    Thread.Sleep(1000);
                }
            });

        public void Configuration(IAppBuilder app)
        {
            app.MapSignalR();

            stockTickerThread.Start();
        }
    }

The important thing to notice here is how we’re getting a reference to the hub via GlobalHost.ConnectionManager.GetHubContext<StockTickerHub>(). Since I’m more concerned with demonstrating how to use SignalR than any stock ticker logic, I’m just sending a fixed value of 5 every second.

The client code is practically the same as the first example, except that we don’t need to do anything once the hub is initialized:

<!DOCTYPE html>
<html>
<head>
    <title>SignalR Example 3</title>
    <meta charset="utf-8" />
    <script src="Scripts/jquery-1.10.2.min.js"></script>
    <script src="Scripts/jquery.signalR-2.1.2.min.js"></script>
    <script src="signalr/hubs"></script>
    <script type="text/javascript">
        $(function () {
            var stockTickerHub = $.connection.stockTickerHub;

            stockTickerHub.client.update = function (value) {
                $("#log").append(value + "<br />");
            }

            $.connection.hub.start().done(function () {
                
            });
        });
    </script>
</head>
<body>
    <div id="log"></div>
</body>
</html>

We can now test that the spontaneous updates are received:

signalr-stockticker-test

Beyond this Article

This article showed basic usage of SignalR. I intentionally kept the examples really simple so that you could get up and running in no time.

Having said that, there’s a lot more to SignalR than we’ve seen here. For instance, we’ve always broadcasted to all clients. SignalR allows you to choose which clients to send to, for example you can send to a specific client, or to clients subscribed to a particular topic. That’s beyond the scope of this article, so if you want to know more about SignalR, check out the multitude of resources on the net.

Software development

Getting Started with RabbitMQ with .NET

7 Comments

This article contains some instructions to make it easy for people to jump into the wonderful world of message queues with RabbitMQ. Its purpose is not to provide any in-depth explanations, as I’m pretty new to the topic myself. 🙂

RabbitMQ is built on Erlang, so you’ll need to install that first:

rabbitmq-download-erlang

You can then download and install RabbitMQ itself:

rabbitmq-download-rabbitmq

Next, you should install the management plugin. This provides a web UI allowing you to manage your queue and perform basic testing (e.g. publish messages).

To do this, navigate to the folder where you installed RabbitMQ, and go into the sbin folder. From here, enter the following command:

rabbitmq-plugins enable rabbitmq_management

Once that is installed, you can access the administrative web UI by going to http://localhost:15672/ with username guest and password guest. Go to the Queues tab and create a new queue:

rabbitmq-add-queue

You’ll now see the new queue listed under “All queues”. Click on the name to select it. This will allow you to view information about the queue (including statistics, such as the number of messages in the queue) and perform a number of operations (including publishing messages).

But before we do that, let’s create a simple client in .NET that can subscribe to the queue and consume any messages we publish. Some basic code is available in this tutorial, so we can just install the RabbitMQ.Client NuGet package and then write some code based on it:

        static void Main(string[] args)
        {
            var factory = new ConnectionFactory() { HostName = "localhost" };

            using (var connection = factory.CreateConnection())
            {
                using (var channel = connection.CreateModel())
                {
                    channel.QueueDeclare("testqueue", true, false, false, null);

                    var consumer = new EventingBasicConsumer(channel);
                    consumer.Received += Consumer_Received;
                    channel.BasicConsume("testqueue", true, consumer);

                    Console.ReadLine();
                }
            }
        }

        private static void Consumer_Received(object sender, BasicDeliverEventArgs e)
        {
            var body = e.Body;
            var content = Encoding.UTF8.GetString(body);
            Console.WriteLine(content);
        }

We can now run this client, and leave it waiting for messages.

To test it, let’s publish a message from the RabbitMQ management web UI:

rabbitmq-publish-message

You can see from the client that the message has been received:

rabbitmq-consume-message

So that shows in a very basic way how you can setup RabbitMQ on Windows, publish messages from the management web UI, and consume messages by writing a simple .NET client.

Update 2nd March 2016: If the management plugin fails to install with something like “unable to contact node”, here are a few things you can try:

Rants

An Overview of Air Malta’s Customer Service

5 Comments

Introduction

Just a few days ago, Air Malta was voted 6th Best Regional Airline in Europe, measured based on customer satisfaction. And as an irate former customer of Air Malta who has experienced a terrible customer experience just a few weeks ago, this really makes me wonder under which rock people are living. Just the day before, another story appeared in which a cellist lost her cello’s seat because of an overbooking.

This article deals with my own bad customer experience with Air Malta, due to being stranded in Budapest towards the end of June 2015. However, it also mentions other incidents which give a deeper illustration of Air Malta’s customer service.

Summary

  • We were meant to fly from Budapest to Malta on 29th June 2015.
  • Due to a technical fault, a smaller plane was used, leaving many passengers stranded in Budapest.
  • We spent 7 hours at the airport.
  • Air Malta never communicated with us directly.
  • We missed all replacement flights for the day since Air Malta took ages to confirm our seats.
  • Budapest airport staff claimed that Air Malta was uncooperative and refused to pay for the overnight accommodation.
  • When we claimed compensation (due by EU law), Air Malta agreed only on condition that we sign an agreement which, among other things, forces us to keep the settlement confidential.
  • Other past incidents show how Air Malta knows nothing about customer service.

Seven Hours at the Airport

On 29th June 2015, I went to Budapest airport with my friends to catch the Air Malta flight back to Malta. Shortly after arriving, we learned from the flight schedule board at the terminal that the flight was being delayed by two hours. We thought nothing of it at the time, as there was still plenty of time to get back to Malta in the evening.

However, things looked a lot more grim as we joined the checkin queue. This queue was at a standstill for quite a long time, with passengers and airport staff alike shuffling back and forth. We eventually learned that the aeroplane suffered some kind of technical fault, and that a smaller one was being sent as a replacement, which meant that not everybody would fit. Some people were rerouted via other flights.

Over the span of the next few hours, airport staff struggled to deal with Air Malta in order to get a replacement flight. Time after time, they brought up our hopes, saying there was another flight we could take – only to bring those hopes back down when we realised that Air Malta had failed to confirm our places on those flights in time.

Once we missed the last flight, via Brussels, we resigned ourselves to the fate of spending an extra night in Budapest. Budapest is a lovely place, but many of us had to go to work the next day, and others would miss a day from their holiday in Malta. At that point, we told the airport staff that we understood there was nothing more that could be done, and that we’d appreciate being settled in a hotel as soon as possible. Spending seven hours in an airport is not fun.

The airport staff apparently tried to get hold of Air Malta to get the accommodation, dinner and transfer booked, but more time went by and we remained at the airport. In the end, the airport pretty much gave up on Air Malta and arranged for everything on their own initiative. In all this time, we never heard anything from Air Malta directly.

On Compensation

During our seven-hour ordeal at Budapest airport, we had already educated ourselves – using intermittent WiFi access that didn’t work most of the time – about our rights under the circumstances. EU law [PDF] entitled us to at least EUR250 plus food expenses. We didn’t count the dinner and accommodation of course, because the airport covered those. But we did incur additional expenses due to this mess, including:

  • Mobile expenses. This is difficult to quantify since Vodafone charge EUR3, and Go want EUR12 and an affidavit, in order to provide a breakdown of charges.
  • Lost day of work. The cost of this varies from person to person.

The EUR250 worth of compensation is not a very fair deal, to be honest. The value of compensation depends mainly on distance, and not on duration (as long as it’s more than 3 hours). So technically, if you’re delayed for 3 hours, 1 day, or a week, you get the same compensation. That can’t possibly make up for a lost day of work.

Also, having such a low compensation fee makes it worthwhile for unscrupulous airlines such as Air Malta to exploit the practice of overbookings (more on this later).

Clarifications and Settlement

One of my friends and fellow passengers wrote to Air Malta on behalf of 8 of the people who spent the extra night at Budapest, requesting compensation. Aside from requesting compensation for the 8 people, the email also asked for clarifications on why we were neglected as described earlier in this article.

Air Malta replied as follows (emphasis mine):

“First of all, we would like to apologise for all the inconvenience caused with reference to the flight you had with us. Although Air Malta tries to offer the best possible service, there are instances that the service given is not in accordance to the passengers’ expectation. In this case, the flight was overbooked and we had to reroute several passengers from the direct flight to other connections available.

We had problems with communicating with the handling agents in Budapest, and we tried to accomodate all the passengers to fly the same day. In fact, some of the passengers who were denied boarding flew via Munich or Athens, and we tried to reroute the others via Brussels. We already had made the necessary changes, however, due to several reasons, the check-in for the flight from Budapest to Brussels closed soon before the passengers could arrive to board.

“Air Malta made it clear that any hotel accomodation, transport and meals required for the passengers, can be refunded, and we never delayed or declined to offer such convenience, as stated in your email. However, as we said beforehand, we had an ongoing miscommunication with our handlers in Budapest, and this caused a lot of confusion and misinformation to the passengers.”

However, they agreed to pay the EUR250 plus food expenses, if we provide our bank details (necessary to transfer the refund) and sign a form (emphasis mine):

“In light of the above, to cover any inconvenience this mishap has caused all of the passengers that were denied boarding, we are offering the amount of €250 each, as per EU Law. Moreover, passengers that were denied boarding can send us the receipts of any meals that they had to buy for their extra stay by mail on the address in the signatory, in order to give the refund of such. Kindly fill in individually the Release and the ex-Gratia Form in order to provide you the amount stated (Reference and PNR listed above), and send them back as a reply to his email by not later than the 17th July 2015.”

The Release form, among other things, included the following (emphasis mine):

“1. The Releasor accepts the sum of EUR TWO HUNDRED AND FIFTY (€250.00) from the Releasees without admission of liability in full and final settlement of all claims for damages, interest, costs or expenses whether past, present or future and whether ascertained or unascertained, which the Releasor may have against the Releasees howsoever arising out of the matters recited above.

2. In consideration or receipt of the said sum Releasor:-

[…]

“(e) undertakes and warrants that he/she (as applicable) will keep this Release and Discharge and settlement recorded in it confidential, and will not disclose such information to any other person or entity (unless required by law to do so or for purposes of taking legal advice).”

So basically they were telling us that, in order to provide us compensation required by EU law, we would have to keep the settlement secret, accept it as final settlement, and sign it within a specified deadline. But more on that later.

My friend, who initiated the communication with Air Malta, followed up, asking for clarification about the use of the word “overbooked” in Air Malta’s email, and highlighting the fact that the airline should have made contact with us (as it is required to do by EU law), especially in the context of “miscommunication” with the airport.

In their subsequent reply, Air Malta explained their interpretation of their earlier use of the word “overbooking”:

“The reason why there was an overbooking is because there was a technical problem, and the aircraft had to be changed in order to avoid any further delays. In fact, instead of the A320, as previously scheduled, the A319 was used for the flight Budapest to Malta, leaving some passengers stranded.”

Conditions for Settlement

The settlement (EUR250 + food) offered was the least Air Malta could offer us. It was required by law, and at the same time did not cover other expenses (e.g. lost day of work) that resulted from this undesirable experience. Nonetheless, I sent Air Malta my bank details for the refund. However, I did not sign the Release Form, which required me among other things to keep the settlement confidential. You see, after being abandoned in Budapest airport like that, imposing conditions and deadlines to give me the least required by EU law felt like a kick in the nuts. It’s a bit like saying “I’ll give you what I owe you, but only if you get down on your knees.” So I wrote as follows:

“Please note that I am not including the Release Form. Under EU law, you are bound to compensate us for the denied boarding, and you have no right to place additional obligations or deadlines.”

They insisted that I sign the Release Form, and made it clear that they would only provide compensation if I complied.

I refused to sign the agreement given the conditions, and threatened to sue them.

Following that, they refused to provide the compensation unless I sign the form, with the excuse that it is a company requirement in order to process refunds.

I replied that their company requirements are irrelevant, because they are obliged by EU law to provide the compensation. That is where I decided to take the matter to the authorities.

The Air Passenger Rights section within the Malta Competition and Consumer Affairs Authority, which is the national authority that deals with consumer complaints, received my complaint. They made it clear that the airline have no right to impose conditions or deadlines to give compensation due by law. Quite the opposite: this compensation does not preclude further action to secure additional compensation – something that I would have renounced, had I signed the Release Form.

Facebook Denial

A few days ago, seeing the introduction of Air Malta’s “Sky Spa” free in-flight massage treatment, I wrote on their Facebook page:

air-malta-facebook-reply

It’s funny how “overbooking” appears again in this context – presumably because it’s so common that it’s one of the primary causes of customer complaints that the airline’s customer service department (if you can call it that) is well-experienced at handling.

But it’s also worrying that in this comment, Air Malta seem to not know anything about the incident. They may either be lying, or else their IT system is in such a complete mess that they are unable to retrieve historical flight data from it.

While I obviously cannot know which of these is actually the case, the latter is not too far-fetched. In fact, a friend of mine recently told me about a problem he had with Air Malta’s customer service. Although he uses their services regularly, the flights he took did not appear under his account on their website. When he reported the problem, instead of retrieving the data from their database, they asked him to produce evidence that he had taken those flights.

As if this wasn’t enough, another friend of mine reported similar conditions when reactivating his Flypass. Quoting Air Malta’s Flypass Terms and Conditions:

“Any closed Flypass account cannot be reactivated unless passenger proves that he/ she has flown at least four scheduled flights within the six months` period preceding his/ her request. The four boarding passes need to be forwarded to Flypass Unit and KMiles for these four flights only will be credited retroactively.”

On Overbookings

We’ve seen earlier how Air Malta used the word “overbooking” quite freely (even though overbooking had nothing to do with the Budapest incident). This is pretty ironic, because it is easy to find evidence of actual overbookings that Air Malta has been responsible for in the past, such as this one in Athens in 2013, and this one in Verona in 2010.

If these incidents sound like they happened a long time ago, check out the recent incident with the cello’s seat, clearly due to an overbooking according to Air Malta’s own words:

cellist-overbooking-fb

This is actually not very different from how Air Malta explained and justified the overbookings in the Athens 2013 article:

“”Overbooking is a normal occurrence in the airline industry and not particular to Air Malta. The reason why airlines overbook flights is due to ‘no-shows’ when people book on a flight and fail to show up at the check-in desks. Unfortunately seats blocked for booked passengers who fail to show up remain empty when the flight departs,” an airline spokesman said.

“These unutilised seats may be required by other travellers. In fact, flights which are not overbooked may result in many prospective passengers being denied an opportunity to fly on seats which would otherwise be unused, the airline said. Air Malta has a very low rate of denied boarding compared to the industry as a whole.”

So what they’re saying here is that they allow extra seats to be purchased, because if someone doesn’t show up, they can sell his seat to someone else. But if the person who doesn’t show up paid for the seat, what gives the airline the right to sell it to someone else, making double the money from a single seat? How is this even legal?

Also, the ridiculous compensation schemes we have make this practice very viable. At best, the airline would have sold two seats for the cost of one, and at worst, it would have sold that seat and compensated the person who was denied boarding, leaving that passenger more or less where he started financially, but with a great inconvenience.

If overbooking is a standard practice in the airline industry, it doesn’t mean all airlines do it. Ryanair, the low-cost airline frowned upon for many other reasons, “does not overbook its flights” [PDF], something that has been confirmed by a Which? investigation.

Similar Incident in 2008

After I explained my ordeal in Budapest, a friend of mine reached out and recounted a similar experience he had in Milan in 2008. He was informed in the morning that his 3pm flight was to be delayed by 3 hours. Despite this, he spent until 3am at the airport; no one communicated with him at the time, there was no Air Malta representative, and the airport staff knew nothing about the flight. Although he asked for compensation, he was never granted any.

Conclusion

Air Malta’s customer service is simply terrible, and the authorities will decide whether it is also illegal. My advice is thus: don’t fly with Air Malta, even if the ticket is free.

Even when flying with other airlines, know your rights. If your airline refuses compensation or imposes conditions, you have other means at your disposal to claim what is yours by right. It might take a little longer, but we can’t let the airlines keep treating us like dirt because they think we’re too afraid or apathetic to pursue them legally.

Software development

Setting up a Connection with StackExchange.Redis

13 Comments

Update 25th October 2016: Just looking to quickly set up a Redis connection? Check out the followup article. Read on for a more detailed article on the topic.

StackExchange.Redis is a pretty good .NET client for Redis. Unfortunately, it can be a little bit tricky to use, and the existing documentation is far from comprehensive.

After installing StackExchange.Redis via NuGet, a Redis connection can be obtained via a special ConnectionMultiplexer object. Working with this is already tricky in itself, and many get this wrong. For instance, check out the implementation in this answer:

public static ConnectionMultiplexer RedisConnection;
public static IDatabase RedisCacheDb;

protected void Session_Start(object sender, EventArgs e)
    {
        if (ConfigurationManager.ConnectionStrings["RedisCache"] != null)
        {
            if (RedisConnection == null || !RedisConnection.IsConnected)
            {
                RedisConnection = ConnectionMultiplexer.Connect(ConfigurationManager.ConnectionStrings["RedisCache"].ConnectionString);
            }
            RedisCacheDb = RedisConnection.GetDatabase();
        }
    }

As I pointed out in my question, this is a bad form of lazy initialization because it lacks thread safety: multiple threads may get through the checks and initialize multiple connections, resulting in connection leaks.

It is not hard to prove that this code is leaky in multithreaded environments. First, let’s set up the ConfigurationOptions with a client name so that we can identify connections coming from our program:

        private static Lazy<ConfigurationOptions> configOptions
            = new Lazy<ConfigurationOptions>(() => 
            {
                var configOptions = new ConfigurationOptions();
                configOptions.EndPoints.Add("localhost:6379");
                configOptions.ClientName = "LeakyRedisConnection";
                configOptions.ConnectTimeout = 100000;
                configOptions.SyncTimeout = 100000;
                return configOptions;
            });

Then, we provide a property with the faulty lazy initialization:

        private static ConnectionMultiplexer conn;

        private static ConnectionMultiplexer LeakyConn
        {
            get
            {
                if (conn == null || !conn.IsConnected)
                    conn = ConnectionMultiplexer.Connect(configOptions.Value);
                return conn;
            }
        }

Finally, we write some code that runs some Redis stuff in parallel:

        static void Main(string[] args)
        {
            for (int i = 0; i < 3; i++)
            {
                Task.Run(() =>
                    {
                        var db = LeakyConn.GetDatabase();
                        Console.WriteLine(i);

                        string key = "key" + i;

                        db.StringSet(key, i);
                        Thread.Sleep(10);
                        string value = db.StringGet(key);
                    }
                );
            }

            Console.WriteLine("Done");
            Console.ReadLine();
        }

When the program does its work, even with just 3 iterations, we get a total of six connections (when normally a single ConnectionMultiplexer should have at most 2 physical connections):

redis-leaky-connections

Another approach from this answer is to use an exclusive lock:

private static ConnectionMultiplexer _redis;
private static readonly Object _multiplexerLock = new Object();

private void ConnectRedis()
{
    try
    {
        _redis = ConnectionMultiplexer.Connect("...<connection string here>...");
    }
    catch (Exception ex)
    {
        //exception handling goes here
    }
}


private ConnectionMultiplexer RedisMultiplexer
{
    get
    {
        lock (_multiplexerLock)
        {
            if (_redis == null || !_redis.IsConnected)
            {
                ConnectRedis();
            }
            return _redis;
        }
    }
}

However, since Redis is often used as a cache in highly concurrent web applications, this approach essentially forces code to degrade into something sequential, and has obvious performance implications.

The correct approach to using ConnectionMultiplexer is described by this answer. It involves use of Lazy<T> for thread-safe lazy initialization (see Jon Skeet’s article on Singletons). Additionally:

  • It sets “abortConnect=false”, which means if the initial connect attempt fails, the ConnectionMultiplexer will silently retry in the background rather than throw an exception.
  • It does not check the IsConnected property, since ConnectionMultiplexer will automatically retry in the background if the connection is dropped.

With this info, we can now fix our code:

        private static Lazy<ConfigurationOptions> configOptions
            = new Lazy<ConfigurationOptions>(() => 
            {
                var configOptions = new ConfigurationOptions();
                configOptions.EndPoints.Add("localhost:6379");
                configOptions.ClientName = "SafeRedisConnection";
                configOptions.ConnectTimeout = 100000;
                configOptions.SyncTimeout = 100000;
                configOptions.AbortOnConnectFail = false;
                return configOptions;
            });

        private static Lazy<ConnectionMultiplexer> conn
            = new Lazy<ConnectionMultiplexer>(
                () => ConnectionMultiplexer.Connect(configOptions.Value));

        private static ConnectionMultiplexer SafeConn
        {
            get
            {
                return conn.Value;
            }
        }

        static void Main(string[] args)
        {
            for (int i = 0; i < 3; i++)
            {
                Task.Run(() =>
                    {
                        var db = SafeConn.GetDatabase();
                        Console.WriteLine(i);

                        string key = "key" + i;

                        db.StringSet(key, i);
                        Thread.Sleep(10);
                        string value = db.StringGet(key);
                    }
                );
            }

            Console.WriteLine("Done");
            Console.ReadLine();
        }

If you run this, you’ll find that there are now only two physical connections generated by the application, which is normal.

redis-safe-connections