The Dangers of async void Event Handlers

When using async/await, you’ll want to use async Task methods most of the time, and use async void methods only for event handlers (see “Async/Await – Best Practices in Asynchronous Programming“, by Stephen Cleary, MSDN Magazine, March 2013).

This conventional wisdom works great if you’re building something like a WPF (GUI) application, and event handlers are invoked occasionally as a result of user interaction (e.g. user presses a button, and an event fires). However, there is another class of event handlers that are invoked as part of a dispatcher loop in a third-party library. async void can be pretty dangerous in these cases.

async void Event Handlers in RabbitMQ

Let’s take the example of RabbitMQ. We’ll set up a basic publisher and consumer. A fundamental property of message queues is that messages are delivered in order, and that’s what we expect to happen.

First, install the RabbitMQ Client library via NuGet:

Install-Package RabbitMQ.Client

Then, we can set up a basic publisher and consumer:

        static void Main(string[] args)
        {
            Console.Title = "async RabbitMQ";

            var factory = new ConnectionFactory();

            using (var connection = factory.CreateConnection())
            using (var channel = connection.CreateModel())
            {
                const string queueName = "testqueue";

                // create queue if not already there

                channel.QueueDeclare(queueName, true, false, false, null);

                // publish

                var props = channel.CreateBasicProperties();
                    
                for (int i = 0; i < 5; i++)
                {
                    var msgBytes = Encoding.UTF8.GetBytes("Message " + i);
                    channel.BasicPublish("", queueName, props, msgBytes);
                }

                // set up consumer

                var consumer = new EventingBasicConsumer(channel);
                consumer.Received += Consumer_Received;
                channel.BasicConsume("testqueue", true, consumer);

                Console.ReadLine();
            }
        }

Our consumer will call the Consumer_Received event handler whenever a message is received. This is the first version of the event handler:

        private static void Consumer_Received(object sender, BasicDeliverEventArgs e)
        {
            var body = e.Body;
            var content = Encoding.UTF8.GetString(body);

            Console.WriteLine("Began handling " + content);

            Thread.Sleep(1000);

            Console.WriteLine("Finished handling " + content);
        }

If we run this now, the messages are processed one at a time and in order just a we expect:

rabbitmq-async-sync

Now, let’s change the event handler to an asynchronous one:

        private static async void Consumer_Received(object sender, BasicDeliverEventArgs e)
        {
            var body = e.Body;
            var content = Encoding.UTF8.GetString(body);

            Console.WriteLine("Began handling " + content);

            await Task.Delay(1000);

            Console.WriteLine("Finished handling " + content);
        }

If we run this now…

rabbitmq-async-async

…we see that our concurrency and ordering guarantees have just gone out the window.

Understanding async void, Revisited

In my recent article, “In-Depth Async in Akka .NET: Why We Need PipeTo()“, I explained what happens when you call async void methods. Let’s recap that.

Say we have this program. We’re calling an async void method in a loop.

        static void Main(string[] args)
        {
            Console.Title = "async void";

            for (int i = 0; i < 5; i++)
                RunJob("Job " + i);

            Console.ReadLine();
        }

        static async void RunJob(string str)
        {
            Console.WriteLine("Start " + str);

            await Task.Delay(1000);

            Console.WriteLine("End " + str);
        }

When you call an async void method, it’s done in a fire-and-forget manner. The caller has no way of knowing whether or when the operation ended, so it just resumes execution immediately, rather than waiting for the async void method to finish. So you end up with parallel and interleaved execution such as this:

rabbitmq-async-asyncvoid

If we change RunJob() to be synchronous instead…

        static void RunJob(string str)
        {
            Console.WriteLine("Start " + str);

            Thread.Sleep(1000);

            Console.WriteLine("End " + str);
        }

…you’ll see that everything happens one at a time and in order:

rabbitmq-async-sync2

So you have to be really careful when using async void:

  1. There is no way for the caller to await completion of the method.
  2. As a result of this, async void calls are fire-and-forget.
  3. Thus it follows that async void methods (including event handlers) will execute in parallel if called in a loop.
  4. Exceptions can cause the application to crash (see the aforementioned article by Stephen Cleary for more on this).

Fixing async void event handlers

Despite these problems, if you want to await in your event handler, you have to make it async void. To prevent parallel and interleaved execution, you have to lock. However, you can’t await in a lock block, so you need to use a different synchronisation mechanism such as a semaphore.

My own Dandago.Utilities provides a ScopedAsyncLock that allows you to neatly wrap the critical section in a using block:

        private static ScopedAsyncLockFactory factory = new ScopedAsyncLockFactory();

        private static async void Consumer_Received(object sender, BasicDeliverEventArgs e)
        {
            using (var scopedLock = await factory.CreateLockAsync())
            {
                var body = e.Body;
                var content = Encoding.UTF8.GetString(body);

                Console.WriteLine("Began handling " + content);

                await Task.Delay(1000);

                Console.WriteLine("Finished handling " + content);
            }
        }

Like this, messages are consumed one at a time, and in order:

rabbitmq-async-scopedasynclock

ScopedAsyncLockFactory uses a semaphore underneath, so don’t forget to dispose it!

Getting Started with Microsoft Orleans

Microsoft Orleans is an actor model implementation for the .NET platform. The main idea of an actor model is that you distribute your computation across a number of fine-grained elements (called actors, or in Orleans, grains) which communicate between each other using message passing, and can each execute only one message at a time. This altogether avoids any need for multithreading synchronisation, and promotes distribution of computation even across physical machines.

The Orleans team have gone to great lengths to develop an API that is friendly to .NET developers, and which hides much of the complexity of building distributed systems. In this article, I hope to get you up and running with Orleans in no time.

The source code for the latter part of this article is available at the Gigi Labs BitBucket repository.

The Visual Studio Extension

If you download and install the Microsoft Orleans Tools for Visual Studio extension, you’ll get a few project templates that you can use to easily create new Orleans projects:

orleans-setup-templates

This is not strictly necessary, and I will show you how to create Orleans projects without it. But it’s quite convenient to have.

The Dev/Test Host project

A really quick way to start playing with Microsoft Orleans is to create a project of type Orleans Dev/Test Host. This is one of the project types provided by the Visual Studio extension (see previous section), and it will generate a bunch of infrastructure code (Program.cs and OrleansHostWrapper.cs) that you can use as-is. In fact, just go ahead and run it:

orleans-setup-quickrun

Just give it a few seconds; Orleans will perform some initialisation tasks, and then tell you that you have a silo running. Orleans applications typically involve a client (a regular console application, Web API, etc which communicates with Orleans) and a server (known as a silo, where all the Orleans magic happens). The Orleans Dev/Test Host is simply a way of having both in the same project.

Hello World

Among the code generated with the Orleans Dev/Test Host template, you’ll find a Main() method that looks something like this:

        static void Main(string[] args)
        {
            // The Orleans silo environment is initialized in its own app domain in order to more
            // closely emulate the distributed situation, when the client and the server cannot
            // pass data via shared memory.
            AppDomain hostDomain = AppDomain.CreateDomain("OrleansHost", null, new AppDomainSetup
            {
                AppDomainInitializer = InitSilo,
                AppDomainInitializerArguments = args,
            });

            var config = ClientConfiguration.LocalhostSilo();
            GrainClient.Initialize(config);

            // TODO: once the previous call returns, the silo is up and running.
            //       This is the place your custom logic, for example calling client logic
            //       or initializing an HTTP front end for accepting incoming requests.

            Console.WriteLine("Orleans Silo is running.\nPress Enter to terminate...");
            Console.ReadLine();

            hostDomain.DoCallBack(ShutdownSilo);
        }

We can put some of our own code instead of that comment. For this, we’ll need to create an interface:

    public interface IPerson : IGrainWithStringKey
    {
        Task SayHelloAsync(string message);
    }

One important thing to note at this stage is that interfaces used by Orleans grains must always return a Task (or Task<T>), because Orleans enforces asynchrony on its actors/grains.

Another thing to note is the IGrainWithStringKey interface we’re extending. All grains have an identifier (that’s how we’re able to find them from our code); identifiers may be of type string (as in this case), Guid, long, or a special compound type.

Next, we can create a class that implements this interface:

    public class PersonGrain : Grain, IPerson
    {
        public Task SayHelloAsync(string message)
        {
            string name = this.GetPrimaryKeyString();
            Console.WriteLine($"{name} says: {message}");

            return TaskDone.Done;
        }
    }

We’re deriving our class from Orleans’ Grain type, and we’re implementing the interface we just declared. We’re getting the grain’s identifier using the GetPrimaryKeyString() method before writing out a message. The TaskDone.Done is simply a utility task that you can return from non-async methods. Similar to the Task.CompletedTask that was introduced in .NET 4.6, this is just a convenient way to avoid doing something like Task.FromResult(0) all the time.

With that done, all we need to do is replace the comment in Main() with code such as the following:

            var joe = GrainClient.GrainFactory.GetGrain<IPerson>("Joe");
            joe.SayHelloAsync("Hello!");

This gets an instance of a grain that implements IPerson, with identifier “Joe”. The grain will be created if it doesn’t already exist; Orleans abstracts out the creation and destruction of actors. Once we have an instance, we can call methods on it. They are all asynchronous, but whether you will await them depends very much on the type of application you will be writing.

orleans-setup-hello

This is already enough for you to go on experimenting with Orleans. In the next sections (which you can safely skip if you’re just starting out), I will explain how to build out a proper project structure.

Grain Class and Grain Interface projects

There are two other project types you get with the Visual Studio extension: Orleans Grain Class Collection, and Orleans Grain Interface Collection. As your project starts to grow, you should move your grain classes and interfaces out into these project types, so that your main application consists solely of client code that interacts with Orleans via the GrainClient.GrainFactory.

NuGet packages

If you don’t want to use the Visual Studio extension, you can create regular class libraries or console applications, and just add in the necessary NuGet packages. These are the projects you need to create and the NuGet packages you need to install for each:

Purpose Type NuGet package References
Client Console Application Microsoft.Orleans.Client Interfaces
Silo Console Application Microsoft.Orleans.Server Interfaces, Grains
Grains Class Library Microsoft.Orleans.Core
Microsoft.Orleans.OrleansCodeGenerator.Build
Interfaces
Interfaces Class Library Microsoft.Orleans.Core
Microsoft.Orleans.OrleansCodeGenerator.Build

While the Microsoft.Orleans.OrleansCodeGenerator.Build package is not strictly necessary, it enables code generation for grains and grain interfaces at build time, which is better than having to do it at runtime.

A More Complete Project Structure

The rest of what we need to do to make this application work (whether using the Visual Studio extension or not) is described in the Minimal Orleans Application tutorial. I’m going to adapt it so that we have separate projects for client and silo/host (which is usually the case in real applications), to use XML configuration throughout, to fit our earlier IPerson example, and to retry to connect from the client to the silo until the silo is up.

The Silo needs the following Garbage Collection configuration to go into its App.config:

<configuration>
  <runtime>
    <gcServer enabled="true"/>
    <gcConcurrent enabled="false"/>
  </runtime>
</configuration>

Add a file called OrleansConfiguration.xml to the Silo and set it to Copy always:

<?xml version="1.0" encoding="utf-8"?>
<OrleansConfiguration xmlns="urn:orleans">
  <Globals>
    <SeedNode Address="localhost" Port="11111" />
  </Globals>
  <Defaults>
    <Networking Address="localhost" Port="11111" />
    <ProxyingGateway Address="localhost" Port="30000" />
  </Defaults>
</OrleansConfiguration>

Silo configuration needs (at a minimum) two things: an endpoint for inter-silo communication (that’s the Networking element) and an endpoint that clients outside the silo can use to talk to it (ProxyingGateway). In other words, our Client needs to talk to the Silo on port 30000.

In fact, we will now add a ClientConfiguration.xml file to the Client (set this also to Copy always):

<ClientConfiguration xmlns="urn:orleans">
  <Gateway Address="localhost" Port="30000"/>
</ClientConfiguration>

The PersonGrain class we saw earlier needs to go in the Grains project, and the IPerson interface needs to go in the Interfaces project.

Our program logic will look something like this (basically adapted from the Minimal Orleans Application tutorial, but without the client code):

    class Program
    {
        static SiloHost siloHost;

        static void Main(string[] args)
        {
            Console.Title = "Silo";

            // Orleans should run in its own AppDomain, we set it up like this
            AppDomain hostDomain = AppDomain.CreateDomain("OrleansHost", null,
                new AppDomainSetup()
                {
                    AppDomainInitializer = InitSilo
                });

            Console.WriteLine("Orleans Silo is running.\nPress Enter to terminate...");
            Console.ReadLine();

            // We do a clean shutdown in the other AppDomain
            hostDomain.DoCallBack(ShutdownSilo);
        }

        static void InitSilo(string[] args)
        {
            siloHost = new SiloHost(Dns.GetHostName());
            siloHost.LoadOrleansConfig();

            siloHost.InitializeOrleansSilo();
            var startedok = siloHost.StartOrleansSilo();
            if (!startedok)
                throw new SystemException(String.Format("Failed to start Orleans silo '{0}' as a {1} node", siloHost.Name, siloHost.Type));
        }

        static void ShutdownSilo()
        {
            if (siloHost != null)
            {
                siloHost.Dispose();
                GC.SuppressFinalize(siloHost);
                siloHost = null;
            }
        }
    }

Program logic in the Client should look something like this:

        static void Main(string[] args)
        {
            Console.Title = "Client";

            var config = ClientConfiguration.LoadFromFile("ClientConfiguration.xml");

            while (true)
            {
                try
                {
                    GrainClient.Initialize(config);
                    Console.WriteLine("Connected to silo!");

                    var friend = GrainClient.GrainFactory.GetGrain<IPerson>("Joe");
                    var result = friend.SayHelloAsync("Hello!");

                    Console.ReadLine();
                    break;
                }
                catch (SiloUnavailableException)
                {
                    Console.WriteLine("Silo not available! Retrying in 3 seconds.");
                    Thread.Sleep(3000);
                }
            }
        }

Set both the Silo and Client as startup projects, and run them. The silo will take a few seconds to initialise, and the client will keep trying to reach it until it’s up:

orleans-setup-dual

Distributed ID Generation with RustFlakes

Who would have thought that generating identifiers for data is hard? In line of business applications, most of the time it is sufficient to simply rely on a database’s autoincrement functionality to generate new primary keys. However, as a system scales and inevitably evolves into a distributed architecture, the database can quickly become a bottleneck.

idgen1_

This can easily be a problem when you’re accepting hideous amounts of user-generated content like Facebook and Twitter do. However, it is even more of a burden when you’re receiving data from a provider and need to map their IDs to yours. Each time you receive data, you need to check if their ID is mapped, and if not, you have to generate an ID – so you have to hit the database table twice. You can use a stored procedure to perform this logic in one round trip, but you’ll still have a performance problem as contention on the table increases.

idgen2_

Jeremiah Peschka’s 2012 article, “The Trouble with Keys“, is an excellent summary of why distributed ID generation is hard. In short, generating IDs in a database causes a bottleneck, whereas using something like GUIDs (aside from being longer and a little more inconvenient to pass around e.g. in a URL, and the rare but real potential for collisions) will cause performance issues at the database due to their random nature, because hard disks work best with sequential data.

The solution he proposes is to use a combination of timestamp, worker/machine identifier and sequence number:

idgen3_

The timestamp is the number of milliseconds since an arbitrary time in the past. The worker identifier is anything that is different for each running instance of the ID generator; it could be a MAC address (assuming you’re running not more than one per machine), or even just an arbitrary number. The sequence number is a small (8- or 16-bit) counter that resets every millisecond. Thus, each ID you generate will be unique at different points in time (because of the combination of timestamp and sequence number) and at different machines (because of the worker identifier). The resulting ID is a 128-bit number, which fits in the C# decimal data type and SQL Server’s DECIMAL data type.

Jeremiah Peschka created a library called Rustflakes which generates ID according to this concept. It is based on earlier implementations of the same algorithm by Twitter and Boundary. You can install Rustflakes via NuGet:

Install-Package RustFlakes

The example at the RustFlakes homepage is obsolete and does not work, but this little code will quickly get you generating IDs in no time:

            var workerId = new byte[] { 0, 1, 2, 3, 4, 5 };
            var idGenerator = new BigIntegerOxidation(workerId);
            var id = idGenerator.Oxidize();

The naming is a little bit funny, but basically you create an oxidation (ID generator) instance, and you call Oxidize() to generate the IDs.

If you want to use a MAC address for the workerId, you can use this code adapted from this StackOverflow answer:

            var macAddr = NetworkInterface.GetAllNetworkInterfaces()
                .Where(nic => nic.OperationalStatus == OperationalStatus.Up)
                .Select(nic => nic.GetPhysicalAddress().GetAddressBytes())
                .FirstOrDefault();

            var workerId = macAddr;

The BigIntegerOxidation is an ID generator that takes a 6-byte worker ID as input and generates a huge ID. However, there are others:

Generator Input (Worker) ID Output (Generated) ID Timestamp portion
BigIntegerOxidation byte[] System.Numerics.BigInteger 64-bit
DecimalOxidation uint decimal 48-bit
SqlServerBigIntOxidation ushort long 32-bit
UInt64Oxidation ushort ulong 32-bit

You can see the difference between these generators by generating a lot of IDs in a loop:

            var idGenerator = new BigIntegerOxidation(workerId);
            //var idGenerator = new DecimalOxidation(1);
            //var idGenerator = new SqlServerBigIntOxidation(1);
            //var idGenerator = new UInt64Oxidation(1);

            while (true)
            {
                var id = idGenerator.Oxidize();
                Console.WriteLine(id);
            }

Now, when you’re picking which ID generator to use, you have to be careful, because for each of these, there is a catch. The most important thing to keep in mind is that the shorter (in bits) the ID, particularly its timestamp portion, the sooner it will run out. If you run the above loop and watch the rate at which the digits turn, you can make a rough calculation of how long it will last – but keep in mind that this is relative to the epoch, which in Rustflakes is 1st January 2013.

  • BigIntegerOxidation generates numbers that internally use 13 bytes (104 bits). Although technically these should fit in a 128-bit decimal, in practice they don’t, because their actual value is greater than decimal.MaxValue. You can pass in an arbitrary epoch to the BigIntegerOxidation constructor which will make the number smaller and it will possibly convert, but you run a serious risk of exceeding decimal.MaxValue fast (the default epoch is just under 4 years old, and we don’t know how long ago decimal.MaxValue was exceeded).
  • DecimalOxidation generates 128-bit decimals, but they’re internally generated as only 96-bit values. This should give you several years of operation, but it’s less than what you’d expect with the full 128 bits.
  • UInt64Oxidation and SqlServerBigIntOxidation have only a 32-bit timestamp portion. These will likely restart in a matter of weeks.

Dependency Injection in Microsoft Orleans

The source code for this article is available at the Gigi Labs BitBucket repository.

Microsoft Orleans added support for dependency injection in version 1.1.0, which was released in December 2015. It is based on ASP .NET Core’s style of dependency injection.

To see how this works, let’s create an interface and a class that implements it. It is not important what they do, but we will use them as dependencies.

    public interface IWhatever
    {
    }

    public class Whatever : IWhatever
    {
    }

In order to set up our dependency injection bindings, we need a special startup class with a ConfigureServices() method with a predefined signature. In our case, it will look like this:

    public class Startup
    {
        public IServiceProvider ConfigureServices(IServiceCollection services)
        {
            services.AddSingleton<IWhatever, Whatever>();

            return services.BuildServiceProvider();
        }
    }

Here we are assuming that our dependency will be in singleton scope, but if you’ve used dependency injection before, you’ll know that there are other possible scopes. In fact, you’ll notice that there are different methods supporting a variety of scopes:

orleans-di-scopes

Once we have a startup class, we need to tell Orleans to use it. This can be done either in XML configuration (refer to the official documentation on Orleans DI to see how to do this) or in code as follows:

            var config = ClusterConfiguration.LocalhostPrimarySilo();
            config.UseStartupType<Startup>();

If you’re using an Orleans Dev/Test Host project type, you’ll find this cluster configuration in OrleansHostWrapper.cs. Note that dependency injection can only be done in the server (silo) and is not currently supported in the client (so you won’t find UseStartupType() on your ClientConfiguration).

Now that we have our dependency all set up, all we need is a Grain that uses it.

    public interface IHello : IGrainWithIntegerKey
    {
        Task HelloAsync();
    }

    public class HelloGrain : Grain, IHello
    {
        private IWhatever whatever;

        public HelloGrain(IWhatever whatever)
        {
            this.whatever = whatever;
        }

        public Task HelloAsync()
        {
            Console.WriteLine("Hello!");
            return TaskDone.Done;
        }
    }

In the main program, let’s create this and call it:

            var grain = GrainClient.GrainFactory.GetGrain<IHello>(0);
            grain.HelloAsync();

Now we can run the program and see that it works without problems:

orleans-di-running

We can verify that dependency injection is working by commenting out the binding configuration in the startup class. Without it, Orleans cannot activate the grain because it cannot resolve the dependency:

orleans-di-failed