Working with Asynchronous Methods in C#

In yesterday’s article, “Motivation for async/await in C#“, we have seen why asynchronous programming is important. We have also seen basic usage of the await keyword, which requires its containing method to be marked as async.

When learning to write asynchronous methods, it is not trivial to get the interactions between various methods (which may or may not be asynchronous) right. In fact, the examples in yesterday’s article which use an async void method should normally only be used in event handlers, and even so, there are caveats to consider.

In this article, we’ll go through various different scenarios in which async/await can be used.

async Task methods

Let’s take another look at the asynchronous (event handler) method from yesterday’s article:

        private async void Button_Click(object sender, RoutedEventArgs e)
        {
            var baseAddress = new Uri("http://mta.com.mt");

            using (var httpClient = new HttpClient() { BaseAddress = baseAddress })
            {
                var response = await httpClient.GetAsync("/");
                var content = await response.Content.ReadAsStringAsync();

                MessageBox.Show("Response arrived!", "Slow website");
            }
        }

Try moving out the code into a separate method, and awaiting it from the event handler. You’ll find that you can’t await an async void method:

In order to be able to await a method, it must return Task (if it doesn’t need to return anything, such as void methods) or Task<T> (if it needs to return a value of type T). We also append an –Async suffix to the method name by convention to make it obvious for people who use such methods that they’re meant to be awaited.

Thus, this example becomes:

        private async void Button_Click(object sender, RoutedEventArgs e)
        {
            await GetHtmlAsync();
        }

        private async Task GetHtmlAsync()
        {
            var baseAddress = new Uri("http://mta.com.mt");

            using (var httpClient = new HttpClient() { BaseAddress = baseAddress })
            {
                var response = await httpClient.GetAsync("/");
                var content = await response.Content.ReadAsStringAsync();

                MessageBox.Show("Response arrived!", "Slow website");
            }
        }

This is an example of an async Task method, which does not return anything. Let’s change it such that it returns the HTML from the response:

        private async void Button_Click(object sender, RoutedEventArgs e)
        {
            string html = await GetHtmlAsync();
        }

        private async Task<string> GetHtmlAsync()
        {
            var baseAddress = new Uri("http://mta.com.mt");

            using (var httpClient = new HttpClient() { BaseAddress = baseAddress })
            {
                var response = await httpClient.GetAsync("/");
                var content = await response.Content.ReadAsStringAsync();

                return content;
            }
        }

We’ve changed the signature of GetHtmlAsync() to return Task<string> intead of just Task. Correspondingly, we are now returning content (a string) from the method. At the event handler, we are now assigning the result of the await into the html variable. Apart from waiting asynchronously until the method completes, await has the additional function of unwrapping the result from the Task that contains it; thus html is of type string.

If you try removing the async keyword from GetHtmlAsync(), you’ll learn a little more about the actual function of the async keyword:

Without async, you are expected to return what the method advertises: a Task<string>. On the other hand, if you mark the method as async, the meaning of the method is changed such that you can return a string directly. The underlying Task-related plumbing is handled by the compiler.

Chaining Asynchronous Methods

In the previous section, we have seen how methods need to return a Task in order to be awaited. Typically, one async Task method will call another and await its result.

The chain of calls ends at a last async Task, typically provided by the .NET Framework or other library, which interfaces directly with I/O (e.g. network or filesystem). It must be an asynchronous method; attempting to disguise a blocking call as async here will lead to deadlocks.

async Task may (and should) be used all the way from an incoming request to the final I/O library method in application types that support top-level asynchronous methods, such as Web API or WCF.

The situation is a little different for other applications (e.g. Windows Forms, WPF) that are event-driven. Asynchronous event handlers are a special case where we need to use async void methods, as we have already seen in the WPF example from yesterday’s article:

async void methods

Event handlers are void methods that are called by the runtime in a dispatcher loop. Thus, async void methods are necessary to allow usage of await within event handlers without requiring them to return a Task

However, as we have seen before, there is no way to await an async void method. This makes async void methods very dangerous to use outside of their intended context, as I have detailed in “The Dangers of async void Event Handlers“. This is one of the more common mistakes when programming with async/await, and it is good to become familiar with the problems in order to avoid repeating the same mistakes in future.

Fake Asynchronous Methods

Sometimes, you’ll have an interface that requires asynchronous methods, yet your implementation does not need anything asynchronous in it. Let’s look at a practical example:

    public interface ISimpleStorage
    {
        Task WriteAsync(string str);
        Task<string> ReadAsync();
    }

You could implement this interface using a simple file as a backing store, in which case your methods will be suitably asynchronous:

    public class FileStorage : ISimpleStorage
    {
        public async Task<string> ReadAsync()
        {
            using (var fs = File.OpenRead("file.txt"))
            using (var sr = new StreamReader(fs))
            {
                var str = await sr.ReadToEndAsync();
                return str;
            }
        }

        public async Task WriteAsync(string str)
        {
            using (var fs = File.OpenWrite("file.txt"))
            using (var sw = new StreamWriter(fs))
            {
                await sw.WriteAsync(str);
                await sw.FlushAsync();
            }
        }
    }

However, you could have another implementation which just uses memory as storage, and in this case there’s nothing asynchronous:

    public class MemoryStorage : ISimpleStorage
    {
        private string str;

        public Task<string> ReadAsync()
        {
            return Task.FromResult(str);
        }

        public Task WriteAsync(string str)
        {
            this.str = str;
            return Task.CompletedTask;
        }
    }

In that case, your methods need not be marked async. However, this means that you will actually need to return Task instances from each method. If you have nothing to return, then just return a Task.CompletedTask (available from .NET Framework 4.6 onwards). You can also use Task.FromResult() to construct a task from a variable that you want to return.

Simplifying Single Line Asynchronous Methods

Consider the following asynchronous method:

        static async Task WaitALittleAsync()
        {
            await Task.Delay(10000);
        }

Here, we are waiting for the delay to finish, and then returning the result.

Instead, we can just return the Task itself, and let the caller do the awaiting:

        static Task WaitALittleAsync()
        {
            return Task.Delay(10000);
        }

Once again, a method does not need to be marked async if (a) it does not await anything, and (b) it can return a Task, rather than some other type that needs to be wrapped in a Task.

Using async/await in Main()

Until recently, you couldn’t use async/await in a console application’s Main() method. You can have an async Task Main() method as from C# 7.1, but you need to make sure you’re using C# 7.1 first from your project properties -> Build -> Advanced…:

You can choose either “C# 7.1”, or “C# latest minor version (latest)”. With that, you can await directly from within Main():

        static async Task Main(string[] args)
        {
            await WaitALittleAsync();
        }

Before C# 7.1, you had to use some other workaround to await from Main(). One option is to use a special AsyncContext such as the one written by Stephen Cleary. Another is to just move asynchrony out of Main() and use a Console.ReadLine() to keep the window open:

        static void Main(string[] args)
        {
            Run();
            Console.ReadLine();
        }

        static async void Run()
        {
            await WaitALittleAsync();
        }

Summary

  1. Use async Task methods wherever you can.
  2. Use async void methods for event handlers, but beware the dangers even there.
  3. Use the –Async suffix in asynchronous method names to make them easily recognisable as awaitable.
  4. Use Task.CompletedTask and Task.FromResult() to satisfy asynchronous contracts with synchronous code.
  5. Asynchronous code in Main() has not been possible until recently. Use C# 7.1+ or a workaround to get this working.

Motivation for async/await in C#

Introduction

In .NET 4.5, the async and await keywords were introduced. They are now a welcome and ubiquitous part of the .NET developer’s toolkit. However, I often run into developers who are not familiar with this concept, and I have had to explain async/await many times. Thus, I felt it would make sense to write up this explanation so that it can be accessible to all.

async/await is really just syntactic sugar for task continuations, but it makes asynchronous, I/O-bound code very elegant to work with. In fact, it is so popular that it has been spreading to other languages such as JavaScript.

As far as I can remember, async/await was introduced mainly to facilitate building Windows Phone applications. Unlike desktop applications, Windows Phone had introduced a hard limit in which no method could block for more than 50ms. Thus it was no longer possible for developers to resort to blocking their UI, and the need arose to use asynchronous paradigms, for which tasks are a very powerful, but sometimes tedious, option. async/await makes asynchronous code look almost identical to synchronous code.

In this article, I will not go into detail about using Tasks in .NET (which deserves a whole series of articles in itself), but I will give a very basic explanation on how to use them for asynchronous processing. More advanced usage is left for future articles.

Sidenote: Rejoice, for this is the 200th article published on Gigi Labs!

Motivation

Although async/await can be used in all sorts of applications, I like to use WPF applications to show why it’s important. You don’t need to know WPF for this; just follow along.

Let’s create a new WPF application, and simply drag a button from the Toolbox onto the WPF window:

Double-clicking that button will cause a click event handler to be generated in the codebehind class (MainWindow.xaml.cs):

        private void Button_Click(object sender, RoutedEventArgs e)
        {

        }

Now, let’s be evil and toss a Thread.Sleep() in there:

        private void Button_Click(object sender, RoutedEventArgs e)
        {
            Thread.Sleep(15000);
        }

Run the application without debugging (Ctrl+F5) and click the button. Try to interact with the window (e.g. drag it around). What happens?

The UI is completely frozen; you can’t click the button again, drag the window, or close it. That’s because we’ve done something very, very stupid. We have blocked the application’s UI thread.

Let us now replace the Thread.Sleep() with this Task.Delay() instead (notice there is also that async in the method signature, and it’s important):

        private async void Button_Click(object sender, RoutedEventArgs e)
        {
            await Task.Delay(15000);
        }

If you run the application now, you’ll find that you can interact with the window after clicking the button, even though the code appears to be doing practically the same thing as before. So what changed?


Image taken from: Asynchronous Programming with Async and Await (C# and Visual Basic)

The way control flow works with async/await is explained in detail on MSDN, but it may feel a little overwhelming if you’re learning this the first time. Essentially, to understand what is happening, we need to break that await Task.Delay(15000) line into the following steps:

  1. Task.Delay(15000) executes, returning a Task representing it. It does not block since it does not necessarily execute on the same thread.
  2. Because of the await, execution of the remainder of the method is temporarily suspended. In terms of method execution, it’s as if we’re blocking.
  3. Eventually, the Task finishes executing. The await part is fulfilled, and the remainder of the method can resume execution.

async/await with HttpClient

Using a simple delay is a nice way to get an initial feel of async/await, but it is also not very realistic. async/await works best when you are dealing with I/O requests such as sockets, databases, NoSQL storage, files, REST, etc. To this effect, let’s adapt our example to use an HttpClient.

First, let’s install the following package via NuGet:

Install-Package Microsoft.Net.Http

We’re going to repeat the same experiment as before: first we’ll do a silly blocking implementation, and then we’ll make it asynchronous.

Let’s work with this code that gets the response from the Google homepage:

        private void Button_Click(object sender, RoutedEventArgs e)
        {
            var baseAddress = new Uri("http://www.google.com");

            using (var httpClient = new HttpClient() { BaseAddress = baseAddress })
            {
                var response = httpClient.GetAsync("/").Result;
                var content = response.Content.ReadAsStringAsync().Result;
            }
        }

If we run this and click the button, the window blocks for a fraction of a second, and we get back the HTML from Google:

Google has a very fast response time, so it is a poor example for visualising the problems of blocking code in a UI. Instead, let’s use a website that takes a very long time to load. One that I’ve written about before is that of the Malta Tourism Authority, which takes over 20 seconds to load. Let’s change our base address in the code:

            var baseAddress = new Uri("http://mta.com.mt");

If you run the application now and click the button, you’ll see that it will be stuck for a fairly long time, just like before. We can sort this out by making the request asynchronous. To do that, we replace each .Result property with an await. In order to use await, we have to mark the method as async. In order to get a notification when the response actually comes back, I’ll also toss in a message box:

        private async void Button_Click(object sender, RoutedEventArgs e)
        {
            var baseAddress = new Uri("http://mta.com.mt");

            using (var httpClient = new HttpClient() { BaseAddress = baseAddress })
            {
                var response = await httpClient.GetAsync("/");
                var content = await response.Content.ReadAsStringAsync();

                MessageBox.Show("Response arrived!", "Slow website");
            }
        }

Run the application and click the button. You can interact with the window while the request is processed, and when the response comes back, you’ll get a notification:

If you put a breakpoint and follow along line by line, you’ll see that everything happens sequentially: the content line will not execute before the response has arrived. So async/await is really just giving us the means to make asynchronous code look very similar to normal sequential execution logic, hiding the underlying complexities. However, as we’ll see in later articles, it is also possible to use this abstraction for more powerful scenarios such as parallel task execution.

The Importance of Asynchronous Code

While I have not really explained how to work with async/await in any reasonable depth yet, the importance should now be evident: making I/O calls asynchronous means that your thread does not need to block, and can be freed to do other work. While this has a great impact on user experience in GUI applications, it is also fundamental in other applications such as APIs. Under high load, it is possible to end up with thread starvation when blocking, because all threads are stuck waiting for I/O, when they could otherwise be processing requests.

For pure asynchronous code, blocking isn’t actually ever necessary. When an I/O request such as HttpClient.GetAsync() is fired, .NET uses something called I/O Completion Ports (IOCP) which can monitor incoming I/O and trigger the caller to resume when ready.

We have merely scratched the surface here. There is a lot to be said about async/await, and likewise there are a lot of pitfalls that one must be made aware of. This article has shown why asynchronous code is important, and future articles may cover different aspects in more detail.

The Adapter Design Pattern for DTOs in C#

In this article, we discuss the Adapter design pattern, which is part of the book “Design Patterns: Elements of Reusable Object-Oriented Software” by Gamma et al. (also known as the Gang of Four).

We will discuss the motivation for this design pattern (including common misconceptions) and see a number of different ways in which this pattern is implemented in C#.

TL;DR Use extension methods for your mapping code.

Defining the Interface of a Class

The above book summaries the Adapter pattern thusly:

“Convert the interface of a class into another interface clients expect. Adapter lets classes work together that couldn’t otherwise because of incompatible interfaces.”

It is fundamental to interpret this description in the context in which it was written. The book was published in 1994, before Java and similar languages even existed (in fact, the examples are in C++). Back then, “the interface of a class” simply meant its public API, to which extent “client code” could use it. C++ has no formal interface construct as in C# and Java, and the degree of encapsulation is dictated as a result of what the class exposes publicly. Note that a class’s interface is not restricted to its public methods alone; C++ also offers other devices external to the class itself, and so does C# (extension methods, for instance).

Given today’s languages where interfaces are formal language constructs, such an interpretation has mostly been forgotten. For instance, the 2013 MSDN Magazine article “The Adapter Pattern in the .NET Framework” illustrates the Adapter design pattern in terms of C# interfaces, but it slightly misses the point. We can discuss the Adapter pattern without referring to C# interfaces at all, and instead focus on Data Transfer Objects (DTOs). DTOs are simply lightweight classes used to carry data, often between remote applications.

An Example Scenario

Let’s say we have this third party class with a single method:

    public class PersonRepository
    {
        public void AddPerson(Person person)
        {
            // implementation goes here
        }
    }

The interface of this class consists of the single AddPerson() method, but the Person class that it expects as a parameter is also part of that interface. There is no way that the third party could give us the PersonRepository (e.g. via a NuGet package) without also including the Person class.

It is often the case that we may have a different Person class of our own (we’ll call it OurPerson), but we can’t pass it to AddPerson(), because it expects its own Person class:

    public class Person // third party class
    {
        public string FullName { get; set; }

        public Person(string fullName)
        {
            this.FullName = fullName;
        }
    }

    public class OurPerson // our class
    {
        public string FirstName { get; set; }
        public string LastName { get; set; }

        public OurPerson(string firstName, string lastName)
        {
            FirstName = firstName;
            LastName = lastName;
        }
    }

Thus, we need a way to transform an OurPerson instance into a Person. For that, we need an Adapter. Next, we’ll go through a few ways in which we can implement this.

Constructor Adapter

One way of creating a Person from an OurPerson is to add a constructor in Person which handles the conversion:

    public class Person // third party class
    {
        public string FullName { get; set; }

        public Person(string fullName)
        {
            this.FullName = fullName;
        }

        public Person(OurPerson ourPerson)
        {
            this.FullName = $"{ourPerson.FirstName} {ourPerson.LastName}";
        }
    }

It is not hard to see why this is a really bad idea. It forces Person to have a direct dependency on OurPerson, so anyone shipping PersonRepository would now need to ship an additional class that may be domain-specific and might not belong in this context. Even worse, this is not possible to achieve if the Person class belongs to a third party and we are not able to modify it.

Wrapper

Another approach (which the aforementioned book describes) is to implement OurPerson in terms of Person. This can be done by subclassing, if the third party class allows it:

    public class OurPerson : Person // our class
    {
        public string FirstName { get; set; }
        public string LastName { get; set; }

        public OurPerson(string firstName, string lastName)
            : base($"{firstName} {lastName}")
        {
            FirstName = firstName;
            LastName = lastName;
        }
    }

Where C# interfaces are involved, an alternative approach to inheritance is composition. OurPerson could contain a Person instance and expose the necessary methods and properties to implement its interface.

The disadvantage of either of these two approaches is that they make OurPerson dependent on Person, which is the opposite of the problem we have seen in the previous approach. This dependency would be carried to wherever OurPerson is used.

Especially when dealing with third party libraries, it is usually best to map their data onto our own objects. Any changes to the third party classes will thus have limited impact in our domain.

AutoMapper

A lot of people love to use AutoMapper for mapping DTOs. It is particularly useful if the source and destination classes are practically identical in terms of properties. If they’re not, you’ll have to do a fair amount of configuration to tell AutoMapper how to construct the destination properties from the data in the source.

Personally, I’m not a fan of AutoMapper, for the following reasons:

  • The mapping occurs dynamically at runtime. Thus, as the DTOs evolve, you will potentially have runtime errors in production. I prefer to catch such issues at compile-time.
  • Writing AutoMapper configuration can get very tedious and unreadable, often more so than it would take to map DTOs’ properties manually.
  • You can’t do anything asynchronous in AutoMapper configuration. While this may sound bizarre, I’ve needed this in the past due to terrible DTOs provided by a third party provider.

Standalone Adapter

The aforementioned MSDN Magazine article simply uses a separate class to convert from source to destination. Applied to our example, this could look like this:

    public class PersonAdapter
    {
        public Person ConvertToPerson(OurPerson person)
        {
            return new Person($"{person.FirstName} {person.LastName}");
        }
    }

We may then imagine its usage as such:

            var ourPerson = new OurPerson("Chuck", "Berry");
            var adapter = new PersonAdapter();
            var person = adapter.ConvertToPerson(ourPerson);

This approach is valid, and does not couple the DTOs together, but it is a little tedious in that you may have to create a lot of adapter classes, and subsequently create instances whenever you want to use them. You can mitigate this a little by making them static.

Extension Methods

A slight but very elegant improvement over the previous approach is to put mapping code into extension methods.

    public static class OurPersonExtensions
    {
        public static Person ToPerson(this OurPerson person)
        {
            return new Person($"{person.FirstName} {person.LastName}");
        }
    }

The usage is very clean, making it look like the conversion operation is part of the interface of the class:

            var ourPerson = new OurPerson("Chuck", "Berry");
            var person = ourPerson.ToPerson();

This works just as well if you’ve converting from a third party object onto your own class, since extension methods can be used to append functionality even to classes over which you have no control.

Summary

There are many ways to apply the Adapter design pattern in mapping DTOs. Extension methods are the best approach I’ve come across for C# because:

  • They don’t couple the source and destination DTOs.
  • The conversion occurs at compile time, so any repercussions of changes to the DTOs will be caught early.
  • They can easily be used to append functionality even to third party classes for which the source code is not available.

The Adapter design pattern has little to do with interfaces as a formal OOP language construct. Instead, it deals with “the interface of a class”, which is embodied by whatever it exposes publicly. The Adapter design pattern provides a means to work with that interface by converting incompatible objects to ones that satisfy its contract.

On Stack Overflow

love Stack Overflow. As a software developer, I think it is one of the most important and useful resources ever made, helping me quickly resolve countless issues in my work and side projects.

I also hate it with a passion, and think it can be a royal pain in the ass. My main reasons for this stem from the community, especially those with moderation powers, being made up of a bunch of kids (not necessarily in the physical sense) who are addicted to the gamification system and will do anything to ramp up their reputation.

Given that I ask about as many questions as I answer, the result was that I have seen many of my legitimate questions closed and/or downvoted for various reasons such as “not constructive” or “duplicate”. Many times, the people casting such judgement would not actually understand the question.

One of my biggest frustrations with the site came when I asked the question “Task.Factory.FromAsync with CancellationTokenSource“. A troll with very high reputation (read: mod_powers++) marked this question as a duplicate of another question which had nothing to do with mine. When I challenged him, he went as far as to edit the unrelated question such that it did answer my question. He actually admitted this in the comments to my question (which have by now been cleaned up by moderators after I reported the matter, but I have conveniently kept screenshots), and even asked me to offer a bounty so that he could get more points:

As a regular user of Stack Overflow, I was highly put off at not being able to ask any reasonable question without it being closed down.

Answering questions is just as frustrating. Typically, when a question is asked, a barrage of short, two-sentence answers will appear. These will be progressively edited into better answers, but as long as they can get something in initially, they will start accumulating points. This is a common tactic and discourages spending the time to write well-thought answers for relatively little reward. Not that I believe that the reward system is more important than the quality of the questions and answers. But alas, many people do, and that is the reason for this sort of phenomenon.

And you can tell that something is really messed up when you get entire courses, such as this 6-hour Pluralsight course, explaining how to use something that is supposed to be a Q&A site.

But in fact, what I am describing is nothing new, and has been going on for many years. In fact, around 2013, Michael Richter wrote a very informative post about the problems plaguing Stack Overflow. It has disappeared from its original location, but can still be read (along with around a couple of weeks’ worth of comments) thanks to the Internet Archive.

I am reproducing Mr. Richter’s article below for ease of future reference. Aside from the issues with Stack Overflow, I would also like to emphasise the story around his “goto” answer, which reverberates a problem I encounter extremely regularly with developers being too religious about what they have been taught, and not using their head to adequately reason about techniques in the particular context they need.

Without further ado…

“Why I no longer contribute to StackOverflow” by Michael Richter

I was active in the StackOverflow (and the broader Stack Exchange) community for a while. I no longer am. Here’s why.

Introduction

I have an account at StackOverflow. Follow the link and check out a few stats:

  • I’ve been a member for almost four years at the time I’m writing this.
  • I’ve scored over 14,000 points in their gaming system.
  • I’m in the top 3% of their contributors overall as of this writing.

I point this out not to brag but to make sure it’s clear that I’m not writing this because of “sour grapes” or because I’m not getting the recognition I think I deserve. Indeed, to be blunt, I’m getting way too much recognition, but more on that below.

Overview

If you’re a software developer of any kind, and if you haven’t been sleeping under a rock, you know of Joel Spolsky and Jeff Atwood’s StackOverflow (and broader StackExchange ecosystem). For the rest of you, allow me to lift up your rock a little and explain what the site is.

In brief, you ask questions related to programming and other people answer them. The site is “gamified” (what an utterly horrendous neologism, that!) so you score points for asking questions that get voted up, answering questions and getting voted up, for doing book-keeping tasks and a variety of other things. You can also earn “badges” (gold, silver and bronze) for accomplishing certain things.

The intent of the system is to provide some form of recognition for people who help others with software development problems. It sounds innocuous, but in the end I find it vaguely distasteful and off-putting.

I’ve been asked more than a few times now why I think this, so today I will commit my explanations in writing once so I can just point to this page instead of repeatedly saying the same thing.

The problems

The problems I see with StackOverflow are summarized in this list:

  1. Poor pedagogy
  2. Poor reward system
  3. Poor community

I’m willing to engage in discussion of any of these flaws. Comments are turned on for this post. They are, however, moderated so if you plan to simply spew bile at someone who dares say bad things about a site you like you:

  • won’t get your 15 seconds of Innarwebs Fame™®;
  • will add to this counter of people who contribute to point #3 above: count so far: 6

Poor pedagogy

As an educator, I find the StackOverflow approach to “helping” people unproductive and contrary to any kind of real learning. For illustration, go back and take a look at my profile. Specifically take a look at the tags tab. See the #1 tag there? Java. Most of my answers (and most of my points) come from Java-related questions.

Now comes a confession.

I’m not a Java programmer. I’ve only ever briefly programmed in Java professionally. I hated the experience and I hate the language. I certainly don’t consider myself a Java expert. Yet I managed to get the bulk of my points from Java. How is this possible?

It’s possible because I did what many of the people whose questions I answered (and got points for) should have done for themselves: I saw a simple Java question, hit Google, read briefly, then synthesized an original answer.

There’s an old cliché in English: give a man a fish, he eats for a day; teach a man to fish, he eats for a lifetime. StackOverflow is filled to the brim with people giving fishes. The people asking are learning nothing useful beyond the shortest of the short terms and the people teaching are not helping in any but the most trivial of ways. In the long term, I submit, StackOverflow is probably holding back the development of programmers (and thus the entire field of programming).

Poor reward system

Incidentally, my #2 tag is C++. There was a time I would cheerfully have called myself a C++ expert. These days I still likely know more about the language than most people working in it, but I have, if it’s at all possible, an even deeper abiding hatred of the language than I do for Java.

So why, if I hate Java and C++ (and several other languages that score well on my tags list), do I bother answering questions on those topics? This is because of the second problem I have with StackOverflow: the reward system is ludicrously designed.

The high school “cool kids table”

The very nature of StackOverflow’s structure has it such that only those who answer simple questions of the most popular programming languages will get a reward. To illustrate this, look at the difference between this Java-related answer and this non-Java one.

The Java answer scored me 460 and took me probably under a minute to write. The one about operating systems scored me 60 points and likely took about fifteen minutes to write.

If you’re going for points (and that’s the entire raison d’être for gamification!), are you going to waste time like that for 60 points when you could fit in a dozen 460-point answers? Of course not! You’re going to go wherever the points are, And the points are the low-hanging fruit of trivial questions from popular languages. The way StackOverflow is structured rewards people who put as little work as possible across as many simple questions as possible within only the most popular segments. Spending thought (and thus time) on answers interferes with points- and badge-mongering. Answering questions outside of the top ten languages similarly interferes.

This “cool kids table” problem has a very real effect. Look again at my profile. 218 answers. 10 questions. Why is this? I could arrogantly claim that this is because I know more than everybody else, but the real reason is that getting answers on any question that requires thought is a non-starter. As my approach to scoring points (Google + hasty rewrite) shows, I’m quite adept at answering trivial questions myself. For a prime example of the problem, consider this question. It got five answers (one of which you probably can’t see), only one of which even really answered part of the question. Why? Because answering the whole question would have taken a lot more work than most people in StackOverflow would be willing to put in. There’s just no percentage in spending time on difficult questions when you can hoover up a cool thousand points in a fraction of the time.

Broken scoring

Even if the “cool kids table” wasn’t an issue at StackOverflow, the system is still largely broken. Remember how I have over 14,000 points as of this writing? Two years and a bit ago, when I decided to stop participating in StackOverflow, this was not the case. I was “only” at OVER NINE THOUSAND! and several hundred points shy of getting moderator status. In well over two years I have contributed nothing to StackOverflow: no questions, no answers, nothing. (Well, that’s not true. When my score went over 10,000 I tried out the moderator powers for a couple of edits, just to test them out.) Over one third of my reputation was “earned” from me doing absolutely nothing for over two years. Indeed I went from the top 4% of contributors at my time of departure to the top 3%, despite, you know, me not doing anything.

Any scoring system that allows this to happen is simply broken in my opinion.

Poor community

And now my reputation score will go down!

Petty children

As of this writing my score was, as mentioned above, over 14,000. (14,076 to be precise.) I predict that this is going to go down as more people find out about this blog entry and start voting down my questions and answers in petty revenge. How do I know? I’ve been on the receiving end of sudden bouts of negative votes before. Consider this question about ‘goto’ constructs, for example. As of this writing it has 72 up votes and 13 down votes. It is simultaneously one of my most popular answers as well as one of my most hated ones.

The people who hated it weren’t content, however, with merely voting it down. No, after I posted that answer I had a mysterious downward turn in my reputation as downvotes appeared all over my answers. People got so upset at my mocking one of the Holy of Holies of computing that it wasn’t enough to just downvote the answer, they had to punish me. (Their selected means of punishment was as highly amusing as it was highly ineffective.) This is not the way a community of mature users acts.

Creeping authoritarianism

That kind of behaviour is, of course, inevitable in any kind of Innarwebs™® interaction. Pseudo-anonymity makes doorknobs of otherwise-normal people. There is something else, however, in the whole Stack Exchange hierarchy that bugs me: the creeping authoritarianism.

The “flavour” of StackOverflow today is entirely different than the flavour it had when I started. When I started the community as a whole still had a bit of a sense of humour. Sure sometimes questions and/or answers would be a bit off-topic or a bit irreverent, but it gave more of a community feel that way, even if it was on occasion less-than-“professional”.

This changed slowly but surely in the way that all “community moderated” things change. Here is the recipe that all such “community-driven” approaches almost, but not quite, invariably follow:

  1. A wide-open community based on “merit” is built.
  2. The community gets a kernel of users who build up “merit” by virtue of, basically, being obsessive twerps.
  3. As this kernel of “serious” users builds up its influence, they start to modify what the standards of the community are to match their own desires.
  4. These standards get enforced on other members of the community who lack sufficient “merit” (read: who have a life outside the site) to fight back.
  5. The tenor of the community changes to match the notions of the obsessive, but “meritous” minority.
  6. Lather. Rinse. Repeat.

This happened at Wikipedia and it’s happened at StackOverflow. StackOverflow was once fun. It is no longer. StackOverflow once had a tolerance for things a little outside of the norm. It does no longer.

Take a look at the site now. Some of the most popular questions and/or answers are now locked down and only kept for “historical reasons”. Consider this answer. It’s likely the best-known and most-loved answer on the entire damned site! It’s funny and it’s informative. But it’s something that makes the current powers-that-be at the site crazy and thus it is locked and we have this ominous note appended: “This post has been locked while disputes about its content are being resolved.”

What. The. Fuck!?

TL;DR

There are a number of reasons why I stopped contributing to StackOverflow. I am disquieted by its poor pedagogical value, I think its scoring system is fundamentally broken and rewards the wrong things, and I think its community lacks maturity even while it becomes more and more pointlessly authoritarian. So what would I recommend as an alternative?

How about learning? You know, that thing that puts information in your head that you can apply later at need. Use Google. Use Wikipedia (if you must). Use RosettaCode for code examples. (Contribute there too!) Engage with other users of the tools you use in the form of user groups, mailing lists, web forums, etc. Learn foundational principles instead of answers to immediate questions.

On Daily Standup Meetings

Agile development methodologies such as Scrum, Kanban and XP have taken the software development industry by storm. By now, most companies have drunk the kool-aid – first the decision-makers, then the developers – and follow it with a manner of religious zeal.

It seems like this exciting new “agile” way of working has made traditional hierarchical management and reporting lines obsolete. Agile has by now entrenched itself deeply within the fabric of software companies, and it seems like we’ve forgotten the old ways.

We are now faced with the conflict of fixed-backlog sprints and shifting requirements, burndown charts that never converge, armies of full-time scrum masters babysitting developers to death, and the progression (or regression) of the Daily Scrum into… the Daily Standup Meeting.

While I understand this article may be controversial and may touch a nerve in a large number of people, I find it much more important to work efficiently and to challenge harmful norms than to mindlessly conform to them. My hope is that at least some people might read this article with an open mind, break out of their stupor, and start questioning these established practices. In order to improve, it is necessary to first acknowledge problems, and then take active steps towards addressing them.

What is the Daily Standup Meeting?

Image credit: taken from here

The Daily Standup Meeting is a “ritual” (quoting Jason Yip’s comprehensive article on the topic) in which the development team (and possibly other stakeholders) gathers to report on the progress of each individual member. They must in turn answer three questions:

  • What did you do yesterday?
  • What are you going to do today?
  • Are there any obstacles hindering your progress?

Let’s take a look at the meaning of each word, aside from “Meeting” which is obvious enough. As a “Daily” meeting, this happens routinely every day at a specific time. All those involved must stop what they are doing and participate, whether it is productive or not.

What I have just described is nothing new: I have participated in Daily Scrum meetings of this nature for years, and I’m pretty sure this goes back much father than my personal experience can account for.

A more recent twist, though, is the Daily “Standup” Meeting. The idea is to force people to stand up in order to keep meetings short, by having the physical discomfort as a disincentive to letting the meetings drag on unnecessarily.

Standing Up

This factor, on its own, speaks volumes. Rather than solving the problem of pointless meetings that are a complete waste of time, we choose to make people feel uncomfortable in the hope that the meetings will be kept short.

The first problem with this is how ridiculous and humiliating it is. Not even in church are you obliged to stand up, and perhaps at school there may have been instances where some posture was enforced in punishment (a practice that is probably considered barbaric by today’s standards).

But no. We are fully grown adults, who have been working for several years in the industry, and there is a perfectly comfortable sofa behind us, but we cannot use it, under pain of menacing glares from everybody in the room.

This also has no regard for any physical issues that the participants may be undergoing that may make this practice more uncomfortable than for the average human being. Such people should not have to disclose their personal problems to the team in order to waive the enforcement of a certain posture on the job, something that nobody has the right to enforce in the first place.

The second problem is that, well, it doesn’t work. Standing up or not, the tendency to ramble on is a lot more powerful than the discomfort (which most people can just as well get used to). Which brings us to the next section.

Excessively Long Standups

Do you remember what we’re supposed to do in standups? Basically, answer the following three questions, and then get on with our work:

  • What did you do yesterday?
  • What are you going to do today?
  • Are there any obstacles hindering your progress?

It is much harder than you would think, to stick to the plan. Despite having been brainwashed into enthusiastic and excited participation in this drudgery, people’s overexcitement tends to take over. There are many different scenarios which result in deviation from this template, and in turn cause Daily Standup Meetings to drag on. These are just a few of the most common I’ve encountered:

  • People want to justify their job and show that they are doing something, so they read out a whole list of every little detail they worked on since the last Daily Standup Meeting.
  • Stakeholders ask a lot of questions to the development team. They have a right to be answered, but the Daily Standup Meeting is not the right place.
  • Developers going off on a tangent and having a technical (or other) discussion between two or three people, but keeping everyone in the room.
  • Lots of people in the room, so it takes a while to go through what everyone has to report. This is mainly a problem when developer teams are large, but it can also be a result of participation of a lot of stakeholders who join the Standup. With 20 people in the room, it’s hard to keep meetings efficient.

Broadcasting Information

The format of the Daily Standup Meeting is such that the development team and any other stakeholders are present (whether physically or otherwise). It allows everybody in the room to be kept up to date with progress (in theory) and to be able to help remove any obstacles (again, in theory).

Personally, I believe that this is one of those cases where too much transparency is a bad thing. The main reason for this is that there is no reason why each person should need to broadcast their progress to so many people in the room. Typically, you work with one or two direct colleagues, and report to your manager. There is no need to keep a whole crowd in the loop.

Rather than focusing on what each of us needs to do, we instead feel the urge to get involved in what the whole team is up to. A consequence of this is that it becomes necessary to involve everybody in the team in order to take the slightest technical decision, which is really the opposite of the empowerment that should be enabled by giving developers such autonomy.

In practice, keeping everyone on the team updated has no real benefit. They may know you’re working on something, but they would not understand the business or technical decisions, and as a result, they would not be able to take up the task you are working on, in your absence.

Another problem is the direct access to the development team from higher-up stakeholders. Part of a manager’s job is to manage expectations, both to his superiors and to his subordinates. And yet, higher-level stakeholders and members now suddenly have full access to the fine details of what the development team is working on.

A real problem here is when members of the development team are cornered into publicly answering uncomfortable questions, possibly by external stakeholders or maybe even by members of their own team. Such things should really go through a manager or scrum master rather than being laid out in a public shit show. It is not uncommon for such meetings to degrade into blame games.

Even in the absence of such situations, it is often uncomfortable for developers to speak (let alone report their progress) in front of a small crowd, even if it consists of their own direct colleagues. While the stereotype of anti-social developers may be questionable, in my experience I have observed that most developers feel intimidated when speaking in front of people, just as they feel uncomfortable when someone (even a direct colleague) sits next to them while they work. It is thus quite easy for developers to go on the defensive if challenged about their work during the Daily Standup Meeting.

And this, of course, is something that should be completely avoided. As any manager worth his salt would know, you should never, ever reprimand someone in front of other people. This is also a fundamental flaw of retrospective meetings, although that is beyond the scope of this article.

Finally, the routine of the Daily Standup Meeting makes it convenient for people to provide updates and feedback during the meeting itself, rather than spontaneously as needed. Communication should be spontaneous and immediate; that is what keeps things efficient.

A trend I have seen is for companies with poor communication to turn to agile ceremonies to try to resolve the problem. Little do they realise that they are making it worse by adding more ritual baggage and introducing more middlemen. There is no substitute for real and effective communication.

What Alternative is There?

Honestly, what was so terrible about the good old system of hierarchical reporting lines? Think about it:

  • You talk directly to your manager when you’ve finished your work and need to work on something new.
  • Your manager can ping you periodically for progress, but otherwise you can focus on your work.
  • If there are impediments, the manager can bring the right people on board.
  • If you need to talk to some direct colleagues (e.g. to integrate with their API), just do so. No need to have the whole team in a meeting for that.
  • Requirements and other outside interferences go through the manager first.
  • Feedback and criticism take place one-on-one with the manager.
  • Each manager reports to their manager. No need for crowds.

That just about solves all the problems I’ve described earlier: notice the contrast of involving only the people who are necessary, as opposed to everyone. The important factors to make this work are having a manager who is organised and has excellent people skills, and that everybody on the team is able to communicate efficiently and effectively.

Agile methodologies will remain a reality for many years to come. But are they really an improvement?

Update 11th September 2017: Here are some reactions to this article on social media (might require login):

RabbitMQ: Who Creates the Queues and Exchanges?

Messaging is a fundamental part of any distributed architecture. It allows a publisher to send a message to any number of consumers, without having to know anything about them. This is great for truly asynchronous and decoupled communications.

The above diagram shows a very basic but typical setup you would see when using RabbitMQ. A publisher publishes a message onto an exchange. The exchange handles the logic of routing the message to the queues that are bound to it. For instance, if it is a fanout exchange, then a copy of the same message would be duplicated and placed on each queue. A consumer can then read messages from a queue and process them.

An important assumption for this setup to work is that when publishers and consumers are run, all this RabbitMQ infrastructure (i.e. the queues, exchanges and bindings) must already exist. A publisher would not be able to publish to an exchange that does not exist, and neither could a consumer take messages from an inexistent queue.

Thus, it is not unreasonable to have publishers and/or consumers create the queues, exchanges and bindings they need before beginning to send and receive messages. Let’s take a look at how this may be done, and the implications of each approach.

1. Split Responsibility

To have publishers and consumers fully decoupled from each other, ideally the publisher should know only about the exchange (not the queues), and the consumers should know only about the queues (not the exchange). The bindings are the glue between the exchange and the queues.

One possible approach could be to have the publisher handle the creation of the exchange, and consumers create the queues they need and bind them to the exchange. This has the advantage of decoupling: as new queues are needed, the consumers that need them will simply create and bind them as needed, without the publisher needing to know anything about them. It is not fully decoupled though, as the consumers must know the exchange in order to bind to it.

On the other hand, there is a very real danger of losing messages. If the publisher is deployed before any consumers are running, then the exchange will have no bindings, and any messages published to it will be lost. Whether this is acceptable is something application-dependent.

2. Publisher Creates Everything

The publisher could be configured to create all the necessary infrastructure (exchanges, queues and bindings) as soon as it runs. This has the advantage that no messages will be lost (because queues will be bound to the exchange without needing any consumers to run first).

However, this means that the publisher must know about all the queues that will be bound to the exchange, which is not a very decoupled approach. Every time a new queue is added, the publisher must be reconfigured and redeployed to create it and bind it.

3. Consumer Creates Everything

The opposite approach is to have consumers create exchanges, queues and bindings that they need, as soon as they run. Like the previous approach, this introduces coupling, because consumers must know the exchange that their queues are binding to. Any change in the exchange (renaming, for instance) means that all consumers must be reconfigured and redeployed. This complexity may be prohibitive when a large number of queues and consumers are present.

4. Neither Creates Anything

A completely different option is for neither the publisher nor the consumer to create any of the required infrastructure. Instead, it is created beforehand using either the user interface of the Management Plugin or the Management CLI. This has several advantages:

  • Publishers and consumers can be truly decoupled. Publishers know only about exchanges, and consumers know only about queues.
  • This can easily be scripted and automated as part of a deployment pipeline.
  • Any changes (e.g. new queues) can be added without needing to touch any of the existing, already-deployed publishers and consumers.

Summary

Asynchronous messaging is a great way to decouple services in a distributed architecture, but to keep them decoupled, a valid strategy for maintaining the underlying messaging constructs (in the case of RabbitMQ, these would be the queues, exchanges and bindings) is necessary.

While publisher and consumer services may themselves take care of creating what they need, there could be a heavy price to pay in terms of initial message loss, coupling, and operational maintenance (in terms of configuration and deployment).

The best approach is probably to handle the messaging system configuration where it belongs: scripting it outside of the application. This ensures that services remain decoupled, and that the queueing system can change dynamically as needed without having to impact a lot of existing services.

.NET Core 2.0: Referencing .NET Framework Libraries: A Topshelf Experiment

Referencing .NET Framework Libraries in .NET Core 1.1

There are many old libraries targeting the .NET Framework which, for various reasons, do not yet target .NET Core or .NET Standard. Topshelf, a fantastic library that helps you to easily create Windows services, is one of these. At the time of writing this article, the last release of Topshelf was 4.0.3 back in October 2016. There was no way that Topshelf could target .NET Standard because the .NET Standard spec did not support the APIs that are required for it to function.

This was a problem because there was no way you could create a Windows service using Topshelf for a .NET Core 1.1 application. Simply trying:

Install-Package Topshelf

…is bound to fail miserably:

If you want to make a Windows service out of an application targeting .NET Core 1.1, then you have to use an alternative such as NSSM.

What Changed in .NET Core 2.0

With the release of .NET Core 2.0 and .NET Standard 2.0, applications or libraries targeting either of these are able to reference old libraries targeting the full .NET Framework. Presumably this is because the .NET Core/Standard 2.0 implementations have enough API coverage to overlap with what the full framework was able to offer.

Quoting the .NET Core/Standard 2.0 announcement linked above:

“You can now reference .NET Framework libraries from .NET Standard libraries using Visual Studio 2017 15.3. This feature helps you migrate .NET Framework code to .NET Standard or .NET Core over time (start with binaries and then move to source). It is also useful in the case that the source code is no longer accessible or is lost for a .NET Framework library, enabling it to be still be used in new scenarios.

“We expect that this feature will be used most commonly from .NET Standard libraries. It also works for .NET Core apps and libraries. They can depend on .NET Framework libraries, too.

“The supported scenario is referencing a .NET Framework library that happens to only use types within the .NET Standard API set. Also, it is only supported for libraries that target .NET Framework 4.6.1 or earlier (even .NET Framework 1.0 is fine). If the .NET Framework library you reference relies on WPF, the library will not work (or at least not in all cases). You can use libraries that depend on additional APIs,but not for the codepaths you use. In that case, you will need to invest singificantly in testing.”

Example with Topshelf

In order to actually test this out, you’ll need to have Visual Studio 15.3 or later. You will also need to separately install the .NET Core 2.0 SDK.

In an earlier section, we tried installing Topshelf in a .NET Core 1.1 application, and failed. Let’s try doing the same thing with a .NET Core 2.0 application:

Install-Package Topshelf

The package installation works pretty well:

However, the warning that shows under the dependency is not very promising:

There’s only one way to find out whether this will actually work in practice.

Let’s steal the code from the Topshelf quickstart documentation:

public class TownCrier
{
    readonly Timer _timer;
    public TownCrier()
    {
        _timer = new Timer(1000) {AutoReset = true};
        _timer.Elapsed += (sender, eventArgs) => Console.WriteLine("It is {0} and all is well", DateTime.Now);
    }
    public void Start() { _timer.Start(); }
    public void Stop() { _timer.Stop(); }
}

public class Program
{
    public static void Main()
    {
        HostFactory.Run(x =>                                 //1
        {
            x.Service<TownCrier>(s =>                        //2
            {
               s.ConstructUsing(name=> new TownCrier());     //3
               s.WhenStarted(tc => tc.Start());              //4
               s.WhenStopped(tc => tc.Stop());               //5
            });
            x.RunAsLocalSystem();                            //6

            x.SetDescription("Sample Topshelf Host");        //7
            x.SetDisplayName("Stuff");                       //8
            x.SetServiceName("Stuff");                       //9
        });                                                  //10
    }
}

Nope, looks like Topshelf won’t work even now.

I guess the APIs supported by .NET Core 2.0 still do not have enough functionality for Topshelf work as-is. Other .NET Framework libraries may work though, depending on the dependencies they require. In the “.NET Core 2.0 Released!” video, one of the demos shows SharpZipLib 0.86 (last released in 2011) being installed in an ASP .NET Core 2.0 application. It is shown to build, but we don’t get to see whether it works at runtime.

It is still early, and I suppose we have yet to learn more about the full extent of support for .NET Framework libraries from .NET Core 2.0 applications and .NET Standard 2.0 libraries. The problem is that when evaluating a third-party library such as Topshelf, it’s difficult to determine whether its own dependencies fall within the .NET Standard API set. This looks to me like a matter of pure trial and error.

The Abysmal State of the Web in June 2017

This will be the last article in the Sorry State of the Web series (at least for the time being). The idea was to learn from the mistakes of other so-called ‘professional’ websites, ranging from silly oversights to illegal practices. Hopefully, the silliness encountered has also made some people smile.

However, with 11 articles over 6 months, I believe I’ve made my point enough times over. Despite all the technological advancements, the web is in a state that I can call sick at best, and that is mainly the result of clueless developers. I have some slight hope that things may get better, but given that most of the issues I pointed out have not been addressed to date, that hope is realistically very slim.

From my part, I want to focus less on beating a dead horse and more on learning technology and writing high quality articles. I don’t exclude revisiting this series in future if I feel it’s worth it though. Once again, I extend my heartfelt thanks to all those who have contributed entries for this article and the ones before it.

Banif: Random Virtual Keyboard

If you think that the mainstream banks in Malta have terrible websites (and recently I covered how Mediterranean Bank’s newly launched online investment platform took them several steps back), then you should really take a look at Banif Bank Malta.

To log into their online banking section, you have to enter a username and a password. This would be understandable, if not for the fact that the password field is disabled so you can’t actually type into it. Instead, you have to click on keys on a virtual keyboard. To make matters worse, this is not your usual QWERTY keyboard: the key placements are randomised.

Let’s consider a few reasons why this is a terrible idea:

  • It makes it a lot harder for users to type in their password (in terms of user experience).
  • It slows down password entry, both because one has to use the mouse vs the keyboard and because the random placement requires the eye to look for keys as opposed to using muscle memory. This makes it easier for people watching you enter the password to identify what you are actually entering, and it also makes you more likely to pick simpler passwords.
  • People looking over your shoulder can easily see what key the cursor is on, which defeats the purpose of password field obfuscation.
  • The restrictions on the password field are client-side and trivial to disable. This does no favours for server-side security, which should really be the main focus.
  • You cannot use a password manager.

Since I’m not a security expert, I presented this case to the community at Information Security Stack Exchange. From there, I got to two related existing questions:

It seems that the main reason why this horrendous technique is used is to counteract keyloggers, which at a basic level can’t track keypresses (since they are not happening) or mouse clicks (since the placement of keys on the screen changes).

However, as one of the best answers points out, this is merely an arms race between the bank and attackers. It’s a vicious circle in which attackers and banks take it in turns to step up their game. The end result is that customers are the ones paying the price, by having to deal with ridiculous security measures like this.

Dealing with keyloggers is hardly an excuse for this kind of rubbish. There are much more robust and orthodox ways of dealing with this sort of thing, such as one-time passwords or two-factor authentication.

Insecure Logins

One of the most common issues we’ve seen throughout this series is that of websites with login forms where the credentials are not transmitted over HTTPS. Thus it is not hard for them to be intercepted and read in clear text. Keeping up with tradition, we have a list of such examples this month.

We can start with American Scientist, which I see has since undergone a complete redesign and does currently use HTTPS for the whole website (including login). This is how it was just a couple of weeks ago:

Then we have the Malta Chamber of Advocates, which aside from very ridiculously presenting a homepage with no content whatsoever, is just another case of insecure login:

But wait! The next one, ironically, is from none other than Bank Info Security:

Then we have Great Malta (whatever that is supposed to mean):

Local newspaper The Malta Independent is no less guilty:

…and neither is Infobel:

In another case if irony, we can look at J. Grima & Co. Ltd. They are “Security & Fire Specialists”, but web security is clearly not one of their areas of expertise.

Excitable Web

I was very excited (!) to come across Excitable Web, because it is a prime example of the clueless developers I was mentioning earlier. It is of little importance that each time you load a page, the page seems to render without CSS for half a second before rendering properly; because we’ll focus on more interesting stuff here. If you click on the “Who We Are” link, we get this:

You can see there are a couple of MySQL errors displaying directly in the page due to deprecated code. Such an experienced professional should know that server-side errors should never be displayed directly to the visitor, as this may reveal vulnerabilities among other things.

These errors seem to have been fixed since then, so we’ll move onto the next thing: the writing. It’s really generous of the webmaster to give us:

“A Breif [sic] Background On With [sic] Whome [sic] You Are Dealing With”

You can find other such gems within the content itself. Thank you, Adrian. Now we really know who we are dealing with.

For extra points, spot one of my own blunders within that screenshot!

Flybussen Translations

Here’s a tiny oversight from Norwegian operator Flybussen. While their site has an English version, their calendar unfortunately doesn’t:

JobsPlus Going Below Minimum Wage

JobsPlus has by now become a regular in this series. Those who believe that we should have equal pay for equal work (which is a legal requirement, by the way) will be delighted to see this vacancy where the position advertises a salary range of between EUR4,500 and EUR70,000. What’s even funnier, though, is that EUR4,500 is actually below the minimum wage (another legal requirement) for a 40-hour full-time work week.

Legal requirements aside, this is just a case of missing validation by our award-winning friends at JobsPlus who should have a central role in avoiding precarious work and exploitation.

Kelly on Yellow Pages

If you take a look at the Yellow Pages entry for Kelly Industries, you’ll come to the conclusion that they have enough business to not give a rat’s ass about what potential customers think about their brand.

Creativity Centre

I’ve received reports about issues with the Malta’s National Centre for Creativity‘s payment processing engine, but I haven’t been able to verify them without actually attempting to make a purchase. However, I did notice this problem with the checkout button actually not being properly visible if you’re using a laptop (and thus a limited screen resolution):

For a National Centre for Creativity, I must also point out that they didn’t quite put a lot of creativity into the website’s design.

Mixed Content

Another common problem we’ve seen throughout the series is that of using HTTPS, but serving some content over HTTP. This is called Mixed Content, and it invalidates the trust guaranteed by a fully HTTPS website.

This month, we have Malta Gift Service (also guilty of using Comic Sans for their main header):

…and our dear friends at Scan:

Apostrophes of Doom

Given that my surname contains an apostrophe, this often makes it a pain to deal with validation that unreasonably decides that an apostrophe is an invalid character. I’ve written about this especially in the original “The Sorry State of the Web in 2016“. There is no real reason to not accept apostrophes if you’re using proper practices (e.g. using prepared statements) to prevent SQL injection.

Unfortunately, Microsoft has decided that my surname cannot have an apostrophe:

I suppose I will need to remove the apostrophe from my identity card if I want to ever get a job at Microsoft.

Piscopo Gardens

The Piscopo Gardens website has been down for I don’t know how long due to some internal server error.

Aron isn’t doing a very good job at keeping the site up and running.

Robert Half

Swiss recruiter Robert Half believes that “It’s time we all work happy.™” (so much that a trademark was apparently filed).

That obviously doesn’t apply to their own website, which clearly doesn’t work if you enter “.net” in the search field:

Now I understand the name. Their website only Half works.

Ryanair Mischief

We noticed a couple of things on Ryanair’s website that are more sneaky practices than examples of bad web design per se.

First, there’s the newsletter checkbox that is opt-out rathern than opt-in (i.e. it automatically signs you up if you ignore it and leave it unchecked):

Then there’s this appeal to fear the middle seat:

Oh dear, not the middle seat!

Image credit: Taken from Wikipedia

Better to go for a team-building treasure hunt in 35-degrees-Celsius weather with a laptop on my back than be stuck in a middle seat! Actually, no. Give us a break, Ryanair.

Conclusion

I am happy to have managed to raise awareness about bad practices in web design with this series. I know this because I have heard several reports of companies that I have pissed off. I am a lot less happy that these companies have not really done much about it despite all this. That is their problem now. No doubt others have learned from the countless issues pointed out.

Let’s continue to make companies with a web presence understand that such a public face requires a high level of professionalism, and that they will lose business if they don’t step up their game.

Once again I would like to thank all the contributors to this series, and also the readers who have loyally followed it.

Your First Microsoft Orleans Cluster

If you’re planning to use Microsoft Orleans in production, you need to look beyond the lonely silos that we’ve built in the articles thus far. Orleans is designed to work in a cluster, such that a large number of grains can be distributed among multiple silos. In case of silo failure, its grains are reactivated at other silos that are still alive.

In this article, we’ll create a simple silo and run multiple interconnected instances of it in order to set up a cluster.

For the scope of this article, we’ll use the default cluster membership provider: MembershipTableGrain. This is not intended to be used in production, but will allow us to focus on getting a simple cluster up and running. Setting up different cluster membership providers is non-trivial and requires separate articles for each.

Note: this article is based on Orleans 1.4.2 using .NET Framework 4.6.2.

Note: the source code for this article is in the OrleansFirstCluster folder within the Gigi Labs BitBucket repository.

Setting Up An Example

To set up a cluster, the Dev/Test Host project template we’ve been using so far is no longer suitable. Instead, we have to set up the full project structure. This is covered by the latter part of “Getting Started with Microsoft Orleans” and there is no point in repeating it here.

Don’t write any code yet though. I recently learned that all that AppDomain stuff is not necessary unless you’re planning to run Silo and Client in the same application, so we’ll go for a cleaner approach.

We’ll also install the Orleans Dashboard (see: “A Dashboard for Microsoft Orleans“) in the Silo project. This will give us an idea how grains are spread across the cluster later.

Install-Package OrleansDashboard

Hence, when setting up the Silo configuration, remember to include the configuration for the Dashboard:

<?xml version="1.0" encoding="utf-8"?>
<OrleansConfiguration xmlns="urn:orleans">
  <Globals>
    <SeedNode Address="localhost" Port="11111" />
    <BootstrapProviders>
      <Provider Type="OrleansDashboard.Dashboard" Name="Dashboard" />
    </BootstrapProviders>
  </Globals>
  <Defaults>
    <Networking Address="localhost" Port="11111" />
    <ProxyingGateway Address="localhost" Port="30000" />
  </Defaults>
</OrleansConfiguration>

We can now start adding some code. First, we need a grain in our Grains project. What the grain actually does doesn’t matter. We just want to create a large number of grains to see them spread out over the cluster.

    public class UselessGrain : Grain, IUselessGrain
    {
        public Task DoNothingAsync()
        {
            return Task.CompletedTask;
        }
    }

Note: if you’re using a .NET Framework version prior to 4.6, then you’ll need to use TaskDone.Done instead of Task.CompletedTask.

The corresponding interface goes in the Interfaces project:

    public interface IUselessGrain : IGrainWithIntegerKey
    {
        Task DoNothingAsync();
    }

Doing away with all the AppDomain junk, the following code should be enough for a simple Silo:

        static void Main(string[] args)
        {
            Console.Title = "Silo";

            try
            {
                using (var siloHost = new SiloHost("Silo"))
                {
                    siloHost.LoadOrleansConfig();
                    siloHost.InitializeOrleansSilo();
                    var startedOk = siloHost.StartOrleansSilo(catchExceptions: false);
                    Console.WriteLine("Silo started successfully!");

                    Console.WriteLine("Press ENTER to exit...");
                    Console.ReadLine();
                    siloHost.ShutdownOrleansSilo();
                }
            }
            catch (Exception ex)
            {
                Console.WriteLine(ex);
            }
        }

Apart from the AppDomain logic, another thing we’re doing differently from usual here is that we’re calling StartOrleansSilo() with catchExceptions set to false. In case the silo fails to initialise, this gives us the ability to inspect the details of the failure within the exception, rather than have Orleans silently swallow it and simply return false.

On the client side, we can use an adaptation of the client code from “Getting Started with Microsoft Orleans“:

        static void Main(string[] args)
        {
            Console.Title = "Client";

            var random = new Random();
            var config = ClientConfiguration.LoadFromFile("ClientConfiguration.xml");

            while (true)
            {
                try
                {
                    GrainClient.Initialize(config);
                    Console.WriteLine("Connected to silo!");

                    while (true)
                    {
                        var grainId = random.Next();
                        var grain = GrainClient.GrainFactory.GetGrain<IUselessGrain>(grainId);
                        grain.DoNothingAsync();
                    }
                }
                catch (SiloUnavailableException)
                {
                    Console.WriteLine("Silo not available! Retrying in 3 seconds.");
                    Thread.Sleep(3000);
                }
            }
        }

In the inner infinite-while-loop we’re taking random grain IDs and bombarding them with messages. The idea is to create a lot of grain instances that we can visualise. Since this is very heavy, you’ll see Orleans giving warnings, and high latencies from the Dashboard at times.

Running a 3-Node Cluster

We will now run 3 instances of the same silo. Each instance must have different ports configured. This is the configuration for the first silo, which we already set up earlier:

<?xml version="1.0" encoding="utf-8"?>
<OrleansConfiguration xmlns="urn:orleans">
  <Globals>
    <SeedNode Address="localhost" Port="11111" />
    <BootstrapProviders>
      <Provider Type="OrleansDashboard.Dashboard" Name="Dashboard" />
    </BootstrapProviders>
  </Globals>
  <Defaults>
    <Networking Address="localhost" Port="11111" />
    <ProxyingGateway Address="localhost" Port="30000" />
  </Defaults>
</OrleansConfiguration>

A silo needs 2 ports: one to communicate with other silos (the Networking endpoint) and one for clients to connect to it (the ProxyingGateway endpoint). The client can connect to any node on the cluster.

In production scenarios, Orleans silos are all equal, and there is no concept of a primary and secondary silo. However, when you use the default MembershipTableGrain cluster membership, then all information regarding the silos on the cluster is stored within a grain in one of the silos. As a result, the silo containing the MembershipTableGrain is denoted as the Primary Silo. It must be started before the others, and the entire cluster is messed up if it goes down. Naturally, this is not good, and you should look into other cluster membership providers.

In such a setup, the SeedNode configuration specified in all silos must be the endpoint of the Primary silo. Let’s see what the configuration for our second silo instance looks like:

<?xml version="1.0" encoding="utf-8"?>
<OrleansConfiguration xmlns="urn:orleans">
  <Globals>
    <SeedNode Address="localhost" Port="11111" />
    <BootstrapProviders>
      <Provider Type="OrleansDashboard.Dashboard" Name="Dashboard" Port="8081" />
    </BootstrapProviders>
  </Globals>
  <Defaults>
    <Networking Address="localhost" Port="11112" />
    <ProxyingGateway Address="localhost" Port="30001" />
  </Defaults>
</OrleansConfiguration>

Aside from changing the Networking and ProxyingGateway ports, we are also using a different port for the Dashboard (default is 8080). Each silo has its own Dashboard (although they all show the same information), and they cannot all run from the same port.

Similarly, the configuration for our third silo instance is just a matter of changing ports:

<?xml version="1.0" encoding="utf-8"?>
<OrleansConfiguration xmlns="urn:orleans">
  <Globals>
    <SeedNode Address="localhost" Port="11111" />
    <BootstrapProviders>
      <Provider Type="OrleansDashboard.Dashboard" Name="Dashboard" Port="8082" />
    </BootstrapProviders>
  </Globals>
  <Defaults>
    <Networking Address="localhost" Port="11113" />
    <ProxyingGateway Address="localhost" Port="30002" />
  </Defaults>
</OrleansConfiguration>

We can then start the 3 silo instances and the client:

On my system, the load is just too much and Orleans just dies after around 64k activations. So let’s add a little delay in the random message loop to give Orleans some room to breathe:

                    while (true)
                    {
                        var grainId = random.Next();
                        var grain = GrainClient.GrainFactory.GetGrain<IUselessGrain>(grainId);
                        Thread.Sleep(50);
                        grain.DoNothingAsync();
                    }

After running it again, what I see is that grains are allocated mainly to the primary silo initially, but they are distributed more evenly across the other silos after around 1,000 activations:

I am not sure why they are not evenly distributed from the start. My guess is that either it is more efficient to have them all in one place if the number of activations is small, or the silos need time to coordinate between themselves before this happens (which would explain why, without a delay, all activations are allocated on the primary node).

Single Point of Failure

Close the primary silo.

Since the Primary silo contains the MembershipTableGrain, all information about the cluster dies with it. The remaining silos and clients will not recover automatically even if the Primary silo is brought up again. They in turn will have to be restarted. This is because, as we saw earlier, Secondary silos must start after the primary one. When the Primary silo is brought back, it effectively starts a fresh cluster and does not know about any other silos until they join.

Conclusion

We have seen how to get a very basic Orleans cluster working with multiple silos sharing the burden of holding the grains. However, this is hardly an ideal setup because (a) cluster membership information is held in memory and represents a single point of failure, and (b) the fact that I ran all silos on the same machine made them subject to the same physical resource constraints as if I were running a standalone silo.

For better results, run different silos on different machines, and use a decent cluster membership provider. Orleans supports the following:

  • MembershipTableGrain (not realiable, use for testing only)
  • SQL Server
  • Azure Table Storage
  • Apache ZooKeeper
  • Consul
  • DynamoDB

Update 20th June 2017: I am told that Azure Service Fabric should also be supported. As for database implementations of cluster membership, these are not limited to just SQL Server. You may use any supported ADO .NET provider, which at the moment includes SQL Server, MySQL/MariaDB, or PostgreSQL. To clarify: while the PostgreSQL storage provider for grain persistence is not yet available, its use as a cluster membership provider is supported.

Saving Screenshots in SDL2

Saving screenshots is a simple and common feature in many games. It allows us to capture the image of the game being rendered on the screen at any given time.

While this might sound easy, let us remember that the image rendered to the screen might have a lot of different overlaid surfaces, meaning that it would be a pain to recompose that image on the side of the CPU in the same way that we’re composing the image to the texture that eventually ends up in video memory.

Fortunately, this Stack Overflow answer provides a simple solution. SDL2 provides the SDL_RenderReadPixels() function, which can be used to read pixel data back from video memory. This is generally discouraged, because it is very costly to do such a thing, but taking screenshots is a one-off operation where it makes perfect sense.

The following code shows how screenshot capture was implemented in Ultima 1 Revenge. It is only slightly different from the code in the aforementioned Stack Overflow answer:

void SosariaInputController::SaveScreenshot()
{
	const Uint32 format = SDL_PIXELFORMAT_ARGB8888;
	const int width = 640;
	const int height = 400;
	auto renderer = sdl2Core->GetRenderer();

	SDL_Surface *surface = SDL_CreateRGBSurfaceWithFormat(0, width, height, 32, format);
	SDL_RenderReadPixels(renderer, NULL, format, surface->pixels, surface->pitch);
	SDL_SaveBMP(surface, "screenshot.bmp");
	SDL_FreeSurface(surface);
}

While I am not showing where the sdl2Core object is coming from, just assume that it stores an instance of SDL_Renderer. With that available, all that is needed is to create a surface using one of the functions that SDL2 provides, call SDL_RenderReadPixels() to transfer the pixels from video memory to the surface, and then actually do something with the surface (in this case, we are saving it to a bitmap file). If you wanted to read back only a portion of the screen, you would pass in an SDL_Rect instead of NULL as the second parameter to SDL_RenderReadPixels().

This simple code is used to save screenshots in Ultima 1 Revenge, such as the one shown earlier.

"You don't learn to walk by following rules. You learn by doing, and by falling over." — Richard Branson