On Daily Standup Meetings

Agile development methodologies such as Scrum, Kanban and XP have taken the software development industry by storm. By now, most companies have drunk the kool-aid – first the decision-makers, then the developers – and follow it with a manner of religious zeal.

It seems like this exciting new “agile” way of working has made traditional hierarchical management and reporting lines obsolete. Agile has by now entrenched itself deeply within the fabric of software companies, and it seems like we’ve forgotten the old ways.

We are now faced with the conflict of fixed-backlog sprints and shifting requirements, burndown charts that never converge, armies of full-time scrum masters babysitting developers to death, and the progression (or regression) of the Daily Scrum into… the Daily Standup Meeting.

While I understand this article may be controversial and may touch a nerve in a large number of people, I find it much more important to work efficiently and to challenge harmful norms than to mindlessly conform to them. My hope is that at least some people might read this article with an open mind, break out of their stupor, and start questioning these established practices. In order to improve, it is necessary to first acknowledge problems, and then take active steps towards addressing them.

What is the Daily Standup Meeting?

Image credit: taken from here

The Daily Standup Meeting is a “ritual” (quoting Jason Yip’s comprehensive article on the topic) in which the development team (and possibly other stakeholders) gathers to report on the progress of each individual member. They must in turn answer three questions:

  • What did you do yesterday?
  • What are you going to do today?
  • Are there any obstacles hindering your progress?

Let’s take a look at the meaning of each word, aside from “Meeting” which is obvious enough. As a “Daily” meeting, this happens routinely every day at a specific time. All those involved must stop what they are doing and participate, whether it is productive or not.

What I have just described is nothing new: I have participated in Daily Scrum meetings of this nature for years, and I’m pretty sure this goes back much father than my personal experience can account for.

A more recent twist, though, is the Daily “Standup” Meeting. The idea is to force people to stand up in order to keep meetings short, by having the physical discomfort as a disincentive to letting the meetings drag on unnecessarily.

Standing Up

This factor, on its own, speaks volumes. Rather than solving the problem of pointless meetings that are a complete waste of time, we choose to make people feel uncomfortable in the hope that the meetings will be kept short.

The first problem with this is how ridiculous and humiliating it is. Not even in church are you obliged to stand up, and perhaps at school there may have been instances where some posture was enforced in punishment (a practice that is probably considered barbaric by today’s standards).

But no. We are fully grown adults, who have been working for several years in the industry, and there is a perfectly comfortable sofa behind us, but we cannot use it, under pain of menacing glares from everybody in the room.

This also has no regard for any physical issues that the participants may be undergoing that may make this practice more uncomfortable than for the average human being. Such people should not have to disclose their personal problems to the team in order to waive the enforcement of a certain posture on the job, something that nobody has the right to enforce in the first place.

The second problem is that, well, it doesn’t work. Standing up or not, the tendency to ramble on is a lot more powerful than the discomfort (which most people can just as well get used to). Which brings us to the next section.

Excessively Long Standups

Do you remember what we’re supposed to do in standups? Basically, answer the following three questions, and then get on with our work:

  • What did you do yesterday?
  • What are you going to do today?
  • Are there any obstacles hindering your progress?

It is much harder than you would think, to stick to the plan. Despite having been brainwashed into enthusiastic and excited participation in this drudgery, people’s overexcitement tends to take over. There are many different scenarios which result in deviation from this template, and in turn cause Daily Standup Meetings to drag on. These are just a few of the most common I’ve encountered:

  • People want to justify their job and show that they are doing something, so they read out a whole list of every little detail they worked on since the last Daily Standup Meeting.
  • Stakeholders ask a lot of questions to the development team. They have a right to be answered, but the Daily Standup Meeting is not the right place.
  • Developers going off on a tangent and having a technical (or other) discussion between two or three people, but keeping everyone in the room.
  • Lots of people in the room, so it takes a while to go through what everyone has to report. This is mainly a problem when developer teams are large, but it can also be a result of participation of a lot of stakeholders who join the Standup. With 20 people in the room, it’s hard to keep meetings efficient.

Broadcasting Information

The format of the Daily Standup Meeting is such that the development team and any other stakeholders are present (whether physically or otherwise). It allows everybody in the room to be kept up to date with progress (in theory) and to be able to help remove any obstacles (again, in theory).

Personally, I believe that this is one of those cases where too much transparency is a bad thing. The main reason for this is that there is no reason why each person should need to broadcast their progress to so many people in the room. Typically, you work with one or two direct colleagues, and report to your manager. There is no need to keep a whole crowd in the loop.

Rather than focusing on what each of us needs to do, we instead feel the urge to get involved in what the whole team is up to. A consequence of this is that it becomes necessary to involve everybody in the team in order to take the slightest technical decision, which is really the opposite of the empowerment that should be enabled by giving developers such autonomy.

In practice, keeping everyone on the team updated has no real benefit. They may know you’re working on something, but they would not understand the business or technical decisions, and as a result, they would not be able to take up the task you are working on, in your absence.

Another problem is the direct access to the development team from higher-up stakeholders. Part of a manager’s job is to manage expectations, both to his superiors and to his subordinates. And yet, higher-level stakeholders and members now suddenly have full access to the fine details of what the development team is working on.

A real problem here is when members of the development team are cornered into publicly answering uncomfortable questions, possibly by external stakeholders or maybe even by members of their own team. Such things should really go through a manager or scrum master rather than being laid out in a public shit show. It is not uncommon for such meetings to degrade into blame games.

Even in the absence of such situations, it is often uncomfortable for developers to speak (let alone report their progress) in front of a small crowd, even if it consists of their own direct colleagues. While the stereotype of anti-social developers may be questionable, in my experience I have observed that most developers feel intimidated when speaking in front of people, just as they feel uncomfortable when someone (even a direct colleague) sits next to them while they work. It is thus quite easy for developers to go on the defensive if challenged about their work during the Daily Standup Meeting.

And this, of course, is something that should be completely avoided. As any manager worth his salt would know, you should never, ever reprimand someone in front of other people. This is also a fundamental flaw of retrospective meetings, although that is beyond the scope of this article.

Finally, the routine of the Daily Standup Meeting makes it convenient for people to provide updates and feedback during the meeting itself, rather than spontaneously as needed. Communication should be spontaneous and immediate; that is what keeps things efficient.

A trend I have seen is for companies with poor communication to turn to agile ceremonies to try to resolve the problem. Little do they realise that they are making it worse by adding more ritual baggage and introducing more middlemen. There is no substitute for real and effective communication.

What Alternative is There?

Honestly, what was so terrible about the good old system of hierarchical reporting lines? Think about it:

  • You talk directly to your manager when you’ve finished your work and need to work on something new.
  • Your manager can ping you periodically for progress, but otherwise you can focus on your work.
  • If there are impediments, the manager can bring the right people on board.
  • If you need to talk to some direct colleagues (e.g. to integrate with their API), just do so. No need to have the whole team in a meeting for that.
  • Requirements and other outside interferences go through the manager first.
  • Feedback and criticism take place one-on-one with the manager.
  • Each manager reports to their manager. No need for crowds.

That just about solves all the problems I’ve described earlier: notice the contrast of involving only the people who are necessary, as opposed to everyone. The important factors to make this work are having a manager who is organised and has excellent people skills, and that everybody on the team is able to communicate efficiently and effectively.

Agile methodologies will remain a reality for many years to come. But are they really an improvement?

Update 11th September 2017: Here are some reactions to this article on social media (might require login):

RabbitMQ: Who Creates the Queues and Exchanges?

Messaging is a fundamental part of any distributed architecture. It allows a publisher to send a message to any number of consumers, without having to know anything about them. This is great for truly asynchronous and decoupled communications.

The above diagram shows a very basic but typical setup you would see when using RabbitMQ. A publisher publishes a message onto an exchange. The exchange handles the logic of routing the message to the queues that are bound to it. For instance, if it is a fanout exchange, then a copy of the same message would be duplicated and placed on each queue. A consumer can then read messages from a queue and process them.

An important assumption for this setup to work is that when publishers and consumers are run, all this RabbitMQ infrastructure (i.e. the queues, exchanges and bindings) must already exist. A publisher would not be able to publish to an exchange that does not exist, and neither could a consumer take messages from an inexistent queue.

Thus, it is not unreasonable to have publishers and/or consumers create the queues, exchanges and bindings they need before beginning to send and receive messages. Let’s take a look at how this may be done, and the implications of each approach.

1. Split Responsibility

To have publishers and consumers fully decoupled from each other, ideally the publisher should know only about the exchange (not the queues), and the consumers should know only about the queues (not the exchange). The bindings are the glue between the exchange and the queues.

One possible approach could be to have the publisher handle the creation of the exchange, and consumers create the queues they need and bind them to the exchange. This has the advantage of decoupling: as new queues are needed, the consumers that need them will simply create and bind them as needed, without the publisher needing to know anything about them. It is not fully decoupled though, as the consumers must know the exchange in order to bind to it.

On the other hand, there is a very real danger of losing messages. If the publisher is deployed before any consumers are running, then the exchange will have no bindings, and any messages published to it will be lost. Whether this is acceptable is something application-dependent.

2. Publisher Creates Everything

The publisher could be configured to create all the necessary infrastructure (exchanges, queues and bindings) as soon as it runs. This has the advantage that no messages will be lost (because queues will be bound to the exchange without needing any consumers to run first).

However, this means that the publisher must know about all the queues that will be bound to the exchange, which is not a very decoupled approach. Every time a new queue is added, the publisher must be reconfigured and redeployed to create it and bind it.

3. Consumer Creates Everything

The opposite approach is to have consumers create exchanges, queues and bindings that they need, as soon as they run. Like the previous approach, this introduces coupling, because consumers must know the exchange that their queues are binding to. Any change in the exchange (renaming, for instance) means that all consumers must be reconfigured and redeployed. This complexity may be prohibitive when a large number of queues and consumers are present.

4. Neither Creates Anything

A completely different option is for neither the publisher nor the consumer to create any of the required infrastructure. Instead, it is created beforehand using either the user interface of the Management Plugin or the Management CLI. This has several advantages:

  • Publishers and consumers can be truly decoupled. Publishers know only about exchanges, and consumers know only about queues.
  • This can easily be scripted and automated as part of a deployment pipeline.
  • Any changes (e.g. new queues) can be added without needing to touch any of the existing, already-deployed publishers and consumers.

Summary

Asynchronous messaging is a great way to decouple services in a distributed architecture, but to keep them decoupled, a valid strategy for maintaining the underlying messaging constructs (in the case of RabbitMQ, these would be the queues, exchanges and bindings) is necessary.

While publisher and consumer services may themselves take care of creating what they need, there could be a heavy price to pay in terms of initial message loss, coupling, and operational maintenance (in terms of configuration and deployment).

The best approach is probably to handle the messaging system configuration where it belongs: scripting it outside of the application. This ensures that services remain decoupled, and that the queueing system can change dynamically as needed without having to impact a lot of existing services.

.NET Core 2.0: Referencing .NET Framework Libraries: A Topshelf Experiment

Referencing .NET Framework Libraries in .NET Core 1.1

There are many old libraries targeting the .NET Framework which, for various reasons, do not yet target .NET Core or .NET Standard. Topshelf, a fantastic library that helps you to easily create Windows services, is one of these. At the time of writing this article, the last release of Topshelf was 4.0.3 back in October 2016. There was no way that Topshelf could target .NET Standard because the .NET Standard spec did not support the APIs that are required for it to function.

This was a problem because there was no way you could create a Windows service using Topshelf for a .NET Core 1.1 application. Simply trying:

Install-Package Topshelf

…is bound to fail miserably:

If you want to make a Windows service out of an application targeting .NET Core 1.1, then you have to use an alternative such as NSSM.

What Changed in .NET Core 2.0

With the release of .NET Core 2.0 and .NET Standard 2.0, applications or libraries targeting either of these are able to reference old libraries targeting the full .NET Framework. Presumably this is because the .NET Core/Standard 2.0 implementations have enough API coverage to overlap with what the full framework was able to offer.

Quoting the .NET Core/Standard 2.0 announcement linked above:

“You can now reference .NET Framework libraries from .NET Standard libraries using Visual Studio 2017 15.3. This feature helps you migrate .NET Framework code to .NET Standard or .NET Core over time (start with binaries and then move to source). It is also useful in the case that the source code is no longer accessible or is lost for a .NET Framework library, enabling it to be still be used in new scenarios.

“We expect that this feature will be used most commonly from .NET Standard libraries. It also works for .NET Core apps and libraries. They can depend on .NET Framework libraries, too.

“The supported scenario is referencing a .NET Framework library that happens to only use types within the .NET Standard API set. Also, it is only supported for libraries that target .NET Framework 4.6.1 or earlier (even .NET Framework 1.0 is fine). If the .NET Framework library you reference relies on WPF, the library will not work (or at least not in all cases). You can use libraries that depend on additional APIs,but not for the codepaths you use. In that case, you will need to invest singificantly in testing.”

Example with Topshelf

In order to actually test this out, you’ll need to have Visual Studio 15.3 or later. You will also need to separately install the .NET Core 2.0 SDK.

In an earlier section, we tried installing Topshelf in a .NET Core 1.1 application, and failed. Let’s try doing the same thing with a .NET Core 2.0 application:

Install-Package Topshelf

The package installation works pretty well:

However, the warning that shows under the dependency is not very promising:

There’s only one way to find out whether this will actually work in practice.

Let’s steal the code from the Topshelf quickstart documentation:

public class TownCrier
{
    readonly Timer _timer;
    public TownCrier()
    {
        _timer = new Timer(1000) {AutoReset = true};
        _timer.Elapsed += (sender, eventArgs) => Console.WriteLine("It is {0} and all is well", DateTime.Now);
    }
    public void Start() { _timer.Start(); }
    public void Stop() { _timer.Stop(); }
}

public class Program
{
    public static void Main()
    {
        HostFactory.Run(x =>                                 //1
        {
            x.Service<TownCrier>(s =>                        //2
            {
               s.ConstructUsing(name=> new TownCrier());     //3
               s.WhenStarted(tc => tc.Start());              //4
               s.WhenStopped(tc => tc.Stop());               //5
            });
            x.RunAsLocalSystem();                            //6

            x.SetDescription("Sample Topshelf Host");        //7
            x.SetDisplayName("Stuff");                       //8
            x.SetServiceName("Stuff");                       //9
        });                                                  //10
    }
}

Nope, looks like Topshelf won’t work even now.

I guess the APIs supported by .NET Core 2.0 still do not have enough functionality for Topshelf work as-is. Other .NET Framework libraries may work though, depending on the dependencies they require. In the “.NET Core 2.0 Released!” video, one of the demos shows SharpZipLib 0.86 (last released in 2011) being installed in an ASP .NET Core 2.0 application. It is shown to build, but we don’t get to see whether it works at runtime.

It is still early, and I suppose we have yet to learn more about the full extent of support for .NET Framework libraries from .NET Core 2.0 applications and .NET Standard 2.0 libraries. The problem is that when evaluating a third-party library such as Topshelf, it’s difficult to determine whether its own dependencies fall within the .NET Standard API set. This looks to me like a matter of pure trial and error.

The Abysmal State of the Web in June 2017

This will be the last article in the Sorry State of the Web series (at least for the time being). The idea was to learn from the mistakes of other so-called ‘professional’ websites, ranging from silly oversights to illegal practices. Hopefully, the silliness encountered has also made some people smile.

However, with 11 articles over 6 months, I believe I’ve made my point enough times over. Despite all the technological advancements, the web is in a state that I can call sick at best, and that is mainly the result of clueless developers. I have some slight hope that things may get better, but given that most of the issues I pointed out have not been addressed to date, that hope is realistically very slim.

From my part, I want to focus less on beating a dead horse and more on learning technology and writing high quality articles. I don’t exclude revisiting this series in future if I feel it’s worth it though. Once again, I extend my heartfelt thanks to all those who have contributed entries for this article and the ones before it.

Banif: Random Virtual Keyboard

If you think that the mainstream banks in Malta have terrible websites (and recently I covered how Mediterranean Bank’s newly launched online investment platform took them several steps back), then you should really take a look at Banif Bank Malta.

To log into their online banking section, you have to enter a username and a password. This would be understandable, if not for the fact that the password field is disabled so you can’t actually type into it. Instead, you have to click on keys on a virtual keyboard. To make matters worse, this is not your usual QWERTY keyboard: the key placements are randomised.

Let’s consider a few reasons why this is a terrible idea:

  • It makes it a lot harder for users to type in their password (in terms of user experience).
  • It slows down password entry, both because one has to use the mouse vs the keyboard and because the random placement requires the eye to look for keys as opposed to using muscle memory. This makes it easier for people watching you enter the password to identify what you are actually entering, and it also makes you more likely to pick simpler passwords.
  • People looking over your shoulder can easily see what key the cursor is on, which defeats the purpose of password field obfuscation.
  • The restrictions on the password field are client-side and trivial to disable. This does no favours for server-side security, which should really be the main focus.
  • You cannot use a password manager.

Since I’m not a security expert, I presented this case to the community at Information Security Stack Exchange. From there, I got to two related existing questions:

It seems that the main reason why this horrendous technique is used is to counteract keyloggers, which at a basic level can’t track keypresses (since they are not happening) or mouse clicks (since the placement of keys on the screen changes).

However, as one of the best answers points out, this is merely an arms race between the bank and attackers. It’s a vicious circle in which attackers and banks take it in turns to step up their game. The end result is that customers are the ones paying the price, by having to deal with ridiculous security measures like this.

Dealing with keyloggers is hardly an excuse for this kind of rubbish. There are much more robust and orthodox ways of dealing with this sort of thing, such as one-time passwords or two-factor authentication.

Insecure Logins

One of the most common issues we’ve seen throughout this series is that of websites with login forms where the credentials are not transmitted over HTTPS. Thus it is not hard for them to be intercepted and read in clear text. Keeping up with tradition, we have a list of such examples this month.

We can start with American Scientist, which I see has since undergone a complete redesign and does currently use HTTPS for the whole website (including login). This is how it was just a couple of weeks ago:

Then we have the Malta Chamber of Advocates, which aside from very ridiculously presenting a homepage with no content whatsoever, is just another case of insecure login:

But wait! The next one, ironically, is from none other than Bank Info Security:

Then we have Great Malta (whatever that is supposed to mean):

Local newspaper The Malta Independent is no less guilty:

…and neither is Infobel:

In another case if irony, we can look at J. Grima & Co. Ltd. They are “Security & Fire Specialists”, but web security is clearly not one of their areas of expertise.

Excitable Web

I was very excited (!) to come across Excitable Web, because it is a prime example of the clueless developers I was mentioning earlier. It is of little importance that each time you load a page, the page seems to render without CSS for half a second before rendering properly; because we’ll focus on more interesting stuff here. If you click on the “Who We Are” link, we get this:

You can see there are a couple of MySQL errors displaying directly in the page due to deprecated code. Such an experienced professional should know that server-side errors should never be displayed directly to the visitor, as this may reveal vulnerabilities among other things.

These errors seem to have been fixed since then, so we’ll move onto the next thing: the writing. It’s really generous of the webmaster to give us:

“A Breif [sic] Background On With [sic] Whome [sic] You Are Dealing With”

You can find other such gems within the content itself. Thank you, Adrian. Now we really know who we are dealing with.

For extra points, spot one of my own blunders within that screenshot!

Flybussen Translations

Here’s a tiny oversight from Norwegian operator Flybussen. While their site has an English version, their calendar unfortunately doesn’t:

JobsPlus Going Below Minimum Wage

JobsPlus has by now become a regular in this series. Those who believe that we should have equal pay for equal work (which is a legal requirement, by the way) will be delighted to see this vacancy where the position advertises a salary range of between EUR4,500 and EUR70,000. What’s even funnier, though, is that EUR4,500 is actually below the minimum wage (another legal requirement) for a 40-hour full-time work week.

Legal requirements aside, this is just a case of missing validation by our award-winning friends at JobsPlus who should have a central role in avoiding precarious work and exploitation.

Kelly on Yellow Pages

If you take a look at the Yellow Pages entry for Kelly Industries, you’ll come to the conclusion that they have enough business to not give a rat’s ass about what potential customers think about their brand.

Creativity Centre

I’ve received reports about issues with the Malta’s National Centre for Creativity‘s payment processing engine, but I haven’t been able to verify them without actually attempting to make a purchase. However, I did notice this problem with the checkout button actually not being properly visible if you’re using a laptop (and thus a limited screen resolution):

For a National Centre for Creativity, I must also point out that they didn’t quite put a lot of creativity into the website’s design.

Mixed Content

Another common problem we’ve seen throughout the series is that of using HTTPS, but serving some content over HTTP. This is called Mixed Content, and it invalidates the trust guaranteed by a fully HTTPS website.

This month, we have Malta Gift Service (also guilty of using Comic Sans for their main header):

…and our dear friends at Scan:

Apostrophes of Doom

Given that my surname contains an apostrophe, this often makes it a pain to deal with validation that unreasonably decides that an apostrophe is an invalid character. I’ve written about this especially in the original “The Sorry State of the Web in 2016“. There is no real reason to not accept apostrophes if you’re using proper practices (e.g. using prepared statements) to prevent SQL injection.

Unfortunately, Microsoft has decided that my surname cannot have an apostrophe:

I suppose I will need to remove the apostrophe from my identity card if I want to ever get a job at Microsoft.

Piscopo Gardens

The Piscopo Gardens website has been down for I don’t know how long due to some internal server error.

Aron isn’t doing a very good job at keeping the site up and running.

Robert Half

Swiss recruiter Robert Half believes that “It’s time we all work happy.™” (so much that a trademark was apparently filed).

That obviously doesn’t apply to their own website, which clearly doesn’t work if you enter “.net” in the search field:

Now I understand the name. Their website only Half works.

Ryanair Mischief

We noticed a couple of things on Ryanair’s website that are more sneaky practices than examples of bad web design per se.

First, there’s the newsletter checkbox that is opt-out rathern than opt-in (i.e. it automatically signs you up if you ignore it and leave it unchecked):

Then there’s this appeal to fear the middle seat:

Oh dear, not the middle seat!

Image credit: Taken from Wikipedia

Better to go for a team-building treasure hunt in 35-degrees-Celsius weather with a laptop on my back than be stuck in a middle seat! Actually, no. Give us a break, Ryanair.

Conclusion

I am happy to have managed to raise awareness about bad practices in web design with this series. I know this because I have heard several reports of companies that I have pissed off. I am a lot less happy that these companies have not really done much about it despite all this. That is their problem now. No doubt others have learned from the countless issues pointed out.

Let’s continue to make companies with a web presence understand that such a public face requires a high level of professionalism, and that they will lose business if they don’t step up their game.

Once again I would like to thank all the contributors to this series, and also the readers who have loyally followed it.

Your First Microsoft Orleans Cluster

If you’re planning to use Microsoft Orleans in production, you need to look beyond the lonely silos that we’ve built in the articles thus far. Orleans is designed to work in a cluster, such that a large number of grains can be distributed among multiple silos. In case of silo failure, its grains are reactivated at other silos that are still alive.

In this article, we’ll create a simple silo and run multiple interconnected instances of it in order to set up a cluster.

For the scope of this article, we’ll use the default cluster membership provider: MembershipTableGrain. This is not intended to be used in production, but will allow us to focus on getting a simple cluster up and running. Setting up different cluster membership providers is non-trivial and requires separate articles for each.

Note: this article is based on Orleans 1.4.2 using .NET Framework 4.6.2.

Note: the source code for this article is in the OrleansFirstCluster folder within the Gigi Labs BitBucket repository.

Setting Up An Example

To set up a cluster, the Dev/Test Host project template we’ve been using so far is no longer suitable. Instead, we have to set up the full project structure. This is covered by the latter part of “Getting Started with Microsoft Orleans” and there is no point in repeating it here.

Don’t write any code yet though. I recently learned that all that AppDomain stuff is not necessary unless you’re planning to run Silo and Client in the same application, so we’ll go for a cleaner approach.

We’ll also install the Orleans Dashboard (see: “A Dashboard for Microsoft Orleans“) in the Silo project. This will give us an idea how grains are spread across the cluster later.

Install-Package OrleansDashboard

Hence, when setting up the Silo configuration, remember to include the configuration for the Dashboard:

<?xml version="1.0" encoding="utf-8"?>
<OrleansConfiguration xmlns="urn:orleans">
  <Globals>
    <SeedNode Address="localhost" Port="11111" />
    <BootstrapProviders>
      <Provider Type="OrleansDashboard.Dashboard" Name="Dashboard" />
    </BootstrapProviders>
  </Globals>
  <Defaults>
    <Networking Address="localhost" Port="11111" />
    <ProxyingGateway Address="localhost" Port="30000" />
  </Defaults>
</OrleansConfiguration>

We can now start adding some code. First, we need a grain in our Grains project. What the grain actually does doesn’t matter. We just want to create a large number of grains to see them spread out over the cluster.

    public class UselessGrain : Grain, IUselessGrain
    {
        public Task DoNothingAsync()
        {
            return Task.CompletedTask;
        }
    }

Note: if you’re using a .NET Framework version prior to 4.6, then you’ll need to use TaskDone.Done instead of Task.CompletedTask.

The corresponding interface goes in the Interfaces project:

    public interface IUselessGrain : IGrainWithIntegerKey
    {
        Task DoNothingAsync();
    }

Doing away with all the AppDomain junk, the following code should be enough for a simple Silo:

        static void Main(string[] args)
        {
            Console.Title = "Silo";

            try
            {
                using (var siloHost = new SiloHost("Silo"))
                {
                    siloHost.LoadOrleansConfig();
                    siloHost.InitializeOrleansSilo();
                    var startedOk = siloHost.StartOrleansSilo(catchExceptions: false);
                    Console.WriteLine("Silo started successfully!");

                    Console.WriteLine("Press ENTER to exit...");
                    Console.ReadLine();
                    siloHost.ShutdownOrleansSilo();
                }
            }
            catch (Exception ex)
            {
                Console.WriteLine(ex);
            }
        }

Apart from the AppDomain logic, another thing we’re doing differently from usual here is that we’re calling StartOrleansSilo() with catchExceptions set to false. In case the silo fails to initialise, this gives us the ability to inspect the details of the failure within the exception, rather than have Orleans silently swallow it and simply return false.

On the client side, we can use an adaptation of the client code from “Getting Started with Microsoft Orleans“:

        static void Main(string[] args)
        {
            Console.Title = "Client";

            var random = new Random();
            var config = ClientConfiguration.LoadFromFile("ClientConfiguration.xml");

            while (true)
            {
                try
                {
                    GrainClient.Initialize(config);
                    Console.WriteLine("Connected to silo!");

                    while (true)
                    {
                        var grainId = random.Next();
                        var grain = GrainClient.GrainFactory.GetGrain<IUselessGrain>(grainId);
                        grain.DoNothingAsync();
                    }
                }
                catch (SiloUnavailableException)
                {
                    Console.WriteLine("Silo not available! Retrying in 3 seconds.");
                    Thread.Sleep(3000);
                }
            }
        }

In the inner infinite-while-loop we’re taking random grain IDs and bombarding them with messages. The idea is to create a lot of grain instances that we can visualise. Since this is very heavy, you’ll see Orleans giving warnings, and high latencies from the Dashboard at times.

Running a 3-Node Cluster

We will now run 3 instances of the same silo. Each instance must have different ports configured. This is the configuration for the first silo, which we already set up earlier:

<?xml version="1.0" encoding="utf-8"?>
<OrleansConfiguration xmlns="urn:orleans">
  <Globals>
    <SeedNode Address="localhost" Port="11111" />
    <BootstrapProviders>
      <Provider Type="OrleansDashboard.Dashboard" Name="Dashboard" />
    </BootstrapProviders>
  </Globals>
  <Defaults>
    <Networking Address="localhost" Port="11111" />
    <ProxyingGateway Address="localhost" Port="30000" />
  </Defaults>
</OrleansConfiguration>

A silo needs 2 ports: one to communicate with other silos (the Networking endpoint) and one for clients to connect to it (the ProxyingGateway endpoint). The client can connect to any node on the cluster.

In production scenarios, Orleans silos are all equal, and there is no concept of a primary and secondary silo. However, when you use the default MembershipTableGrain cluster membership, then all information regarding the silos on the cluster is stored within a grain in one of the silos. As a result, the silo containing the MembershipTableGrain is denoted as the Primary Silo. It must be started before the others, and the entire cluster is messed up if it goes down. Naturally, this is not good, and you should look into other cluster membership providers.

In such a setup, the SeedNode configuration specified in all silos must be the endpoint of the Primary silo. Let’s see what the configuration for our second silo instance looks like:

<?xml version="1.0" encoding="utf-8"?>
<OrleansConfiguration xmlns="urn:orleans">
  <Globals>
    <SeedNode Address="localhost" Port="11111" />
    <BootstrapProviders>
      <Provider Type="OrleansDashboard.Dashboard" Name="Dashboard" Port="8081" />
    </BootstrapProviders>
  </Globals>
  <Defaults>
    <Networking Address="localhost" Port="11112" />
    <ProxyingGateway Address="localhost" Port="30001" />
  </Defaults>
</OrleansConfiguration>

Aside from changing the Networking and ProxyingGateway ports, we are also using a different port for the Dashboard (default is 8080). Each silo has its own Dashboard (although they all show the same information), and they cannot all run from the same port.

Similarly, the configuration for our third silo instance is just a matter of changing ports:

<?xml version="1.0" encoding="utf-8"?>
<OrleansConfiguration xmlns="urn:orleans">
  <Globals>
    <SeedNode Address="localhost" Port="11111" />
    <BootstrapProviders>
      <Provider Type="OrleansDashboard.Dashboard" Name="Dashboard" Port="8082" />
    </BootstrapProviders>
  </Globals>
  <Defaults>
    <Networking Address="localhost" Port="11113" />
    <ProxyingGateway Address="localhost" Port="30002" />
  </Defaults>
</OrleansConfiguration>

We can then start the 3 silo instances and the client:

On my system, the load is just too much and Orleans just dies after around 64k activations. So let’s add a little delay in the random message loop to give Orleans some room to breathe:

                    while (true)
                    {
                        var grainId = random.Next();
                        var grain = GrainClient.GrainFactory.GetGrain<IUselessGrain>(grainId);
                        Thread.Sleep(50);
                        grain.DoNothingAsync();
                    }

After running it again, what I see is that grains are allocated mainly to the primary silo initially, but they are distributed more evenly across the other silos after around 1,000 activations:

I am not sure why they are not evenly distributed from the start. My guess is that either it is more efficient to have them all in one place if the number of activations is small, or the silos need time to coordinate between themselves before this happens (which would explain why, without a delay, all activations are allocated on the primary node).

Single Point of Failure

Close the primary silo.

Since the Primary silo contains the MembershipTableGrain, all information about the cluster dies with it. The remaining silos and clients will not recover automatically even if the Primary silo is brought up again. They in turn will have to be restarted. This is because, as we saw earlier, Secondary silos must start after the primary one. When the Primary silo is brought back, it effectively starts a fresh cluster and does not know about any other silos until they join.

Conclusion

We have seen how to get a very basic Orleans cluster working with multiple silos sharing the burden of holding the grains. However, this is hardly an ideal setup because (a) cluster membership information is held in memory and represents a single point of failure, and (b) the fact that I ran all silos on the same machine made them subject to the same physical resource constraints as if I were running a standalone silo.

For better results, run different silos on different machines, and use a decent cluster membership provider. Orleans supports the following:

  • MembershipTableGrain (not realiable, use for testing only)
  • SQL Server
  • Azure Table Storage
  • Apache ZooKeeper
  • Consul
  • DynamoDB

Update 20th June 2017: I am told that Azure Service Fabric should also be supported. As for database implementations of cluster membership, these are not limited to just SQL Server. You may use any supported ADO .NET provider, which at the moment includes SQL Server, MySQL/MariaDB, or PostgreSQL. To clarify: while the PostgreSQL storage provider for grain persistence is not yet available, its use as a cluster membership provider is supported.

Saving Screenshots in SDL2

Saving screenshots is a simple and common feature in many games. It allows us to capture the image of the game being rendered on the screen at any given time.

While this might sound easy, let us remember that the image rendered to the screen might have a lot of different overlaid surfaces, meaning that it would be a pain to recompose that image on the side of the CPU in the same way that we’re composing the image to the texture that eventually ends up in video memory.

Fortunately, this Stack Overflow answer provides a simple solution. SDL2 provides the SDL_RenderReadPixels() function, which can be used to read pixel data back from video memory. This is generally discouraged, because it is very costly to do such a thing, but taking screenshots is a one-off operation where it makes perfect sense.

The following code shows how screenshot capture was implemented in Ultima 1 Revenge. It is only slightly different from the code in the aforementioned Stack Overflow answer:

void SosariaInputController::SaveScreenshot()
{
	const Uint32 format = SDL_PIXELFORMAT_ARGB8888;
	const int width = 640;
	const int height = 400;
	auto renderer = sdl2Core->GetRenderer();

	SDL_Surface *surface = SDL_CreateRGBSurfaceWithFormat(0, width, height, 32, format);
	SDL_RenderReadPixels(renderer, NULL, format, surface->pixels, surface->pitch);
	SDL_SaveBMP(surface, "screenshot.bmp");
	SDL_FreeSurface(surface);
}

While I am not showing where the sdl2Core object is coming from, just assume that it stores an instance of SDL_Renderer. With that available, all that is needed is to create a surface using one of the functions that SDL2 provides, call SDL_RenderReadPixels() to transfer the pixels from video memory to the surface, and then actually do something with the surface (in this case, we are saving it to a bitmap file). If you wanted to read back only a portion of the screen, you would pass in an SDL_Rect instead of NULL as the second parameter to SDL_RenderReadPixels().

This simple code is used to save screenshots in Ultima 1 Revenge, such as the one shown earlier.

The Sorry State of The Web: 3 Group Special

It’s been around 15 years since I first came across Web Pages That Suck. Coming from a time when flashy Geocities-style websites were the order of the day, it was a web nitpicker’s paradise. This is where the term Mystery Meat Navigation (which I have written about in the past) was actually invented.

The very premise behind Web Pages That Suck, “learn good web design by looking at bad web design”, is something that has fascinated me back then, and still does to this day (in fact, it is one of the main reasons behind the Sorry State of the Web series).

Today, we will look at a family of related websites (belonging to a single group of companies) which I’m sure would qualify as first class citizens of Web Pages That Suck.

Enter MyKrypto

I first heard about MyKrypto on the radio. The ad described Bitcoin as a currency just like any other – and said that you could produce it! An old version of their website, which I obtained via the Google Web Cache, is along the same lines of the radio ad:

“Malta has the Euro, UK has the Pound and USA has the Dollar, the Internet has the Bitcoin. Bitcoin is digital and produced by computers..start producing money today!!”

While it’s true you can produce Bitcoins, this feels a lot like a scam in that it’s urging people to print their own money (in a way) without telling them about the risk or the difficulty involved in actually mining Bitcoins. In fact, the website also used to say that Bitcoin mining is a secure investment:

Whether Bitcoin mining is really a secure investment is debatable (although one can get an idea by looking at market crashes that have occurred in the past). In any case, while I’m not a lawyer, I don’t believe a company can legally give investment advice unless it is an authorised financial institution, especially without evaluating the risk portfolio of potential investors.

MyKrypto Home Page

The above selling points were removed, and the site transitioned into a different realm of madness. The site’s homepage had this image with Comic Sans text:

This was eventually replaced by the text image we see today:

Although the text changed, the link behind it remained the same. It’s basically a Google link (notice the URL) that takes you to this Satasoshi graphic on Deviantart:

So, in case it’s not clear, let’s summarise the fails that occur just within that little text image:

  • Using an image to show text with a particular font.
  • Using Comic Sans on what is supposed to be a serious website.
  • Linking to a Google search result rather than to an actual webpage.
  • Completely failing to understand what you’re selling (it’s a Satoshi, not Satasoshi, and the horse has nothing to do with it).

MyKrypto Mining Page

Let’s move over to MyKrypto’s Mining page. It now looks something like this:

While the image is totally out of place – and we’ll get to it – this is on the whole not too bad. Let’s take a look at what the previous version of this page (from just a couple of weeks ago) used to look like:

First, the title.

“What is minning?”

I don’t know. Perhaps this lovely lady might be able to answer that.

Image credit: taken from Wikipedia’s Minnie Mouse page

Next, we can take a look at the text.

“Do you look up for information on the internet? Did you ever google for a product or information? Did you ever wondered how google is so intelligent by search for the right information? Google is based on mathematical calculations. FIND X. Let’s say I want to google for ‘this week top 20 music list’ as you type in the google box and press search google, google will do all the calculations to find ‘this week top 20 music list ‘. this is called Math (Algebra) when we were at school we used to have the same problem solving; – FIND X.”

It is simply beyond belief how much crap these guys have managed to fit into a single paragraph. But beyond that, take a look at the image below that paragraph in the screenshot. It’s a popular joke that has been circulating for many years: a clueless student answered a mathematical problem in a witty manner. Of course, whoever built the MyKrypto website didn’t get the joke, and put the image there as an example of mathematics. Go figure.

List of fails in this section:

  • Terrible use of English (if it can be called that).
  • Google does not solve algebraic problems to give you your search results.
  • Don’t lie about what you used to do at school, if you evidently know nothing about English, mathematics, or computing.
  • Try to understand what an image actually means, before ripping it off.
  • Try to understand what you’re talking about in the first place.

MyKrypto Audio

MyKrypto automatically plays audio.

That’s something really annoying, especially if you happen to already be playing music. It’s also of questionable legality depending on whether the site has the right to distribute that music.

Besides, using some cheesy 70s disco background music – reminiscent of Earth, Wind & Fire – is totally not appropriate on a company website.

MyKrypto Mobile

Mobile users will be disappointed to find out that they can’t really browse the entirety of the site because the navigation is simply not available:

MyKrypto Plagiarism

To be fair, messed up paragraphs like the one we saw earlier are a rare sight on this site. In fact, a lot of MyKrypto’s content is blatantly stolen from other websites.

Let’s see some examples:

I think they should plagiarise more. It would make them look a lot less silly.

CEO Plagiarism

Of course, the plagiarism on MyKrypto makes perfect sense if we look at the LinkedIn profile of 3 Group CEO Dario Azzopardi (MyKrypto is part of 3 Group):

When I first read this, I thought it was really weird as a job description. So I Googled part of it.

Google did its algebra (!), and what do you know

…and further down…

3 Group: Questionable Stuff

Having seen all this, I thought it was just as well to check what else 3 Group actually do.

3 Group do IT Services, IPTV, and E-Money. That’s a nice name for the Bitcoin stuff we’ve seen above. They actually got the link wrong, and E-Money points to IPTV.

If we take a look at IT Services, we get to this horrendous page with a background animation driving you nuts while you try to read text with very bad contrast:

Further down that page, 3 Group are trying to convince people that free antiviruses are bad, and that they should instead pay 3 Group to install McAfee for them:

Towards the bottom, you can see some grey text representing a link to Intel’s homepage. Of course, they didn’t bother to actually make it a link that you can just click on. What’s even worse is that the superhero on the left is an image overlaid onto the text where the link is, so you can’t even select and copy it.

Right, what else do 3 Group do? Ah yes, IPTV. It’s interesting how they have this “Legal” page under the IPTV section, claiming that “IPTV is 100% legal”, and quoting some court case from the European Court of Justice.

This is noteworthy because:

  • Naturally, a company encourages trust by stressing that its services are 100% legal.
  • This company knows a lot about copyright, given the aforementioned plagiarism.
  • It claims that “watching streams even those which are illegal is not an act of copyright infringement”. So it’s ok if it’s illegal, as long as it’s not copyright infringement, right?

Well, they say IPTV is legal, so it must be true.

Image credit: taken from here

That’s curious, because I could swear I recently read an article about this Kodi TV streaming service being declared illegal across the EU:

Conclusion

While 3 Group’s web design is appalling at best, this is not nearly as worrying as their questionable business practices. As an exception in this series, I hope not only that web designers/developers learn from the mistakes we have seen here, but also that potential customers do some proper research and understand what these guys are actually trying to sell to them.

The Pitiful State of the Web in May 2017 (Part 2)

This article is a continuation of The Pitiful State of the Web in May 2017 (Part 1) and a part of the Sorry State of the Web series. I and the others who contributed to the content of this series hope that web developers will learn from the mistakes of others and produce better quality websites.

Dakar: Language Issues and Insecure Login

Dakar Software Solutions is a well-known name locally, especially in the realm of payroll systems.

They had this little glitch with the language of dates in the news of their Dakinet product (which might be fixed now):

Also, Dakar joins the long list of websites that offer insecure login:

Insecure Login Galore

As you can imagine, Dakar is not alone in failing to transmit user credentials securely. We’ve seen a lot of these before, and we have a lot more to show here.

For starters, we have Freelance Malta. Since all of the site is based on insecure HTTP, the login form and both registration forms transmit credentials insecurely:

Then we have Gizmodo, the popular tech website:

KeepMePosted is a similar offender:

And then we have MyMoneyBox (part of the MFSA family), which given its name should know better about security. In fact, it seems to have now gone HTTPS, so the login is now secure. But as you can see below, it wasn’t until recently:

Couchsurfing: Invalid SSL Certificate

I recently caught the Couchsurfing blog giving invalid certificate warnings:

Oops. Looks like the certificate had expired.

Needless to say, it is useless to use HTTPS if it is not trusted. Fortunately, this issue has since been fixed.

Malta Police Force: Passwords In Email, Freedom of Information Link

The Malta Police Force website offers a number of services including filing a police report online. At the bottom of the declaration where you’re about to file a report, there’s a link to the Freedom of Information Act:

Unfortunately, it doesn’t work:

That error is actually coming from elsewhere on the government network (looks like it’s the Department of Information). Either the Malta Police Force need to fix their link to point elsewhere, or the DOI needs to fix a problem in their SharePoint system.

There’s something a lot worse, though. Some people have reported that when you file a police report, you choose a password, which is then sent to you via email.

This image was contributed by someone who actually filed a real police report. Aside from various spelling issues in the email, you can see that the password (obfuscated here for obvious reasons) is included.

This is something you always want to avoid because you can never assume that email is a secure channel on which to send sensitive information such as credentials or credit card information. Good practice is to let users choose their password over a secure channel (which the system reportedly already does), store it securely using a one-way hash, and provide the means to reset it using limited-time tokens in case the password is forgotten.

Rizzo Farrugia – Broken Link For New Equity

When new shares under the symbol “PG” were listed on the Malta Stock Exchange, Rizzo Farrugia were quick to add it to their own list:

However, they were not nearly as fast at creating the detail page that the listing links to:

No big deal there. It was fixed the next day.

PWC Refresh Form

PricewaterhouseCoopers has this newsletter signup form. It has a reset button. Something pretty normal, you’d think, until you see that it refreshes the entire page!

Form resetting functionality has been built into browsers since long before I started creating websites (15 years and counting). In this case, I see they wanted to reset the CAPTCHA. But they already have functionality to reset the CAPTCHA without reloading the page (the orange round arrow next to the CAPTCHA), so why reload the whole page just to reset a form?

IDPC: Line Spacing

The Office of the Information and Data Protection Commissioner has a form where you can submit complaints:

What I’d like to call out here is the questionable design choice of using massive line spacing, which is especially noticeable in the Complaint text area since about half of the tiny box is wasted with empty space.

Line Separator Characters

JobsPlus, whose encoding issues we have already visited in “The Broken Web of March 2017“, is now also exhibiting these weird LSEP characters:

It’s okay though. Perhaps they can’t sanitise their data, but they still get to keep their eBusiness Award!

I have also spotted the same problem at Creative Jobs:

Summary

Transmitting credentials insecurely remains one of the most common issues on websites today, and it is completely unacceptable. Depending on the nature of the user account, this might not be as risky as transmitting credit card details insecurely (something we’ve also seen in abundance over the past few months), but that does not relieve websites from their duty as data controllers to transmit sensitive data securely.

It is also important to test websites properly in order to identify broken links and data-related issues as we have seen.

Finally, secure transmission of sensitive data does not stop at using HTTPS. SSL certificates must be integral and trusted, otherwise it is just as good as not having HTTPS. Email is not a secure channel, so don’t use it to send sensitive data, especially if there exist alternative data flows where you don’t have to.

How to Update All Orleans Packages

Sometimes, you will want to upgrade to a newer Microsoft Orleans version. Like today, since Orleans 1.4.2 has just been released.

Thing is, it can be a bit of a pain to update all these packages manually across different projects. It’s also quite error-prone. It has happened in the past that some of those packages ended up on one version, and others ended up on another, causing runtime mayhem.

Fortunately, there’s a simple way to get them all in sync with minimal effort. Just use the following command (taken from the Grain Persistence documentation) in the Package Manager Console:

Get-Package | where Id -like 'Microsoft.Orleans.*' | foreach { update-package $_.Id }

This will go through all the packages that start with “Microsoft.Orleans.” and update them.

Orleans Grain Persistence with the ADO .NET Storage Provider

Yesterday, we introduced grain persistence in Microsoft Orleans, using the volatile MemoryStorage provider to quickly and easily learn how to set up and work with storage providers.

Today, we will use the ADO .NET Storage Provider in order to save our grain state to a database. Currently, we can use this with either SQL Server or MySQL.

Note: MariaDB is a drop-in replacement for MySQL. Thus, the ADO .NET Storage Provider works with it too.

Note: While there is a partial script for PostgreSQL, it is not yet supported.

Update 9th June 2017: Some material from this article was contributed towards the official Grain Persistence documentation.

Example Scenario

Like in yesterday’s article, we’ll use a very simple setup based on the Orleans Dev/Test Host project. We’ll use the same classes and interface:

    public interface IPersonGrain : IGrainWithStringKey
    {
        Task SayHelloAsync();
    }

    public class PersonGrainState
    {
        public bool SaidHello { get; set; }
    }

    [StorageProvider(ProviderName = "OrleansStorage")]
    public class PersonGrain : Grain<PersonGrainState>, IPersonGrain
    {
        public async Task SayHelloAsync()
        {
            string primaryKey = this.GetPrimaryKeyString();

            bool saidHelloBefore = this.State.SaidHello;
            string saidHelloBeforeStr = saidHelloBefore ? " already" : null;

            Console.WriteLine($"{primaryKey}{saidHelloBeforeStr} said hello!");

            this.State.SaidHello = true;
            await this.WriteStateAsync();
        }
    }

If we give our PersonGrain the name of “Joe” and call SayHelloAsync(), then it will write “Joe said hello!” to the console. After that, it sets the SaidHello property in its grain state to true, and saves the updated state to the database via the configured storage provider. The next time SayHelloAsync() is called, it will find that the value of SaidHello is true, so the output will be “Joe already said hello!” instead.

In the main program, let’s add the following code (same as in yesterday’s article) to invoke SayHelloAsync() on a grain. This needs to go in the place of the TODO comment that was generated as part of the project template.

            var joe = GrainClient.GrainFactory.GetGrain<IPersonGrain>("Joe");
            joe.SayHelloAsync();

Now, let’s move on to configure our storage provider.

Setting up the ADO .NET Storage Provider

Install the following package:

Install-Package Microsoft.Orleans.OrleansSqlUtils

Navigate to the folder where the package was installed alongside the project, and you’ll find folders in there for different databases.

In there you’ll find scripts you’ll need to run in order to set up the Orleans database, for different database vendors. Each script contains a table for storage and several others for cluster membership and other internal Orleans operations. For the scope of this article, we’re only concerned with storage.

Note: while there is a script for PostgreSQL, at the time of writing this article, it does not support storage.

The steps to configure an ADO .NET Storage Provider are as follows:

  1. Create the database specific for the database vendor you want.
  2. Install a NuGet package specific for the database vendor you want.
  3. Set up the storage provider in code or XML configuration. Set the AdoInvariant setting (optional only if you’re using SQL Server).

Using SQL Server for Grain Persistence

Setting Up the Database

When using SQL Server, you’ll want to use the SQLServer\CreateOrleansTables_SqlServer.sql script.

Open SQL Server Management Studio. Right click on Databases, and create a new one.

Call it whatever you like, e.g. OrleansStorage.

Open a query window based on your new database. Copy in the aforementioned script, and run it.

Installing the NuGet Package

Next, install the following package:

Install-Package System.Data.SqlClient

Configuring via Code

Finally, we need to configure the storage provider itself. We can do this in code by replacing the following line in OrleansHostWrapper.cs:

config.AddMemoryStorageProvider();

…with code that sets up an ADO .NET storage provider. First, we declare a connection string:

const string connStr = "Server=.\SQLEXPRESS;Database=OrleansStorage;Integrated Security=True";

Then, we can set up the ADO .NET storage provider using either a registration call that takes the storage provider’s type name as a string:

            var typeName = "Orleans.Storage.AdoNetStorageProvider";
            var properties = new Dictionary<string, string>()
            {
                ["DataConnectionString"] = connStr,
                ["AdoInvariant"] = "System.Data.SqlClient"
            };

            config.Globals.RegisterStorageProvider(typeName, "OrleansStorage", properties);

…or its generic equivalent:

config.Globals.RegisterStorageProvider<AdoNetStorageProvider>("OrleansStorage", properties);

Note that there is also a generic extension method we could use as a shortcut:

config.AddAdoNetStorageProvider("OrleansStorage", connStr);

However, I recommend against using this method. As per an issue I just opened, there is no way to set the AdoInvariant (which is optional for SQL Server but necessary when working with other database vendors), and the default serialization format is XML (whereas the default is usually binary). Thus, if you switch between this method and the earlier ones while relying on defaults, you will get deserialization errors.

Configuring via XML

On the other hand, to configure the storage provider using an XML configuration file (which is generally a better approach), first we have to add a file named OrleansConfiguration.xml and set it to copy to the output directory.

Add the following to the file:

<?xml version="1.0" encoding="utf-8"?>
<OrleansConfiguration xmlns="urn:orleans">
  <Globals>
    <StorageProviders>
      <Provider Type="Orleans.Storage.AdoNetStorageProvider"
                Name="OrleansStorage"
                AdoInvariant="System.Data.SqlClient"
                DataConnectionString="Server=.\SQLEXPRESS;Database=OrleansStorage;Integrated Security=True"/>
    </StorageProviders>
    <SeedNode Address="localhost" Port="22222" />
  </Globals>
  <Defaults>
    <Networking Address="localhost" Port="22222" />
    <ProxyingGateway Address="localhost" Port="40000" />
  </Defaults>
</OrleansConfiguration>

Then, replace the following code in OrleansHostWrapper.cs:

            var config = ClusterConfiguration.LocalhostPrimarySilo();
            config.AddMemoryStorageProvider();
            siloHost = new SiloHost(siloName, config);

…with this:

            siloHost = new SiloHost(siloName);
            siloHost.ConfigFileName = "OrleansConfiguration.xml";

Testing It Out

Once you have configured everything, run the program:

It says “Joe said hello!” Remember that the state should have been updated at the end of the method that writes that message. Let’s verify it by closing the program, and running it again:

This time, it says “Joe already said hello!” That means that the state was correctly read from the database.

Using MySQL for Grain Persistence

Note: this also works for MariaDB, which is a drop-in replacement for MySQL.

Setting Up the Database

Deep beneath the folder where the Microsoft.Orleans.SqlUtils package gets installed, you’ll find a MySql\CreateOrleansTables_MySql.sql script. We’ll use this for MySQL or MariaDB.

Use your favourite administrative tool to create a new database, and give it a name (e.g. “orleansstorage”). Paste the script in a query window, and run it against the new database.

Installing the NuGet Package

Install the following package:

Install-Package MySql.Data

Configuring via Code

We can replace the AddMemoryStorageProvider() call in OrleansHostWrapper.cs with code to set up our ADO .NET provider. First, let’s put our connection string in a variable to keep the rest of the setup code concise:

const string connStr = "Server=localhost;Database=orleansstorage;uid=xxx;pwd=xxx";

We can now configure the storage provider, using either a named type:

            var typeName = "Orleans.Storage.AdoNetStorageProvider";
            var properties = new Dictionary<string, string>()
            {
                ["DataConnectionString"] = connStr,
                ["AdoInvariant"] = "MySql.Data.MySqlClient"
            };

            config.Globals.RegisterStorageProvider(typeName, "OrleansStorage", properties);

…or a generic method based on the type itself:

config.Globals.RegisterStorageProvider<AdoNetStorageProvider>("OrleansStorage", properties);

As you can see, this is almost identical to the configuration for SQL Server. We’re using the exact same ADO .NET storage provider, and changing just the connection string and the AdoInvariant which identifies the underlying database vendor.

Configuring via XML

Instead of hardcoding our configuration, we have the option to read it from an XML file. Let’s create a file named OrleansConfiguration.xml, set it to copy to the output directory, and give it the following content:

<?xml version="1.0" encoding="utf-8"?>
<OrleansConfiguration xmlns="urn:orleans">
  <Globals>
    <StorageProviders>
      <Provider Type="Orleans.Storage.AdoNetStorageProvider"
                Name="OrleansStorage"
                AdoInvariant="MySql.Data.MySqlClient"
                DataConnectionString="Server=localhost;Database=orleansstorage;Uid=xxx;Pwd=xxx"/>
    </StorageProviders>
    <SeedNode Address="localhost" Port="22222" />
  </Globals>
  <Defaults>
    <Networking Address="localhost" Port="22222" />
    <ProxyingGateway Address="localhost" Port="40000" />
  </Defaults>
</OrleansConfiguration>

Then, as we did before in the SQL Server section, replace the following code in OrleansHostWrapper.cs:

            var config = ClusterConfiguration.LocalhostPrimarySilo();
            config.AddMemoryStorageProvider();
            siloHost = new SiloHost(siloName, config);

…with this:

            siloHost = new SiloHost(siloName);
            siloHost.ConfigFileName = "OrleansConfiguration.xml";

This allows us to bypass the default localhost silo configuration and read the configuration from our file instead.

A Word On Formats

ADO .NET Storage providers can save data in one of three formats: JSON, XML, or a compact binary format.

The default format is binary, and while it is compact, it means the data is opaque and you can’t really work with it directly. This could be a problem if you need to patch, migrate, or troubleshoot the data. XML is bloated, so I generally recommend using JSON. It’s lightweight (although not as much as the binary format) and is easy to work with.

You can switch between them by setting the appropriate property in either code or XML configuration:

  • UseJsonFormat="true" uses JSON.
  • UseXmlFormat="true" uses XML.
  • UseBinaryFormat="true" uses binary, which is the default setting.

Ideally you should set this only the first time. If you change this setting when you already have data, you’ll have a mess of formats and will not be able to read old data.

Summary

Install the base package for the ADO .NET Storage Provider:

Install-Package Microsoft.Orleans.OrleansSqlUtils

Then set up as follows depending on the database vendor:

SQL Server MySQL
Script SQLServer\CreateOrleansTables_SqlServer.sql MySql\CreateOrleansTables_MySql.sql
NuGet Package System.Data.SqlClient MySql.Data
AdoInvariant System.Data.SqlClient MySql.Data.MySqlClient

You can set the following propeties:

Name Type Description
Name String Arbitrary name that persistent grains will use to refer to this storage provider
Type String Set to Orleans.Storage.AdoNetStorageProvider
AdoInvariant String Identifies the database vendor (see above table for values; default is System.Data.SqlClient)
DataConnectionString String Vendor-specific database connection string (required)
UseJsonFormat Boolean Use JSON format
UseXmlFormat Boolean Use XML format
UseBinaryFormat Boolean Use compact binary format (default)

When configuring via code, prefer the generic registration call, as it avoids magic strings and potential runtime errors by using a type provided at compile time. Avoid the shortcut extension method since it does not let you configure the AdoInvariant and has a different default format from the other API methods.

"You don't learn to walk by following rules. You learn by doing, and by falling over." — Richard Branson