Category Archives: Software development

First Steps with RedisGraph

RedisGraph is a super-fast graph database, and like others of its kind (such as Neo4j), it is useful to represent networks of entities and their relationships. Examples include social networks, family trees, and organisation charts.

Getting Started

The easiest way to try RedisGraph is using Docker. Use the following command, which is based on what the Quickstart recommends but instead uses the edge tag, which would have the latest features and fixes:

sudo docker run -p 6379:6379 -it --rm redislabs/redisgraph:edge
Redis with RedisGraph running in Docker

You will also need the redis-cli tool to run the example queries. On Ubuntu (or similar), you can get this by installing the redis-tools package.

Tom Loves Judy

We’ll start by representing something really simple: Tom Loves Judy.

Tom Loves Judy.

We can create this graph using a single command:

GRAPH.QUERY TomLovesJudy "CREATE (tom:Person {name: 'Tom'})-[:loves]->(judy:Person {name: 'Judy'})"

When using redis-cli, queries will also follow the format of GRAPH.QUERY <key> "<cypher_query>". In RedisGraph, a graph is stored in a Redis key (in this case called “TomLovesJudy“) with the special type graphdata, thus this must always be specified in queries. The query itself is the part between double quotes, and uses a language called Cypher. Cypher is also used by Neo4j among other software, and RedisGraph implements a subset of it.

Cypher represents nodes and relationships using a sort of ASCII art. Nodes are represented by round brackets (parentheses), and relationships are represented by square brackets. The arrow indicates the direction of the relationship. RedisGraph at present does not support undirected relationships. When you run the above command, Redis should provide some output indicating what happened:

2 nodes and one relationship. Makes sense.

Since our graph has been created, we can start running queries against it. For this, we use the MATCH keyword:

GRAPH.QUERY TomLovesJudy "MATCH (x) RETURN x"

Since round brackets represent a node, here we’re saying that we want the query to match any node, which we’ll call x, and then return it. The output for this is quite verbose:

1) 1) "x"
2) 1) 1) 1) 1) "id"
            2) (integer) 0
         2) 1) "labels"
            2) 1) "Person"
         3) 1) "properties"
            2) 1) 1) "name"
                  2) "Tom"
   2) 1) 1) 1) "id"
            2) (integer) 1
         2) 1) "labels"
            2) 1) "Person"
         3) 1) "properties"
            2) 1) 1) "name"
                  2) "Judy"
3) 1) "Query internal execution time: 61.509847 milliseconds"

As you can see, this has given us the whole structure of each node. If we just want to get something specific, such as the name, then we can specify it in the RETURN clause:

GRAPH.QUERY TomLovesJudy "MATCH (x) RETURN x.name"
1) 1) "x.name"
2) 1) 1) "Tom"
   2) 1) "Judy"
3) 1) "Query internal execution time: 0.638126 milliseconds"

We can also query based on relationships. Let’s see who loves who:

GRAPH.QUERY TomLovesJudy "MATCH (x)-[:loves]->(y) RETURN x.name, y.name"
1) 1) "x.name"
   2) "y.name"
2) 1) 1) "Tom"
      2) "Judy"
3) 1) "Query internal execution time: 54.642536 milliseconds"

It seems like Tom Loves Judy. Unfortunately, Judy does not love Tom back.

Company Shareholding

Let’s take a look at a slightly more interesting example.

Company A is owned by individuals X (85%) and Y (15%). Company B is owned by individuals Y (55%) and Z (45%).

In this graph, we have companies (blue nodes) which are owned by multiple individuals (red nodes). We can’t create this as a single command as we did before. We also can’t simply issue a series of CREATE commands, because we may end up creating multiple nodes with the same name.

Instead, let’s create all the nodes separately first:

GRAPH.QUERY Companies "CREATE (:Individual {name: 'X'})"
GRAPH.QUERY Companies "CREATE (:Individual {name: 'Y'})"
GRAPH.QUERY Companies "CREATE (:Individual {name: 'Z'})"

GRAPH.QUERY Companies "CREATE (:Company {name: 'A'})"
GRAPH.QUERY Companies "CREATE (:Company {name: 'B'})"

You’ll notice here that the way we are defining nodes is a little different. A node follows the structure (alias:type {properties}). The alias is not much use in such CREATE commands, but on the other hand, the type now (unlike in the earlier example) gives us a way to distinguish between different kinds of nodes.

Now that we have the nodes, we can create the relationships:

GRAPH.QUERY Companies "MATCH (x:Individual { name : 'X' }), (c:Company { name : 'A' }) CREATE (x)-[:owns {percentage: 85}]->(c)"
GRAPH.QUERY Companies "MATCH (x:Individual { name : 'Y' }), (c:Company { name : 'A' }) CREATE (x)-[:owns {percentage: 15}]->(c)"
GRAPH.QUERY Companies "MATCH (x:Individual { name : 'Y' }), (c:Company { name : 'B' }) CREATE (x)-[:owns {percentage: 55}]->(c)"
GRAPH.QUERY Companies "MATCH (x:Individual { name : 'Z' }), (c:Company { name : 'B' }) CREATE (x)-[:owns {percentage: 45}]->(c)"

In order to make sure we apply the relationships to existing nodes (as opposed to creating new ones), we first find the nodes we want with a MATCH clause, and then CREATE the relationship between them. You’ll notice that our relationships now also have properties.

Now that our graph is set up, we can start querying it! Here are a few things we can do with it.

Return the names of all the nodes:

GRAPH.QUERY Companies "MATCH (x) RETURN x.name"
1) 1) "x.name"
2) 1) 1) "X"
   2) 1) "Y"
   3) 1) "Z"
   4) 1) "A"
   5) 1) "B"
3) 1) "Query internal execution time: 0.606600 milliseconds"

Return the names only of the companies:

GRAPH.QUERY Companies "MATCH (c:Company) RETURN c.name"
1) 1) "c.name"
2) 1) 1) "A"
   2) 1) "B"
3) 1) "Query internal execution time: 0.515959 milliseconds"

Return individual ownership in each company (separate fields):

GRAPH.QUERY Companies "MATCH (i)-[s]->(c) RETURN i.name, s.percentage, c.name"
1) 1) "i.name"
   2) "s.percentage"
   3) "c.name"
2) 1) 1) "X"
      2) (integer) 85
      3) "A"
   2) 1) "Y"
      2) (integer) 15
      3) "A"
   3) 1) "Y"
      2) (integer) 55
      3) "B"
   4) 1) "Z"
      2) (integer) 45
      3) "B"
3) 1) "Query internal execution time: 1.627741 milliseconds"

Return individual ownership in each company (concatenated strings):

GRAPH.QUERY Companies "MATCH (i)-[s]->(c) RETURN i.name + ' owns ' + round(s.percentage) + '% of ' + c.name"
1) 1) "i.name + ' owns ' + round(s.percentage) + '% of ' + c.name"
2) 1) 1) "X owns 85% of A"
   2) 1) "Y owns 15% of A"
   3) 1) "Y owns 55% of B"
   4) 1) "Z owns 45% of B"
3) 1) "Query internal execution time: 1.281184 milliseconds"

Find out who owns at least 50% of the shares in Company A:

GRAPH.QUERY Companies "MATCH (i)-[s]->(c) WHERE s.percentage >= 50 AND c.name = 'A' RETURN i.name"
1) 1) "i.name"
2) 1) 1) "X"
3) 1) "Query internal execution time: 1.321579 milliseconds"

Wrapping Up

In this article, we’ve seen how to:

  • get up and running with RedisGraph
  • create simple graphs
  • perform basic queries

We’ve obviously scratched the surface of RedisGraph and Cypher, but hopefully these examples will help others who, like me, are new to this space.

XAML Hot Reload

Having been away from WPF for a long time, it was a pleasant surprise for me to find this when building a small tool a few days ago:

The panel at the top says that Hot Reload is available.

XAML Hot Reload is a feature that causes changes in XAML to immediately be reflected in an application running in debug mode. It applies to WPF and UWP applications, and is currently in preview for Xamarin Forms apps.

Update 7th November 2019: thanks to the Twitter user who pointed out that this feature has been around for three years under the name “XAML Edit and Continue” for WPF and UWP apps. It recently got rebranded and extended to Xamarin Forms.

Basically, if I go and change the XAML for the above window (making it even uglier than it already is) while the application is still running, the changes are applied instantly as soon as I save the file:

Changes to styling from the XAML were instantly reflected in the window on save.

This kind of live-reload has existed in the web development space for a while thanks to technologies such as Browsersync. However, it is nice to see it finally arrive in the much-neglected realm of desktop application development, for those still stuck in it.

Encrypting Strings in C# using Authenticated Encryption

Encryption is fundamental and ubiquitous. Whether it’s to prevent sensitive settings (such as passwords and API tokens) from falling into the wrong hands, or making sure no one listens in on confidential communications, encryption is extremely important. Many people do not even realise that they use it every day.

Encrypting data using the .NET Framework or .NET Core libraries, however, is not trivial. There are different ways to encrypt and decrypt data, and sometimes this requires some knowledge about the underlying algorithm.

To keep things really simple, we’ll use a third party library that provides a simple interface for encryption and decryption. Because this library uses strings and byte arrays, it is not suitable for encryption of large amounts of data, such as huge files, which would bloat the application’s memory. However, it is perfectly fine for small strings.

Later in the article, I also share a simple tool that I built to help generate keys and test encryption and decryption. You can find this tool under the AuthenticatedEncryptionTester folder in the Gigi Labs BitBucket repository.

Using AuthenticatedEncryption

AuthenticatedEncryption is a library that provides simple methods for encryption and decryption:

“The library consists of a single static class. This makes it very easy to use. It uses Authenticated Encryption with Associated Data (AEAD), using the approach called “Encrypt then MAC” (EtM). It uses one key for the encryption part (cryptkey) and another key for the MAC part (authkey).”

All we need to start using this is to install the corresponding NuGet package, either using the Package Manager Console:

Install-Package AuthenticatedEncryption

…or using the .NET Core command line tools:

dotnet add package AuthenticatedEncryption

The project’s readme file (which is the first thing you see in the GitHub repo) explains how it’s used, and it is really simple. First, you generate two keys, called the cryptkey and authkey respectively:

var cryptKey = AuthenticatedEncryption.AuthenticatedEncryption.NewKey();
var authKey = AuthenticatedEncryption.AuthenticatedEncryption.NewKey();

This is something you will typically do once, since you have to encrypt and decrypt using the same pair of keys.

Next, we need something to encrypt. We can get this from user input:

Console.Write("Enter something to encrypt: ");
string plainText = Console.ReadLine();

We can now encrypt the plain text by using the keys we generated earlier:

string encrypted = AuthenticatedEncryption.AuthenticatedEncryption
    .Encrypt(plainText, cryptKey, authKey);
Console.WriteLine($"Encrypted: {encrypted}");

And we can also decrypt the cipher text using a similar mechanism:

string decrypted = AuthenticatedEncryption.AuthenticatedEncryption
    .Decrypt(encrypted, cryptKey, authKey);
Console.WriteLine($"Decrypted: {decrypted}");

You will by now have noted the double AuthenticatedEncryption that is constantly repeated throughout the code. This is a result of the unfortunate choice of the library author to use the same for the class and namespace. There is already an open issue for this.

Let’s run this code and see what happens:

Simple encryption and decryption using the AuthenticatedEncryption library. Running on Kubuntu 19.10 using .NET Core.

As you can see, the input string was encrypted and the result was encoded in base64. This was later decrypted to produce the original input string once again.

Authenticated Encryption Tester

To facilitate key generation as well as experimentation, I wrote this small tool:

Authenticated Encryption Tester. A simple tool to quickly use the functions of the AuthenticatedEncryption library.

This lets you use the AuthenticatedEncryption library functionality that we have just seen in the previous section. It’s useful to initially generate your keys, and also to test that you are actually able to encrypt and decrypt your secrets successfully.

It is a WPF application running on .NET Core 3, so unlike the AuthenticatedEncryption library, unfortunately it only works on Windows. However, for those of you who, like me, have the misfortune of already using Windows, it can turn out to be a handy utility.

You can get the code from the AuthenticatedEncryptionTester folder in the Gigi Labs BitBucket repository. While I won’t go through all the code in the interest of brevity, I’d like to go through some parts and show that it’s doing pretty much what we’ve seen in the previous section.

        private void GenerateCryptKeyButton_Click(object sender, RoutedEventArgs e)
            => GenerateKeyInTextBox(this.CryptKeyField);

        private void GenerateAuthKeyButton_Click(object sender, RoutedEventArgs e)
            => GenerateKeyInTextBox(this.AuthKeyField);

// ...

        private void GenerateKeyInTextBox(TextBox textBox)
        {
            string key = AuthenticatedEncryption
                .AuthenticatedEncryption.NewKeyBase64Encoded();
            textBox.Text = key;
        }

The first two fields in the window expect to have the two keys in base64 format. You can either use keys you had generated earlier and stored, or you can hit the Generate buttons to create new ones. These buttons create new keys using the NewKeyBase64Encoded() method, which is just like NewKey() except that it returns a base64-encoded string instead of a byte array. This is handy in situations where you want a string representation, such as in a GUI like this.

Encryption and decryption also work just like in the previous section, and the implementation merely adds some extra code for validation and I/O. This is the method that runs when you click the Encrypt button:

        private void EncryptButton_Click(object sender, RoutedEventArgs e)
        {
            const string operation = "Encrypt";

            string cryptKeyBase64 = this.CryptKeyField.Text;
            string authKeyBase64 = this.AuthKeyField.Text;
            string plainText = this.PlainTextField.Text;

            try
            {
                if (string.IsNullOrWhiteSpace(cryptKeyBase64)
                    || string.IsNullOrWhiteSpace(authKeyBase64)
                    || string.IsNullOrWhiteSpace(plainText))
                {
                    ShowWarning("Both keys and the plain text must have a value.",
                        operation);
                }
                else
                {
                    byte[] cryptKey = Convert.FromBase64String(cryptKeyBase64);
                    byte[] authKey = Convert.FromBase64String(authKeyBase64);

                    string cipherText = AuthenticatedEncryption
                        .AuthenticatedEncryption.Encrypt(plainText, cryptKey, authKey);
                    this.CipherTextField.Text = cipherText;
                }
            }
            catch (Exception ex)
            {
                ShowError(ex, operation);
            }
        }

The Encrypt button takes what’s in the Plain Text field and puts an encrypted version in the Cipher Text field. The Decrypt button does the opposite, taking the Cipher Text and putting the decrypted version in the Pain Text field. The code for the Decrypt button is very similar to that of the Encrypt button so I won’t include it here.

One thing you’ll note as you experiment with this is that the encrypted output string changes every time. This is an expected behaviour that provides better security. By clearing the value in the Plain Text field before hitting Decrypt, you can verify that it is always decrypted correctly to the original input string, even with different encrypted values.

Summary

The AuthenticatedEncryption library is great for encryption and decryption of simple strings. For large amounts of data, you should instead use streams together with the cryptographic APIs available in the .NET Framework or .NET Core.

You can use my Authenticated Encryption Tester to generate keys or experiment with encryption and decryption using the AuthenticatedEncryption library. It is built on WPF so it only works on Windows.

Using Time-Based One-Time Passwords for Two-Factor Authentication

Introduction

Two-factor authentication (2FA) is becoming more and more important, as its adoption is driven by a need for major software companies to secure their systems against threats, as well as due to legal requirements of strong customer authentication, such as the PSD2 directive that came in force in Europe last month.

2FA can be implemented in a number of ways. Typically, it is a combination of the usual username/password login as well as something else, often being a one-time password (OTP) that is sent via SMS or email, or generated by an algorithm.

In this article, we’ll focus entirely on generating and verifying Time-Based One-Time Passwords (TOTP) using Google Authenticator and the Otp.NET library.

Update 20th October 2019: This also works if you use Microsoft Authenticator instead of Google Authenticator. Microsoft Authenticator requires more permissions on your device, sends usage data to Microsoft by default, and is slightly more confusing because you have to choose the type of account.

Update 22nd October 2019: I discovered another mobile app called Authy, and it works just as well to acquire the TOTP secret and generate codes. It is interesting because it has a mechanism to take encrypted backups in the cloud and synchronise across devices, addressing the problem of when you lose or change your phone.

About TOTP

TOTP is an algorithm used to generate one-time passwords based on a shared secret and the current time. It is defined in RFC6238, and is a variant of the HOTP algorithm (RFC4226) which uses a counter instead of time.

The client and server use the same algorithm, the same shared secret and (roughly) the same time to generate the same code.

TOTP can be thought of as a function that takes the shared secret and current time as inputs, and generates a one-time password as output. Given that the client and server both know the same shared secret, and that their software clocks are more or less in sync without major clock skew, then they would generate the same code. This allows a code generated on a mobile device to be verified on the server side.

Generating a Shared Secret

We will use Otp.NET to perform most operations related to TOTP generation and verification. This can easily be intalled in a .NET (Core) console application via NuGet:

Install-Package Otp.NET

It is then really easy to generate and output a shared secret using the following code:

var secret = KeyGeneration.GenerateRandomKey(20);
var base32Secret = Base32Encoding.ToString(secret);
Console.WriteLine(base32Secret);

The secret that we generated on the first line is an array of bytes. However, we output it in base32 encoding. This is important for the next step when we will pass the secret to the mobile device. As I learned the hard way, it does not work if the secret is an arbitrary string and not base32-encoded.

Running the above, I just got the following in the output:

6L4OH6DDC4PLNQBA5422GM67KXRDIQQP

Generating a QR Code for the Secret

Stefan Sundin made this great 2FA QR code generator. The two required fields are the Secret (where we paste the value generated above) and a Label (which is arbitrary and identifies the application — we’ll simply put “MFA Test 1” in there).

The QR code helps to synchronise the secret between the server and the mobile device.

Setting up Google Authenticator

Find Google Authenticator in your phone’s app store and install it. It requires access to your camera as we’ll see in a second.

Get Google Authenticator from your phone’s app store.

After installation and its brief in-built tutorial, you get to the point where you can set up your first TOTP code generator (they call it an “account”):

To synchronise a shared secret onto your mobile device, you can scan a barcode or type in the secret directly.

This step is where you enter the shared secret into Google Authenticator. You can do that by scanning a QR code (first option), or by typing it in (second option). The latter is slow and painful, especially on a mobile device, and should be kept as a fallback in case there is some kind of problem scanning the QR code. Scanning the QR code is really just a convenience mechanism and is an encoded version of the same secret.

Scan the barcode to get the shared secret into Google Authenticator.

Once you’ve scanned the QR code, Google Authenticator has acquired the shared secret and starts generating TOTP codes every 30 seconds:

Google Authenticator is generating TOTP codes.

Since you can have more than one of these code generators in here (for different applications), they come with a label. In this case, you’ll notice that we have “MFA Test 1”, which is exactly what we entered in the Label field when generating the QR code.

Generating TOTP codes from Otp.NET

If you need to generate TOTP codes from .NET code (essentially to do what Google Authenticator is doing), then Otp.NET makes it very easy to do that:

            string base32Secret = "6L4OH6DDC4PLNQBA5422GM67KXRDIQQP";
            var secret = Base32Encoding.ToBytes(base32Secret);

            var totp = new Totp(secret);
            var code = totp.ComputeTotp();

            Console.WriteLine(code);

The ComputeTotp() method takes an optional DateTime parameter as the current time to use for the code generation algorithm. If not provided, it uses DateTime.UtcNow, which is typically what you want to use.

The TOTP code generated from the C# program (top right) is identical to the one generated from Google Authenticator on my phone (bottom centre).

Since we are using Google Authenticator, we don’t actually need this at all, so this is just something to keep in mind if you ever actually need it. It also gives some assurance that we’re on the right track, because what we’re doing in C# and on the mobile device are evidently well in sync.

Verifying TOTP Codes

Like every other operation we’ve seen, verifying TOTP codes with Otp.NET is also very easy. The following code shows how to do this, although most of the code is actually handling input and output.

            string base32Secret = "6L4OH6DDC4PLNQBA5422GM67KXRDIQQP";
            var secret = Base32Encoding.ToBytes(base32Secret);

            var totp = new Totp(secret);

            while (true)
            {
                Console.Write("Enter code: ");
                string inputCode = Console.ReadLine();
                bool valid = totp.VerifyTotp(inputCode, out long timeStepMatched,
                    VerificationWindow.RfcSpecifiedNetworkDelay);

                string validStr = valid ? "Valid" : "Invalid";
                var colour = valid ? ConsoleColor.Green : ConsoleColor.Red;
                Console.ForegroundColor = colour;
                Console.WriteLine(validStr);
                Console.ResetColor();
            }

Here’s what it might look like while you test it out repeatedly:

A number of tests show interesting results.

As you can see above, I did a number of things:

  1. I entered two invalid codes, and got invalid responses.
  2. I entered a valid code, and got a valid response as expected.
  3. I waited for a new code to be generated, then entered the same code as before, and it was accepted.
  4. I entered the new code that was generated, and it was validated.
  5. I entered another invalid code, and it was marked as such.

The most interesting part of the above is the third step, and it requires further explanation. Codes are generated in time windows, by default every 30 seconds. That doesn’t necessarily mean that the previous code should be rejected. The time window might have shifted just as the user was typing the code, or there could be network delays, etc. Typically, some leeway is allowed when validating these codes. The RFC recommends allowing codes from one time window in the past or future, and that’s what the value of VerificationWindow.RfcSpecifiedNetworkDelay that we passed in as the third parameter to VerifyTotp() does. If you want, you can pass in something different that is more lenient or more restrictive.

On the other hand, accepting the same code twice is wrong, considering we are supposed to be generating one time passwords. In order to make sure that a code isn’t used twice, we need to store something that we can later check to know whether a code has been used. That’s the reason for the second parameter to VerifyTotp(). It gives us back a number indicating the time step used, so we can save this whenever a code is used, and later check whether the same time step has already been used before.

Assuming a single shared secret, a very quick-and-dirty dummy implementation using a HashSet instead of real persistence could look something like this:

            string base32Secret = "6L4OH6DDC4PLNQBA5422GM67KXRDIQQP";
            var secret = Base32Encoding.ToBytes(base32Secret);

            var totp = new Totp(secret);

            var usedTimeSteps = new HashSet<long>();

            while (true)
            {
                Console.Write("Enter code: ");
                string inputCode = Console.ReadLine();
                bool valid = totp.VerifyTotp(inputCode, out long timeStepMatched,
                    VerificationWindow.RfcSpecifiedNetworkDelay);

                valid &amp;= !usedTimeSteps.Contains(timeStepMatched);
                usedTimeSteps.Add(timeStepMatched);

                string validStr = valid ? "Valid" : "Invalid";
                var colour = valid ? ConsoleColor.Green : ConsoleColor.Red;
                Console.ForegroundColor = colour;
                Console.WriteLine(validStr);
                Console.ResetColor();
            }

Sorry about that &amp; in there – the operator is supposed to be &=. There’s a glitch in the editor I’m using and I hope it’ll be sorted out at some point.

Like this, there’s no way you can ever have the same code be valid twice:

The same code, even within the same time window, is invalid the second time.

Conclusion

In this article we’ve seen how Time-Based One-Time Passwords can be generated and verified. We’ve focused mainly on:

  1. Generating a shared secret using Otp.NET
  2. Bringing it to a mobile device with Google Authenticator
  3. Using Google Authenticator to generate TOTP codes
  4. Using Otp.NET to validate these codes

In a two-factor authentication implementation, this is of course only one of the factors, and usually takes place after a regular username/password login.

Retrieving Stock Prices using AWS Lambda

AWS Lambda functions are great for simple logic running periodically (among other things). In this article, we’ll create a simple AWS Lambda function in Python that retrieves stock prices from a REST API every minute. Let’s get straight to it!

Create a Lambda Function

First, we need to create a function. Follow the instructions illustrated below to do this.

From the AWS Console dashboard, locate the Lambda service. You can also do this via the Services drop-down panel at the top, or from your recently visited services (if you’ve already been using Lambda).
Once you are in the Lambda service, create a new function by clicking on the “Create a function” button as shown above.
Choose a name for the Lambda function, and also the runtime. We’re using Python 3.7 (which is the latest supported Python version in AWS Lambda at the time of writing this) for this example, but other options are available (e.g. Node.js, NET Core, Go, etc). Leave everything else as is and hit the Create function button.
Once the Lambda is created, you are taken to the new function itself. A green status message at the top indicates that it has been created successfully.

Editing the Lambda’s Code

The function’s configuration screen can seem quite confusing at first, but all you need to do is scroll down to get to the code editor. While there are a few different ways to add code to your Lambda, using the provided editor (which is the default option for Python) is the easiest.

Replace the default code in the editor with the following, and hit the Save button at the top-right.

from urllib.request import urlopen
from contextlib import closing
import json

def lambda_handler(event, context):
    with closing(urlopen("https://financialmodelingprep.com/api/v3/stock/real-time-price/GOOGL")) as responseData:
        jsonData = responseData.read()
        deserialisedData = json.loads(jsonData)
        price = deserialisedData['price']
        print(price);
    return price

Here we are simply retrieving Google’s stock price using the Financial Modeling Prep Stock Realtime Price API, which is open and doesn’t require any authentication.

Next to the Save button at the top-right, there’s a Test button. Click it, and the following screen comes up.

The Configure test event screen. Just enter a name and hit Create.

Just enter a name (e.g. “Test”) and hit the Create button further below. We’re not using the input JSON data, so you can just ignore it.

Next, click the Test button at the top-right again, and your Lambda function will be executed:

After clicking Test again, the lambda is executed and the results are shown below the code.

The results are shown below the code, and these include various metadata (such as a Request ID and execution time) as well as Google’s stock price of 1082.38, which we retrieved from the REST API and logged using the print statement in the code.

Running Periodically

We now have a working Lambda function, but so far we have to invoke it manually every time. Let’s set it up so that it runs every minute.

At the top of the screen, click CloudWatch Events on the left to add a CloudWatch trigger.

Scroll back to the top, and you’ll see a placeholder telling you to “Add triggers from the list on the left“. Following that advice, click on “CloudWatch Events” to the left.

A CloudWatch Event trigger is added to the function.

This has the effect of adding “CloudWatch Events” as a trigger in the slot where the placeholder text was, but what you might not notice at first is that the lower part of the page changes from the code editor to a “Configure triggers panel“. This can be quite confusing for those new to AWS Lambda who might not intuit right away that clicking on the boxes will affect the content in some other part of the page.

By scrolling down, we can configure the new trigger.

Here we use a Schedule expression of rate(1 minute) to make the function run every minute.

Filling in most of the settings (e.g. choosing a name) is easy, bearing in mind that there are some restrictions (e.g. some characters, such as spaces, are restricted in the name).

The only tricky part is where we specify how frequently we want the function to be executed. For this, we can use cron or rate expressions (refer to AWS documentation: Schedule Expressions Using Rate or Cron). By using an expression of rate(1 minute), we configure the function to run every minute, which is the smallest supported interval.

Once this is all set up, click the Add button to set up the trigger. Then, don’t forget to click the Save button at the top-right of the page to apply the changes to the Lambda function.

Checking Output in CloudWatch

After waiting a few minutes for the function to run a few times, we can go into CloudWatch and check the output of each execution.

CloudWatch logs.

From the AWS Services, locate CloudWatch. Go into Logs from the left menu, and locate the log group for our Lambda function (in this case it’s /aws/lambda/StockChecker).

Select the most recent log stream (the one at the top), and if you scroll to the end, you should see logs showing the function’s execution every minute, as well as whatever we’re writing to standard output (in this case, Google’s stock price).

CloudWatch logs show that the Lambda function is executing every minute.

We can see that the function is executing every minute, and we’re logging a stock price each time. The US stock market is closed right now, and that’s why the stock price is always the same (you’d expect it to change frequently when the market is active).

Conclusion

At this point, we have a simple, working AWS Lambda function (written in Python) that runs every minute and retrieves Google’s stock price. To keep things simple, we’re just writing it to standard output, which means we can see the value in CloudWatch – but we could also expand the code to build something useful from this.

Getting Started with Angular 8

Angular is an open-source framework built and maintained by Google, which is mainly used to develop Single-Page Applications (SPAs). It provides a structured approach towards creating front-end web applications.

Originally known as AngularJS, the framework underwent a complete rewrite that resulted in Angular 2.0 (dropping the -JS suffix from the name). The versions that came after 2.0 (with Angular 8 being the latest, released just over two weeks ago) are incremental upgrades, thus it is possible to upgrade between them. However, AngularJS is a different beast and there is no easy way to upgrade from AngularJS from Angular 2.0+.

In this article, we’re going to go through the steps necessary to start working with Angular. In order to keep this concise, there won’t be a lot of background.

npm

On Windows, download the Node.js installer from their website. Either version should be fine to get started.

The first thing we need to do is get npm, a package manager for JavaScript libraries. On Windows, download and install Node.js. On Linux or Mac, use the relevant package manager for your system (e.g. apt-get on Linux Ubuntu), possibly along with the sudo command for elevated privileges, to install npm.

Angular CLI

The Angular CLI homepage shows the commands you need to set up and use the Angular CLI tools.

Next, we need the Angular CLI to help us with our development workflow. Use npm to install it as a global tool, as follows (prefix this with sudo if using Linux or Mac):

npm install -g @angular/cli
Using npm to install the Angular CLI.

ng is the command-line tool we just installed. Use ng --version to make sure it’s in working order:

After executing ng --version, we can see some “Angular CLI” ASCII art and other information. This means that it’s working fine.

Creating a Project

Use ng new to create an Angular app from a template. You’ll be asked some questions to determine what features you need, but for now just press ENTER at each question to use the defaults.

ng new myproject
ng new myproject creates a folder called myproject with the Angular files in it. Press ENTER when asked questions to use defaults for now.

Note: when I first ran this, I got an error along the lines of “EPERM: operation not permitted, unlink“, even when using an elevated command prompt. The problem was likely caused by an old version of npm I had on my machine before, and I fixed it by running npm cache clean --force.

Running the application

Go into the project directory you’ve just created (e.g. myproject), and use ng serve to run the web application you just generated:

cd myproject
ng serve
ng serve runs a web server that you can use to access the running web application. Look in the output for the endpoint to use in your browser.

When ng serve is done building the project, it runs a web server hosting the web application. The output tells you where to access it, in this case http://localhost:4200/. Put that in your browser’s address bar, and you should see the homepage from the project template that we set up earlier:

A simple page shows us that Angular is in fact working.

Data Binding Illustration

We’ve created and run a web application using Angular, so we’re done in terms of getting started. However, let’s make a small change to the web application to get a little more comfortable with it and see something working.

Visual Studio Code is a popular choice for frontend development, despite having been made by Microsoft.

With ng serve still running, locate the src/app directory under your project’s root directory. Using a text editor or IDE of your choice, add the lines highlighted below to app.component.html:

<!--The content below is only a placeholder and can be replaced.-->
<div style="text-align:center">
  <h1>
    Welcome to {{ title }}!
  </h1>
  <img width="300" alt="Angular Logo" src="data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZpZXdCb3g9IjAgMCAyNTAgMjUwIj4KICAgIDxwYXRoIGZpbGw9IiNERDAwMzEiIGQ9Ik0xMjUgMzBMMzEuOSA2My4ybDE0LjIgMTIzLjFMMTI1IDIzMGw3OC45LTQzLjcgMTQuMi0xMjMuMXoiIC8+CiAgICA8cGF0aCBmaWxsPSIjQzMwMDJGIiBkPSJNMTI1IDMwdjIyLjItLjFWMjMwbDc4LjktNDMuNyAxNC4yLTEyMy4xTDEyNSAzMHoiIC8+CiAgICA8cGF0aCAgZmlsbD0iI0ZGRkZGRiIgZD0iTTEyNSA1Mi4xTDY2LjggMTgyLjZoMjEuN2wxMS43LTI5LjJoNDkuNGwxMS43IDI5LjJIMTgzTDEyNSA1Mi4xem0xNyA4My4zaC0zNGwxNy00MC45IDE3IDQwLjl6IiAvPgogIDwvc3ZnPg==">
  <input type="text" [(ngModel)]="name" />
  <br />{{ name }}
</div>
<h2>Here are some links to help you start: </h2>
<ul>
  <li>
    <h2><a target="_blank" rel="noopener" href="https://angular.io/tutorial">Tour of Heroes</a></h2>
  </li>
  <li>
    <h2><a target="_blank" rel="noopener" href="https://angular.io/cli">CLI Documentation</a></h2>
  </li>
  <li>
    <h2><a target="_blank" rel="noopener" href="https://blog.angular.io/">Angular blog</a></h2>
  </li>
</ul>

Then, add the lines highlighted below to app.module.ts:

import { BrowserModule } from '@angular/platform-browser';
import { NgModule } from '@angular/core';
import { FormsModule } from '@angular/forms';

import { AppComponent } from './app.component';

@NgModule({
  declarations: [
    AppComponent
  ],
  imports: [
    FormsModule,
    BrowserModule
  ],
  providers: [],
  bootstrap: [AppComponent]
})
export class AppModule { }

When ng serve detects these changes, it should reload the web application (in your browser) automatically, so you don’t need to stop and start it again whenever you change something.

We’ve added the text input box under the image. When you type in it, the text below it is automatically updated accordingly.

Thanks to the changes we made, we now have a text input box under the Angular logo. When you type in it, the text below it is synchronised with it.

The changes we made might seem alien at first, but we’ve actually used two important features of Angular: data binding and string interpolation. While explaining these is beyond the scope of this introductory article, I hope that seeing this power at work — with such a small change — has given a taste of why Angular is so useful.

Woodchuck Translation with Amazon Translate

This article is an attempt to have fun with Amazon Translate, and is not intended to be taken as any sort of serious review.

Amazon Web Services (AWS) includes a machine translation service called Amazon Translate:

“Amazon Translate is a neural machine translation service that delivers fast, high-quality, and affordable language translation. Neural machine translation is a form of language translation automation that uses deep learning models to deliver more accurate and more natural sounding translation than traditional statistical and rule-based translation algorithms. Amazon Translate allows you to localize content – such as websites and applications – for international users, and to easily translate large volumes of text efficiently.” — Amazon Translate homepage

Sounds pretty cool. If you log into the AWS Console and select the Amazon Translate service, it gives you an interface where you can easily play with it:

If we hit the “Swap languages” button, this translates the result back to the original language. In this really simple case, it matches perfectly:

However, this is not always the case. Translating back and forth will often result in things being “lost in translation”, for various reasons ranging from context to linguistic differences.

Since we all know machine translation isn’t perfect, I could not resist being a complete bastard and testing AWS Translate against none other than… How much wood would a woodchuck chuck:

Translating back and forth for each of the supported languages yields some interesting and often hilarious results before the translation converges (translating back and forth starts giving you the same thing every time). Let’s take a look at the results of this little experiment.

English <-> German

  1. How much wood would a woodchuck chuck if a woodchuck could chuck wood?
  2. Wie viel Holz würde ein Holzfutter, wenn ein Holzfutter Holz abwerfen könnte?
  3. How much wood would a wood fodder if a wood fodder could drop wood?
  4. Wie viel Holz würde ein Holzfutter, wenn ein Holzfutter Holz fallen könnte?
  5. How much wood would a wood lining if a wood lining could fall wood?
  6. Wie viel Holz würde eine Holzverkleidung, wenn eine Holzverkleidung Holz fallen könnte?
  7. How much wood would a wood panelling if a wood paneling could fall wood?

It’s easy to see how, as we translate back and forth, we not only lose the sense of the original question (if it ever had any), but the result also makes no grammatical sense. However, what really catches my eye is the two different spellings of “panelling”/”paneling” in the same phrase.

English <-> French

  1. How much wood would a woodchuck chuck if a woodchuck could chuck wood?
  2. Combien de bois un mandrin de bois serait-il un mandrin de bois si un mandrin de bois pouvait mander du bois ?
  3. How much wood would a wood mandrel be a wood mandrel if a wood mandrel could mandrel wood?
  4. Combien de bois un mandrin de bois serait-il un mandrin de bois si un mandrin de bois pouvait mandrin de bois ?
  5. How much wood would a wood chuck be a wood chuck if a wood chuck could wood chuck?

What’s really interesting about this one is that after going on a nice big detour (becoming a “wood mandrel”), the English phrase eventually returns to something relatively close to the original (even if still wrong).

English <-> Portuguese

  1. How much wood would a woodchuck chuck if a woodchuck could chuck wood?
  2. Quanta madeira uma marmota manteria se uma marmota pudesse atirar madeira?
  3. How much wood would a groundhog keep if a groundhog could throw wood?

This one’s pretty good – it got the animal right. The only mistake is on the first translation of “chuck” becoming “keep” – you’ll notice that all other translations also interpret the different “chuck”s differently.

English <-> Spanish

  1. How much wood would a woodchuck chuck if a woodchuck could chuck wood?
  2. ¿Cuánta madera haría un Woodchuck si un Woodchuck pudiera tirar madera?
  3. How much wood would a Woodchuck make if a Woodchuck could throw wood?
  4. ¿Cuánta madera ganaría un Woodchuck si un Woodchuck pudiera tirar madera?
  5. How much wood would a Woodchuck gain if a Woodchuck could throw wood?

The Spanish Mr. Woodchuck is a businessman, apparently.

English <-> Arabic

  1. How much wood would a woodchuck chuck if a woodchuck could chuck wood?
  2. كم من الخشب يمكن أن تشوك الخشب إذا كان الخشب يمكن أن يرمي الخشب؟
  3. How much wood can a wood fork if the wood can throw wood?
  4. كم الخشب يمكن شوكة الخشب إذا الخشب يمكن رمي الخشب؟
  5. How much wood can a wood fork if wood can throw wood?

Okay, this is starting to get pretty weird.

English <-> Chinese

  1. How much wood would a woodchuck chuck if a woodchuck could chuck wood?
  2. 如果木头可以木头卡住木头,木头卡住多少木头?
  3. If wood can be stuck in wood, how many wood is stuck?
  4. 如果木头可以被卡在木头上,有多少木头被卡住?
  5. If wood can be stuck on wood, how many wood is stuck?
  6. 如果木头可以卡在木头上,有多少木头被卡住?
  7. If the wood can be stuck on the wood, how many wood is stuck?

Uhhh…

English <-> Japanese

  1. How much wood would a woodchuck chuck if a woodchuck could chuck wood?
  2. ウッドチャックが木材をチャックできるなら、どのくらいの木材をチャックしますか?
  3. If a Woodchuck can chuck wood, how much wood would you chuck?
  4. もしウッドチャックが木材をチャックできるなら、どのくらいの木材をチャックしますか?
  5. If the wood chuck can chuck wood, how much wood would you chuck?
  6. 木材チャックが木材をチャックできるなら、どれくらいの木材をチャックしますか?
  7. If a wood chuck can chuck wood, how much wood do you chuck?
  8. 木材チャックが木材をチャックできる場合、どれくらいの木材をチャックしますか?
  9. If a wood chuck can chuck wood, how much wood do you chuck?

I don’t know… how much wood would you chuck?

English <-> Russian

  1. How much wood would a woodchuck chuck if a woodchuck could chuck wood?
  2. Сколько дров будет завалить, если дрочак сможет заправлять древесину?
  3. How much wood will it fill if the wok can fill the wood?
  4. Сколько древесины она заполнит, если вок сможет заполнить дерево?
  5. How much wood will it fill if the wok can fill the tree?
  6. Сколько дерева она заполнит, если вок сможет заполнить дерево?
  7. How much tree would she fill if the wok could fill the tree?

Image credit: source

Sounds like one of Google’s interview questions.

English <-> Traditional Chinese

  1. How much wood would a woodchuck chuck if a woodchuck could chuck wood?
  2. 如果一個木頭能扔木頭的話一個木頭會扔多少木頭
  3. How much wood would a wood throw if one could throw wood?
  4. 如果能扔木頭的話木頭會扔多少木頭
  5. How much wood would it throw out if we could throw it?
  6. 如果我們能扔掉的話會丟多少木頭
  7. How much wood would we lose if we could throw it away?
  8. 如果我們能把它扔掉我們會損失多少木頭
  9. How much wood would we lose if we could get rid of it?
  10. 如果能除掉它我們會損失多少木頭
  11. How much wood would we lose if we could get rid of it?

I think we’ve found the Spanish woodchuck businessman’s rival.

English <-> Italian

  1. How much wood would a woodchuck chuck if a woodchuck could chuck wood?
  2. Quanta legna manderebbe una marmotta se una marmotta potesse buttare legna?
  3. How much wood would a groundhog send if a groundhog could throw out wood?
  4. Quanto legna manderebbe una marmotta se una marmotta potesse buttare la legna?
  5. How much wood would a groundhog send if a groundhog could throw out the wood?
  6. Quanto legna manderebbe una marmotta se una marmotta potesse buttare via la legna?
  7. How much wood would a groundhog send if a groundhog could throw away the wood?

This one is interesting as there are a lot of very subtle changes before convergence.

English <-> Turkish

  1. How much wood would a woodchuck chuck if a woodchuck could chuck wood?
  2. Eğer bir dağ sıçanı odunları çöpe atabilseydi, bir dağ sıçanı ne kadar ağaç gönderirdi?
  3. If a groundhog could throw wood away, how many trees would a groundhog send?
  4. Eğer bir dağ sıçanı tahta atabilseydi, bir dağ sıçanı kaç ağaç gönderirdi?
  5. If a groundhog could throw a throne, how many trees would a groundhog send?
  6. Eğer bir dağ sıçanı tahtı atabilseydi, bir dağ sıçanı kaç ağaç gönderirdi?
  7. If a groundhog could throw the throne, how many trees would a groundhog send?

It is really bizarre to see how “wood” transitions into “throne” and “trees” in two different parts of the same question.

English <-> Czech

  1. How much wood would a woodchuck chuck if a woodchuck could chuck wood?
  2. Kolik dřeva by dřevorubec sklízl, kdyby dřevorubec mohl sklíčit dřevo?
  3. How much wood would a lumberjack harvest if a lumberjack could deceive the wood?
  4. Kolik dřeva by dřevorubec sklízel, kdyby dřevorubec mohl klamat dřevo?
  5. How much wood would a lumberjack harvest if a lumberjack could deceive wood?

Image credit: source

Conclusion

I had fun playing around with Amazon Translate and seeing how the woodchuck tongue-twister degenerates when translated across different languages. I hope it was just as much fun for you to read this.

Please do not make any judgements about the accuracy of Amazon Translate based on this, for the following reasons:

  1. This is a very specific case and certainly doesn’t speak for the accuracy across entire languages.
  2. Translation isn’t easy. We’ve all heard of situations where things got “lost in translation”. Translation depends very much on context and linguistic differences. Hopefully the varying performance across languages is an illustration of this.
  3. Machine translation isn’t easy either. There’s a reason why it’s considered a field of artificial intelligence.

Microsoft Orleans 2.0.4 Released

Those using (or learning about) Microsoft Orleans, especially the newer 2.0.x releases that target .NET Standard and are cross-platform, might be interested to know that version 2.0.4 has just been released.

This release includes a couple of important bugfixes:

  • A number of Orleans users observed grain calls getting really slow after the silo has been running for around 12 hours. The long issue discussion reveals a lot of collective findings and ultimately provides the means to reproduce the problem. The root cause was traced to a bug in BlockingCollection<T> in .NET Core, which can lead to memory leaks and even lost items (Orleans messages in this case). A workaround has been implemented to sort this out.
  • Another issue prevented Orleans build-time code generation from being built when targeting .NET Core 2.1. This has also been fixed.

If you’re using Orleans 2.0.x, it’s therefore a good idea to upgrade to 2.0.4, especially if you are running Orleans in production.

Microsoft Orleans Use Case: Reservations System

Microsoft Orleans is an implementation of the actor model, and many people have leveraged it to build highly scalable distributed systems while completely avoiding the pain of multithreaded programming.

The actor model is still not a very mainstream thing, and people who come across it are often confused about what it is and why it is useful.

In order to address this, .NET contractor Jakub Konecki (Twitter | GitHub) has kindly agreed to share with us how he has been using Microsoft Orleans in his own particular use case. You can also learn more about his project from the Orleans Virtual Meetups in which he presented (Meetup #1: Event Sourced Grains, and  Meetup #12: Deploying Orleans).

DD: What is the problem you’re addressing with Microsoft Orleans?

JK: Currently I’m working for a company in the hospitality domain that manages bookings for a number of luxury resorts in the Caribbean.

I’m responsible for designing and delivering a greenfield system for a multi-tenant system for managing reservations.

The main features of the system are:

  • ability to register resort accommodation,
  • ability to manage pricing: rate plans, special offers,
  • integration with third-party marketplace used by tour operators
  • integration with third-party systems for flight searching and ticket purchasing
  • integration with property management systems used by resorts
  • a bespoke website that resort customers can use to search for and make their reservations.

The non-functional requirements include elastic scaling to allow for easy onboarding of new tenants and allow flexibility for existing tenants – for example the traffic may change drastically when special offers are introduced.

DD: How did Microsoft Orleans help you develop a solution?

Image taken from Orleans Virtual Meetup #12 presentation and used with permission.

The system is designed using DDD principles and benefits from event sourcing and event-driven architecture.

An actor framework is a good fit for this kind of system – mapping between actors and aggregate roots is natural, and implementation of event sourcing is quite straightforward and encapsulated by actors.

The Microsoft Orleans framework was selected as it was the most advanced actor framework implementation at the time that used technologies familier to the team (C#, Azure), was battle-tested, and was implicitly backed by Microsoft. Open-sourcing Orleans (and an active community that emerged shortly after) was another argument for using it.

DD: What benefits did Microsoft Orleans provide, and what challenges did you face?

JK: The most important benefits for using Orleans are scalability and programming model. Scaling an Orleans solution is as easy as moving a slider in the Azure portal. The ability to specify auto-scaling triggers in Azure means that changes in load can be handled with ease. We haven’t run into any problems related to scaling – there is no difference between running a cluster in Azure and a single node on local machine during development.

This brings me nicely to Orleans’ programming model, which makes development of distributed systems straightforward. Orleans handles a lot of complexity allowing developers to concentrate on business logic within essentially single-threaded grains. On the other hand Orleans doesn’t go to the other extreme and pretend the issues inherently related to distributed systems do not exist. That balance allows for rapid development – we’ve seen senior developers being able to pick up Orleans fundamentals and be productive within a day or two.

AWS Lambda .NET Core 2.1 Support Released

Amazon Web Services (AWS) has just announced that its serverless function offering, AWS Lambda, now supports the .NET Core 2.1 runtime, which was released towards the end of May 2018.

Quoting the official announcement:

“Today we released support for the new .NET Core 2.1.0 runtime in AWS Lambda. You can now take advantage of this version’s more performant HTTP client. This is particularly important when integrating with other AWS services from your AWS Lambda function. You can also start using highly anticipated new language features such as Span<T> and Memory<T>.

“We encourage you to update your .NET Core 2.0 AWS Lambda functions to use .NET Core 2.1 as soon as possible. Microsoft is expected to provide long-term support (LTS) for .NET Core 2.1 starting later this summer, and will continue that support for three years. Microsoft will end its support for .NET Core 2.0 at the beginning of October, 2018[2]. At that time, .NET Core 2.0 AWS Lambda functions will be subject to deprecation per the AWS Lambda Runtime Support Policy. After three months, you will no longer be able to create AWS Lambda functions using .NET Core 2.0, although you will be able to update existing functions. After six months, update functionality will also be disabled.

“[1] See Microsoft Support for .NET Core for the latest details on Microsoft’s .NET Core support.
“[2] See this blog post from Microsoft about .NET Core 2.0’s end of life.”

The choice here seems obvious: upgrade and get faster HttpClient, new language features, and long-term support; or lose support for your functions targeting .NET Core 2.0 (whatever that actually means).

In order to migrate to .NET Core 2.1, you’ll need the latest tooling – either version 1.14.4.0 of the AWS Toolkit for Visual Studio, or version 2.2.0 of the Amazon.Lambda.Tools NuGet package.

Check out the official announcement at the AWS blog for more information, including additional tips on upgrading.