Tag Archives: Python

Software

Migrating Cartography to Memgraph

Leave a comment

I’ve recently written about how to use Cartography to collect infrastructural data from both AWS and Okta into a Neo4j graph database for security analysis.

Neo4j is a long-standing player in the graph database market, with a robust product, great documentation, and a massive following. However, its long legacy is in a way also a disadvantage, as it can be costly, slow, and resource-hungry (due in no small part to its reliance on the JVM). Sometimes people would like to use an alternative for any of these reasons.

Memgraph, on the other hand, is a relatively young graph database, and certainly not as fully-featured as Neo4j. A key difference is that it is written in C++, meaning it’s designed to be faster and more lightweight than Neo4j (whether it lives up to this is something you’ll need to evaluate for your own use cases). Memgraph also made a very wise decision to support the Bolt protocol and the Cypher language – both of which Neo4j uses – meaning that it’s compatible with existing Neo4j clients and queries. Although there are variations in Cypher dialect, the incompatibilities are few, and moving from Neo4j to Memgraph is significantly less painful than, say, transitioning to a graph database that uses Gremlin as its query language.

At the time of writing this article, Cartography requires Neo4j 4.x, and does not work with Memgraph. However, I’m going to show you how to make at least part of it (the Okta intel module) work with minor alterations to the Cartography codebase. This serves as a demonstration of how to get started migrating an existing application from Neo4j to Memgraph.

Running Memgraph

Before we start looking at Cartography, let’s run an instance of Memgraph. To do this, we’ll take a tip from my earlier article, “Using the Neo4j Bolt Driver for Python with Memgraph“, and run it under Docker as follows (drop the sudo if you’re on Mac or Windows):

sudo docker run --rm -it -p 7687:7687 -p 3000:3000 -e MEMGRAPH="--bolt-server-name-for-init=Neo4j/" memgraph/memgraph-platform

That --bolt-server-name-for-init=Neo4j/ is a first critical step in Neo4j compatibility. As explained in that same article, the Neo4j Bolt Driver (i.e. client) for Python (which Cartography uses) checks whether the server sends an “agent” value that starts with “Neo4j/”. By setting this, Memgraph is effectively posing as a Neo4j server, and the Neo4j Bolt Driver for Python can’t tell the difference.

Update 19th September 2023: as of Memgraph v2.11, --bolt-server-name-for-init has a default value compatible with the Neo4j Bolt Driver, and therefore no longer needs to be provided.

If it’s successful, you should see output such as the following:

Memgraph is running. You can also execute queries directly from here.

Cloning the Cartography Repo

The next thing to do is grab a copy of the Cartography source code from the Cartography GitHub repo:

git clone https://github.com/lyft/cartography.git

Next, run the following command to install the necessary dependencies:

pip3 install -e .

Note: in the past, I’ve usually had to upgrade the Neo4j Bolt Driver for Python to 5.2.1 to get anything working, but as I try this again, it seems to work even with the default 4.4.x that Cartography uses. If you have problems, try changing setup.py to require neo4j>=5.2.1 and run the above command again.

Creating a Launch Configuration in Visual Studio Code

In order to run Cartography from its source code, you could run it directly from the terminal, for instance:

cd cartography/cartography
python3 __main__.py

However, as I’ve recently been using Visual Studio Code for all my polyglot software development needs, I find it much more convenient to set up a launch configuration that allows me to easily debug Cartography and pass whatever command-line arguments and environment variables I want.

The following launch.json is handy to run Cartography with an Okta configuration as described in “Getting Started with Cartography for Okta“:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Run Cartography",
            "type": "python",
            "request": "launch",
            "program": "cartography/__main__.py",
            "console": "integratedTerminal",
            "justMyCode": true,
            "args": [
                "--neo4j-user",
                "ignore",
                "--neo4j-password-env-var",
                "NEO4J_PASS",
                "--okta-org-id",
                "dev-xxxxxxxx",
                "--okta-api-key-env-var",
                "OKTA_API_TOKEN"
            ],
            "env": {
                "NEO4J_PASS": "ignore",
                "OKTA_API_TOKEN": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
            }
        }
    ]
}

You might notice we’re telling Cartography to connect to Neo4j (Memgraph actually) with username and password both set to “ignore”. The reason for this is that while the Community Edition of Memgraph does not require (or support) authentication, the Neo4j Bolt Driver for Python (i.e. Neo4j client) does require a username and password to be provided. So, as a second critical compatibility step, we pass any arbitrary value for the Neo4j username and password so long as they are not left empty.

As for the Okta configuration, remember to replace the Organisation ID and API Token with real ones.

Incompatible Index Creation Cypher

Pressing F5, we can now run Cartography from inside Visual Studio Code, and we immediately run into the first problem:

The error says: “no viable alternative at input ‘CREATEINDEXIF'”.

Memgraph is choking on the index creation step in indexes.cypher (in VS Code, use Ctrl+P / Command+P to quickly locate the file) because the index creation syntax is one aspect of Memgraph’s Cypher implementation that is not compatible with that of Neo4j. If we take the first line in the file, the Neo4j-compatible syntax is:

CREATE INDEX IF NOT EXISTS FOR (n:AWSConfigurationRecorder) ON (n.id);

…whereas the equivalent on Memgraph would be:

CREATE INDEX ON :AWSConfigurationRecorder(id);

Note: in Memgraph, the “IF NOT EXISTS” bit is implicit: an index is created if it doesn’t exist; if it does, the operation is a no-op that does not cause any error.

Fortunately, this syntactic difference is easily resolved by replacing (using VS Code search & replace syntax, with regex enabled) this:

CREATE INDEX IF NOT EXISTS FOR \(n:(.*?)\) ON \(n.(.*?)\);

…with this:

CREATE INDEX ON :$1($2);

Tip: although not in scope here, you’ll need to make a similar change also in querybuilder.py and tx.py if you also want to get other intel modules (e.g. AWS) working.

Neo4j Result Consumption

After fixing the index creation syntax and rerunning Cartography, we run into another problem:

The error says: “The result is out of scope. The associated transaction has been closed. Results can only be used while the transaction is open.”

I’m told that consume() is used to fix a problem in which Neo4j connections hang in situations where internal buffers fill up, although the Cartography team is re-evaluating whether this is necessary. In practice, I have seen that removing this doesn’t seem to cause problems with datasets I’ve tested with, although your mileage may vary. Let’s fix this problem by removing usage of consume() in statement.py.

First, we drop the .consume() at the end of line 76 inside the run() function:

    def run(self, session: neo4j.Session) -> None:
        """
        Run the statement. This will execute the query against the graph.
        """
        if self.iterative:
            self._run_iterative(session)
        else:
            session.write_transaction(self._run_noniterative)
        logger.info(f"Completed {self.parent_job_name} statement #{self.parent_job_sequence_num}")

Then, in the _run_iterative() function, we remove the entire while loop (lines 120-128) except for line 121, which we de-indent:

        # while True:
        result: neo4j.Result = session.write_transaction(self._run_noniterative)

            # Exit if we have finished processing all items
            # if not result.consume().counters.contains_updates:
            #     # Ensure network buffers are cleared
            #     result.consume()
            #     break
            # result.consume()

When we run it again, it should finish the run without problems and return control of the terminal with the prompt showing:

...
INFO:cartography.sync:Finishing sync stage 'duo'
INFO:cartography.sync:Starting sync stage 'analysis'
INFO:cartography.intel.analysis:Skipping analysis because no job path was provided.
INFO:cartography.sync:Finishing sync stage 'analysis'
INFO:cartography.sync:Finishing sync with update tag '1689401212'
daniel@andromeda:~/git/cartography$

Querying the Graph

The terminal we’re using to run Memgraph has the mgconsole client running (that’s the memgraph> prompt you see in the earlier screenshot), meaning we can try running queries directly there. For starters, we can try the ubiquitous “get everything” Cypher query:

memgraph> match (n) return n;

Note: if you get a “mg_raw_transport_send: Broken pipe”, just run the query again and it should reconnect.

This gives us some data back:

Querying Memgraph using mgconsole.

As you can see, this is not great to visualise results. Fortunately, Memgraph has its own web client (similar to Neo4j Browser) called Memgraph Lab, that you can access on http://localhost:3000/:

Memgraph Lab: Quick Connect page.

On the Quick Connect page, click the “Connect now” button. Then, switch to the “Query Execution” page using the left navigation sidebar, and you can run queries and view results more comfortably:

Seeing some nodes in Memgraph Lab.

Unlike Neo4j Browser, Memgraph Lab does not return relationships by default when you run this query. If you want to see them as well, you can run this instead:

match (a)-[r]->(b)
return a, r, b
Nodes and relationships in Memgraph Lab.

If the graph looks too cluttered, just drag the nodes around to rearrange them in a way that is more pleasant.

More Cartography with Memgraph

Cartography is a huge project that gathers data from a variety of data sources including AWS, Azure, GitHub, Okta, and others.

I’ve intentionally only covered the Okta intel module in this article because it’s small in scope and easy to digest. To use Cartography with other data sources, additional effort is required to address other problems with incompatible Cypher queries. For instance, at the time of writing this article, there are at least 9 outstanding issues that need to be fixed before Cartography can be used with Memgraph for AWS (that’s quite impressive considering that the AWS intel module is the biggest). Other intel modules may have other problems that need solving; nobody has explored them with Memgraph yet.

Summary

In this article, I’ve shown how one could go about taking an existing application that depends on Neo4j and migrating it to Memgraph. I’ve used Cartography with its Okta intel module to keep things relatively straightforward. The steps involved include:

  1. Running Memgraph with --bolt-server-name-for-init=Neo4j/
  2. Using the same Bolt-compatible Neo4j client, providing arbitrary Neo4j username and password values
  3. Fixing any incompatible Neo4j client code (in this case, consume()), if applicable
  4. Adjusting any incompatible Cypher queries
Software development

Minimal Web Application with Flask and AlpineJS

1 Comment

I recently needed to develop a small application for personal use. Since I had a specific need to use Python, I decided to build the backend with Flask. For the frontend, I only needed (literally) a single page to visualise some data, so I resisted the urge to go with something like Angular or React, and instead decided to try this minimal Alpine.js thing I had heard about a few days earlier.

In this article, I’m going to show you how you can really quickly create a minimal web application (REST API, static files and web UI) like I did. This can be handy for putting a simple tool together in very little time, but please consider it as a quick and dirty thing rather than any kind of best practice. I’m new to both Flask and Alpine.js, so I’m certainly not going to be giving advice on how best to use them.

Getting Started with Flask

After creating a new folder for your project, create a file called main.py (or whatever you want, really). We can kick off our API development by using the Flask Quickstart example:

from flask import Flask

app = Flask(__name__)

@app.route("/")
def hello_world():
    return "<p>Hello, World!</p>"

The same page also explains how to run the application so that it can listen for connections and serve requests:

“To run the application, use the flask command or python -m flask. You need to tell the Flask where your application is with the --app option.”

— Flask Quickstart

This is nice and all, but I like to be able to debug my code from Visual Studio Code, and for that it’s necessary to create a launch.json file with the right settings. This can be done as illustrated by the following images and their captions. If you need to do anything more advanced, see “Working with VS Code Launch Configurations“.

First, select the Debug tab on the left, and click “create a launch.json” file.
From the list that appears, select “Flask”.
Change the default suggested filename of app.py to the one you’re using, e.g. main.py.

Once that is done, it generates a launch.json file with the following contents:

{
    // Use IntelliSense to learn about possible attributes.
    // Hover to view descriptions of existing attributes.
    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Python: Flask",
            "type": "python",
            "request": "launch",
            "module": "flask",
            "env": {
                "FLASK_APP": "main.py",
                "FLASK_DEBUG": "1"
            },
            "args": [
                "run",
                "--no-debugger",
                "--no-reload"
            ],
            "jinja": true,
            "justMyCode": true
        }
    ]
}

You can now press F5 to start the application in debug mode. The integrated terminal informs you that the application is running at http://localhost:5000/, so you can open a browser window to see that it’s working. By running it in this way, you can also add breakpoints and debug your Python code as needed.

The Flask “Hello World” is definitely working.

A Simple REST API

To build a reasonable example, we need to return some data from our API. We can do this quite easily by returning a list of dictionary objects from a function annotated with the route “/products”:

@app.route("/products")
def get_products():
    return [
        {
            "name": "Rope",
            "description": "Used to climb heights or swing across chasms.",
            "price": 15
        },
        {
            "name": "Whip",
            "description": "An old and trusty friend.",
            "price": 20
        },
        {
            "name": "Notebook",
            "description": "Contains lots of valuable information!",
            "price": 80
        }
    ]

After restarting the application and visiting http://localhost:5000/products, we can see that the list gets translated quite nicely to JSON:

The response of the /products endpoint.

Serving a Static Webpage

It is now time to build a simple user interface. Before we do that though, we need to figure out how to serve static files so that we can return an HTML file and then use Alpine.js to dynamically populate it using the API.

It turns out that this is quite simple: Flask automatically serves any files you put under the static folder. So let’s create a static folder and add an index.html file with some minimal HTML in it:

<!doctype html>
<html lang="en">
<meta charset="utf-8">
<title>My Products</title>
<body>

<h1>My Products</h1>

</body>
</html>

We can now see this page if we visit http://localhost:5000/static/index.html in a browser:

The HTML page is served from the static folder.

It works, but it’s not great. I’d like this to come up by default when I visit http://localhost:5000/, instead of having such a long URL. Fortunately, this is easy to achieve. All we need to do is replace our “Hello world” code in the default endpoint with the following:

from flask import Flask, send_from_directory

app = Flask(__name__)

@app.route('/')
def serve_index():
    return send_from_directory('static', 'index.html')

Displaying the Products

Now that we have an API and an easily accessible webpage, we can display the data after retrieving it from the API. We can now add Alpine.js to our website by adding a <script> tag, and then build out a table that retrieves data from the API:

<!doctype html>
<html lang="en">
<meta charset="utf-8">
<title>My Products</title>
<script defer src="https://cdn.jsdelivr.net/npm/alpinejs@3.x.x/dist/cdn.min.js"></script>
<style>
    body {
        font-family: sans-serif;
    }

    table, th, td {
        border: solid 1px #555;
        border-collapse: collapse;
    }
</style>
<body>

<h1>My Products</h1>

<div
    x-data="{ products: [] }"
    x-init="products = await (await fetch('/products')).json()"
>
    <table>
        <tr>
            <th>Name</th>
            <th>Description</th>
            <th>Price</th>
        </tr>
        <template x-for="product in products">
            <tr>
                <td x-text="product.name"></td>
                <td x-text="product.description"></td>
                <td x-text="product.price"></td>
            </tr>
        </template>
    </table>
</div>

</body>
</html>

In the above HTML, the main thing to look at is the stuff inside the <div>. Outside of that, the new things are just the <script> tag that imports Alpine.js, and a little CSS to make the webpage look almost bearable.

The <div> tag’s attributes set up the data model that the table is based on. Using this x-data attribute, we’re saying that we have an array called “products”. The x-init attribute is where we use the Fetch API to retrieve data from the /products endpoint and assign the result to the “products” array.

Once this data is retrieved, all that’s left to do is display it. In the <table>, we use an x-for attribute in a <template> to loop over the “products” array. Then, we bind the content of each cell (<td> element) to the properties of each product using the x-text attribute, effectively displaying them.

The result is simple but effective:

Indiana Jones would be proud of this inventory.

Conclusion

As you can see, it’s pretty quick and easy to put something simple together and then build whatever you need on top of that. For instance, you might want to filter the data you display, apply conditional highlighting, or add other functionality.

Honestly, I’m not quite sure how far you can take Alpine.js. For instance, it doesn’t seem possible to create reusable components, which somewhat limits its usefulness. And I don’t yet know how to avoid having heaps of JavaScript within HTML attributes. I’ve only started using this a few days ago, after all.

But it’s great to be able to use something like this to create simple tools without having to incur the learning curves of things like Angular, React or Vue.

Software, Software development

The Wonderful World of Barcodes

Leave a comment

There are lots of different ways to encode data in software. For instance, text encodings such as ASCII or UTF-8 allow us to represent strings of arbitrarily complex characters, while base64 allows us to represent even binary data as text and transmit it over channels that are more suited to deal with text (such as many internet protocols).

A barcode scanner next to a stack of books, each with its own barcode.

Barcodes, on the other hand, allow us to label physical objects and read them efficiently, providing a bridge between software and the physical world.

Although barcodes could represent any data, they are usually used to represent numbers. They are most commonly used in retail for fast processing of items being sold, especially by supermarkets, but also by pharmacies, bookshops, electronics stores, and other shops.

Many items we buy, from everyday groceries to books, are labelled with numbers (which I’ll call “product codes”) that form part of one or more standards, such as EAN-13, UPC-A, or ISBN. These numbers are encoded as a barcode on the product itself or its packaging. Several websites exist where you can look up these numbers. Other products, such as fruit, do not have standard identifiers, but that doesn’t prevent shops from applying their own.

Barcode Scanners

Barcode scanners can be used to read these product codes. They are quite cheap and can connect via USB to any system, making them widely compatible with Windows, Linux, MacOS, and various cash registers. A barcode scanner is held like a gun and, when the trigger is pulled, it emits a red light that reads the information on the barcode. This is immediately written to any text field that currently has focus, followed by a newline. For instance, if I open a text editor and quickly scan the barcodes on the four books shown in the above picture, I get the following:

9780201633542
9780201633467
9780201433074
9780201633610

Scanning these numbers into a text file is useful for testing or to later bulk-process them, but we can also use this handy input method in other ways. For instance, if we open ISBN Search and click on the search field, scanning a book’s barcode will not only populate the search field with the ISBN number, but also execute the search:

The ISBN Search homepage.
ISBN Search result for 9780201633542 is “TCP/IP Illustrated: The Implementation, Vol. 2”.

Barcodes in Retail

Shops that sell products with barcodes on them can use those barcodes as primary keys to identify each product. This makes it very easy to create a database with information about the various products (including name, price, taxes, discounts, etc) and then have the cashier look this up by just entering the barcode. We can simulate this by writing a simple program, e.g. in Python.

First, we define a simple mapping between each product’s barcode and its price:

from decimal import Decimal

prices = {
    "9780201633542": Decimal(35.75),
    "9780201633467": Decimal(28.90),
    "9780201433074": Decimal(42.30),
    "9780201633610": Decimal(0.50)
}

Then, we write a simple loop that reads product codes one by one. When an empty input is received, it means there are no more products, and the total price is calculated:

total_price = Decimal(0)

while True:
    product_code = input("Product code: ")

    if product_code == "":
        break
    else:
        price = prices.get(product_code)
        if price == None:
            print('Unknown product code')
        else:
            total_price += price

print(f'Total: {total_price:.2f}')

Running this, we can scan a couple of barcodes and process a sale in a jiffy:

Product code: 9780201633542
Product code: 9780201633467
Product code: 
Total: 64.65

Building a Book Catalogue

Just like a shop can build a database by scanning barcodes and enhancing them with additional information, anyone can. In fact, you can find several online databases where you can look up barcodes, such as the aforementioned ISBN Search, EAN-Search, Barcode Lookup, and others. Some even offer APIs so that you can automate the lookup and build a fully-featured database very quickly by just scanning a bunch of barcodes.

Scanning a book’s ISBN number with the Handy Library mobile app.

If you run a library or are just a private individual with a lot of books, you can use this approach to build a catalogue of your books. The easiest way is to use the Handy Library mobile app, which allows you to scan a book’s ISBN number and through it acquire all the book’s metadata too. If you prefer to have your data on a computer, you can export data from Handy Library, or else build your database yourself by scanning barcodes and looking up the information afterwards.

QR Codes

Barcodes may be great in retail, but most people don’t carry barcode scanners with them wherever they go. Now that everybody and his dog has a smartphone with a powerful camera, QR codes have emerged as a more recent standard to encode data that smartphones can easily scan with their default camera app. The most common use case is to encode a URL, allowing people to visit it by scanning it with their phone’s camera rather than typing it in. The following are a few creative uses I’ve come across in my travels:

  • Exhibitions: putting a QR code beside the label of an exhibit in an art gallery or museum allows people to scan it and visit a webpage with more information. It’s also possible to have this for multiple languages.
  • Paying bills: in Switzerland, the new standard way of paying bills is to scan a QR code containing all payment details with a bank’s mobile app.
  • Contact details: A long time ago, I had an idea that people could easily exchange contact details by scanning a QR code. Someone has probably done this already. Similar applications I’ve seen in the wild are physical business cards with QR codes that lead to the company or owner’s website, and the LinkedIn app which displays a QR code leading to the user’s profile.
  • Queue ticket info: when you take a ticket and wait your turn in some customer service departments, the ticket contains a QR code that tells you how many people are ahead of you and how long you can expect to wait. You could potentially run a quick errand and keep track of your place while you’re away.
  • Two-Factor Authentication: QR codes are the easiest way to initially exchange the secret in Time-Based One-Time Passwords (TOTP), as shown in “Using Time-Based One-Time Passwords for Two-Factor Authentication“.

Conclusion

Nowadays, we talk a lot about artificial intelligence, augmented reality, robotics the internet of things, and so on. Many of these fields are awe-inspiring, yet not very accessible to the man in the street.

On the other hand, barcodes have provided a very simple way for software to interact with physical objects for decades. The advantages in efficiency are hard to dispute, as can be seen in the retail industry.

The ubiquity of smartphones and the more recent widespread use of QR codes has enabled the same concept to extend to a lot of other use cases, allowing us to interact with some very traditional settings in novel ways.

Software, Software development

Working with VS Code Launch Configurations

3 Comments

Visual Studio Code (VS Code) is a wonderful IDE. I’m not generally known to praise Microsoft’s products, but VS Code lets me develop and debug code productively in all the languages I’ve needed to work with, on any operating system. I don’t even use Visual Studio any more.

Launch configurations are at the heart of debugging with VS Code. In this article, I’ll explain how you can debug code using different languages, even at the same time. I’ll also show you how you can customise these launch configurations to pass command-line arguments, set environment variables, run pre-launch tasks, and more.

Getting Started with Launch Configurations

Anytime you want to debug something, you just press F5 (like in Visual Studio). Initially, you probably won’t have any launch configurations set up, in which case you’ll be prompted to choose what kind of launch configuration you want to create, as shown in the screenshot below. This list might vary depending on the language runtimes you have installed. When you select one, you’ll be guided towards creating a sample launch configuration.

Pressing F5 brings up a list of languages for which you can create launch configurations.

Another way is to click on the Debug tab on the left, which looks like a play button. You can then click the link to “create a launch.json” file, shown in the screenshot below. We’ll see this in practice in the next sections as we create launch configurations for various languages.

The Debug tab allows you to “create a launch.json file”.

Debugging Python

Before we debug anything, we need some code. Let’s add a folder with a Python file in it, and add the following code:

import time
import datetime

while True:
    time.sleep(1)
    print(f"Swiss church bells say it's {datetime.datetime.now()}")

If I press F5, this actually works for me out of the box:

Pressing F5 runs the Python program and lets us debug it.

Clicking next to a line number adds a breakpoint. When the program hits that point, it pauses and allows us to inspect variables and other things. You can press F10 to go to the next statement, F11 to step into a function call, and Shift+F11 to step out. While a tutorial on how to debug code is outside the scope of this article, if you’re unfamiliar with debugging, this should at least be enough to get you started.

After hitting a breakpoint in the Python program, we can advance step by step and see the state of local variables.

A Launch Configuration for Python

Follow either of the methods in the earlier “Getting Started” section to create your first Python launch configuration. This creates a launch.json file under a .vscode folder with the following contents. As launch.json may contain personally customised configurations for different developers (e.g. with different input parameters), it’s best not to commit it to source control.

{
    // Use IntelliSense to learn about possible attributes.
    // Hover to view descriptions of existing attributes.
    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Python: Current File",
            "type": "python",
            "request": "launch",
            "program": "${file}",
            "console": "integratedTerminal",
            "justMyCode": true
        }
    ]
}

There isn’t much to it: this will simply run whatever file you have open in VS Code (which is what ${file} means). This is useful if you want to run specific files (I do this with tests in Go for instance), but not so much if you have a program with a single entry point and want to run that regardless of what you have open in VS Code. In that case, it’s easy to change the program value:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Python: Main File",
            "type": "python",
            "request": "launch",
            "program": "${workspaceFolder}/pyticker/main.py",
            "console": "integratedTerminal",
            "justMyCode": true
        }
    ]
}

${workspaceFolder} represents the folder we have open in VS Code, so this makes sure we’re relative to that. ${workspaceFolder}, ${file} and other such variables are documented in VS Code’s Variables Reference.

Command Line Arguments

Let’s modify our Python program as follows:

import time
import datetime
import sys

while True:
    time.sleep(int(sys.argv[2]))
    print(f"{sys.argv[1]} church bells say it's {datetime.datetime.now()}")

The program now takes the nationality of the church bells as well as the sleep interval from command-line arguments. We’re not doing validation or error-handling for the sake of brevity. The following is an example of how to execute this successfully from a terminal:

$ python3 main.py Maltese 5
Maltese church bells say it's 2023-02-15 19:04:28.311805
Maltese church bells say it's 2023-02-15 19:04:33.315787
Maltese church bells say it's 2023-02-15 19:04:38.319811

To pass the same command-line arguments when debugging with VS Code, we add args to the launch configuration:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Python: Main File",
            "type": "python",
            "request": "launch",
            "program": "${workspaceFolder}/pyticker/main.py",
            "console": "integratedTerminal",
            "justMyCode": true,
            "args": ["Maltese", "5"]
        }
    ]
}

Multiple Launch Configurations

As you no doubt noticed, launch.json contains a JSON array of configurations. That means it’s very easy to add more of these configurations, for instance, when you need to provide different inputs:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Python Ticker (Maltese)",
            "type": "python",
            "request": "launch",
            "program": "${workspaceFolder}/pyticker/main.py",
            "console": "integratedTerminal",
            "justMyCode": true,
            "args": ["Maltese", "5"]
        },
        {
            "name": "Python Ticker (Swiss)",
            "type": "python",
            "request": "launch",
            "program": "${workspaceFolder}/pyticker/main.py",
            "console": "integratedTerminal",
            "justMyCode": true,
            "args": ["Swiss", "1"]
        }
    ]
}

From the Debug tab, you can then select the configuration you want to run from the drop-down before hitting F5 to debug with that configuration:

The drop-down in the Debug tab lets you select which Launch Configuration you want to debug.

A Launch Configuration for Node.js

Let’s add a new folder as a sibling to our Python folder, and add the following code in a new app.js file, completely disregarding cleanup for the sake of brevity and YOLO:

const fs = require('fs');

setInterval(() => {
    message = `Swiss church bells say it's ${new Date()}`;
    console.log(message)
    fs.appendFileSync('ding-dong.txt', message + '\n');
}, 1000);

We can set up a simple launch configuration to run this in launch.json:

...
    "configurations": [
        {
            "name": "Node.js Ticker",
            "program": "${workspaceFolder}/jsticker/app.js",
            "request": "launch",
            "type": "node"
        },
...

It works, writing output every second both to standard output and a file:

The Node.js program’s output can be seen in the Debug Console as well as the output file. The overall folder structure is also shown on the left.

The only problem is that the file was created top-level, and since we only specified a filename in the code (not a folder or path), then this must mean that Node.js is running with the top-level folder in VS Code as the current working directory. We can change that in the launch configuration using cwd:

...
    "configurations": [
        {
            "name": "Node.js Ticker",
            "program": "${workspaceFolder}/jsticker/app.js",
            "request": "launch",
            "cwd": "${workspaceFolder}/jsticker",
            "type": "node"
        },
...

When we run this again, the file is created under the jsticker folder.

Pre-Launch Tasks

Sometimes you want to run something before starting your program. For instance, in my work setup, I generate Swagger docs and perform other prerequisite tasks. But since we’re not doing anything that fancy, we’ll just delete the ding-dong.txt file every time we run the “Node.js Ticker” launch configuration.

To do this, we first need to add a tasks.json file inside the .vscode folder, next to launch.json, and add the following to it:

{
    "version": "2.0.0",
    "tasks": [
      {
        "label": "rmfile",
        "command": "rm",
        "args": ["ding-dong.txt"],
        "options":{
            "cwd": "${workspaceFolder}/jsticker"
        }
      }
    ]
  }

This defines a task called rmfile that will run rm ding-dong.txt from the jsticker folder. We then refer to this task in the relevant launch configuration using preLaunchTask:

...
        {
            "name": "Node.js Ticker",
            "program": "${workspaceFolder}/jsticker/app.js",
            "request": "launch",
            "cwd": "${workspaceFolder}/jsticker",
            "preLaunchTask": "rmfile",
            "type": "node"
        },
...

A Launch Configuration for Go (GoLang)

For Go, we need to:

  1. Run go work init from the top-level folder opened in VS Code
  2. Create a new folder alongside pyticker and jsticker
  3. Run go mod init main in it
  4. Add the following code to a new main.go file:
package main

import (
	"fmt"
	"time"
)

func main() {
	for {
		now := time.Now()
		fmt.Printf("Swiss bells won't stop ringing! It's %s!\n", now)
		time.Sleep(time.Second)
	}
}

We can now add a launch configuration for this Go program:

...
        {
            "name": "Go Ticker",
            "type": "go",
            "request": "launch",
            "mode": "debug",
            "program": "${workspaceFolder}/goticker/main.go"
        },
...

And we can have lots of fun running and debugging it, as a reminder that Swiss church bells need to tell you the time all the time. It’s not like they sell watches in Switzerland or anything.

Our Go program runs via the “Go Ticker” launch configuration.

Environment Variables

As we’ve seen with the Python example, however, other countries also have church bells. Instead of using command-line arguments to customise the output, we’ll instead pass environment variables. First, we need to modify the code a little:

package main

import (
	"fmt"
	"os"
	"strconv"
	"time"
)

func main() {
	for {
		country := os.Getenv("COUNTRY")
		intervalSecsStr := os.Getenv("INTERVAL_SECS")
		intervalSecs, _ := strconv.Atoi(intervalSecsStr)
		now := time.Now()
		fmt.Printf("%s bells won't stop ringing! It's %s!\n", country, now)
		time.Sleep(time.Duration(intervalSecs) * time.Second)
	}
}

Then we add the relevant environment variables as key-value pairs in an env block in the launch configuration:

...
        {
            "name": "Go Ticker",
            "type": "go",
            "request": "launch",
            "mode": "debug",
            "program": "${workspaceFolder}/goticker/main.go",
            "env": {
                "COUNTRY": "Maltese",
                "INTERVAL_SECS": "3"
            }
        },
...

When you run it again, it makes all the difference:

The Go program runs with environment variables set.

Go Build Flags

Go has some specific build flags that can’t be passed as regular command-line parameters, such as build tags or the Data Race Detector. If you want to use these, you’ll have to pass them via buildFlags instead:

...
            "buildFlags": "-race",
...

Running Multiple Programs with Compounds

It’s common to need to run multiple programs at once, especially in a microservices architecture, or if there are separate backend and frontend applications. This can be done in VS Code using compound launch configurations. For instance, if we wanted to run both the Python and Go programs, we could define a compound as follows (compounds go after configurations in launch.json):

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Go Ticker",
            "type": "go",
            "request": "launch",
            "mode": "debug",
            "program": "${workspaceFolder}/goticker/main.go",
            "env": {
                "COUNTRY": "Maltese",
                "INTERVAL_SECS": "3"
            }
        },
        {
            "name": "Node.js Ticker",
            "program": "${workspaceFolder}/jsticker/app.js",
            "request": "launch",
            "cwd": "${workspaceFolder}/jsticker",
            "preLaunchTask": "rmfile",
            "type": "node"
        },
        {
            "name": "Python Ticker (Maltese)",
            "type": "python",
            "request": "launch",
            "program": "${workspaceFolder}/pyticker/main.py",
            "console": "integratedTerminal",
            "justMyCode": true,
            "args": ["Maltese", "5"]
        },
        {
            "name": "Python Ticker (Swiss)",
            "type": "python",
            "request": "launch",
            "program": "${workspaceFolder}/pyticker/main.py",
            "console": "integratedTerminal",
            "justMyCode": true,
            "args": ["Swiss", "1"]
        }
    ],
    "compounds": [
        {
            "name": "Go and Python Tickers",
            "configurations": ["Go Ticker", "Python Ticker (Swiss)"]
        }
    ]
}

The compound simply refers to the individual launch configurations by name, and has a name of its own. It can then be selected from the drop-down in the Debug tab just like any other launch configuration. By running a compound, you can debug and hit breakpoints in any application that is part of that compound.

In fact, you’ll notice there is a new drop-down next to the debugging buttons towards the central-top part of the screen. This allows you to switch certain views (e.g. the Debug Console) between running applications.

A compound can be selected from the list of launch configurations in the Debug tab. When it is run, a dropdown appears next to the debugging buttons in the top part of the screen.

Conclusion

We’ve seen that Launch Configurations allow you to run and debug applications in VS Code with great flexibility, including:

  • Run the current file or a specific one
  • Set the current working directory
  • Run different programs
  • Use different programming languages
  • Set up different configurations even for the same program
  • Pass command-line arguments
  • Set environment variables
  • Run/debug multiple programs at the same time

This provides a great IDE experience even for more complex application architectures in a monorepo.

Software development

Using the Neo4j Bolt Driver for Python with Memgraph

Leave a comment

Memgraph is a relatively young graph database. It supports the Cypher query language and the Bolt protocol – just like Neo4j – therefore it is usually possible to use Neo4j client libraries (called “drivers”) with Memgraph. In fact, according to the Memgraph Drivers documentation, using the Neo4j drivers is the recommended way to communicate with Memgraph from several languages like Go, Java and C#.

Memgraph Python Client Libraries

For Python, there are actually a number of options to choose from:

  • pymgclient: I discovered this recently and haven’t used it, but it seems to be a lower-level client that works well.
  • gqlalchemy: Memgraph’s recommended client for Python, which uses pymgclient underneath. Perhaps named after Python ORM sqlalchemy, it provides three different ways to query Memgraph, none of which I found to be very practical:
    • Basic execute() and execute_and_fetch(): this seems simple enough, but I haven’t found any way to pass parameters to queries, making it useless for my use case.
    • OGM: This is a graph equivalent of an ORM. It’s no secret that ORMs are one of the things I avoid like the plague – I’ve already written some of my thoughts on the subject in “ADO .NET Part 1: Introduction“, and time-permitting it will also be the subject of a future article. In a nutshell: I just want to write Cypher queries and execute them, not have to translate them to some library’s arbitrary API.
    • Query builder: A fluent query builder, similar in approach to what Elasticsearch provides for .NET. I’m not a fan for the same reasons that apply to ORMs (see previous point above).
  • Neo4j Bolt Driver for Python: This doesn’t work with Memgraph out of the box, but we’ll talk more about this.

It’s unfortunate that the Neo4j Bolt Driver for Python doesn’t work with Memgraph by default, because if you already have Python code that works with Neo4j, you could otherwise use Memgraph as a drop-in replacement for Neo4j with minimal changes (e.g. fixing incompatible Cypher).

For the rest of this article, I will be focusing on the Neo4j Bolt Driver for Python, to understand why we can’t use it with Memgraph and explain how to get around the problem.

Update 21st November 2022: TL;DR: if you need a quick solution, go to the end of this article.

Why the Neo4j Driver Fails with Memgraph

Let’s make a first attempt to use the Neo4j Bolt Driver for Python with Memgraph.

First, we need to have an instance of Memgraph running. The easiest way is to run it with Docker, e.g. as follows (assuming you’re on Linux):

sudo docker run --rm -it -p 7687:7687 -p 3000:3000 memgraph/memgraph-platform

If this works, it will start a Memgraph shell, and you can also access Memgraph Lab (Memgraph’s web user interface) by visiting http://localhost:3000/.

Memgraph shell after running it with Docker, and Memgraph Lab (UI) open in Firefox in the background.

Next, create a folder for your Python code. Run the following to install the Neo4j driver:

pip3 install neo4j

At the time of writing this article, the version of the Neo4j Python driver is 5.2.1. With earlier versions, it’s possible you might run into errors such as:

neobolt.exceptions.SecurityError: Failed to establish secure connection to ‘[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1131)’

In this case, update the driver as follows:

pip3 install neo4j --upgrade

At this point, we can steal some example code from the Neo4j Bolt Driver for Python, as follows, and put it in a file called main.py:

from neo4j import GraphDatabase

driver = GraphDatabase.driver("neo4j://localhost:7687",
                              auth=("neo4j", "password"))

def add_friend(tx, name, friend_name):
    tx.run("MERGE (a:Person {name: $name}) "
           "MERGE (a)-[:KNOWS]->(friend:Person {name: $friend_name})",
           name=name, friend_name=friend_name)

def print_friends(tx, name):
    query = ("MATCH (a:Person)-[:KNOWS]->(friend) WHERE a.name = $name "
             "RETURN friend.name ORDER BY friend.name")
    for record in tx.run(query, name=name):
        print(record["friend.name"])

with driver.session(database="neo4j") as session:
    session.execute_write(add_friend, "Arthur", "Guinevere")
    session.execute_write(add_friend, "Arthur", "Lancelot")
    session.execute_write(add_friend, "Arthur", "Merlin")
    session.execute_read(print_friends, "Arthur")

driver.close()

Once we run this with python3 main.py, we get a nice big error:

$ python3 main.py
Traceback (most recent call last):
  File "main.py", line 18, in <module>
    session.execute_write(add_friend, "Arthur", "Guinevere")
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/work/session.py", line 712, in execute_write
    return self._run_transaction(
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/work/session.py", line 484, in _run_transaction
    self._open_transaction(
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/work/session.py", line 396, in _open_transaction
    self._connect(access_mode=access_mode)
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/work/session.py", line 123, in _connect
    super()._connect(access_mode, **access_kwargs)
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/work/workspace.py", line 198, in _connect
    self._connection = self._pool.acquire(**acquire_kwargs_)
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/io/_pool.py", line 778, in acquire
    self.ensure_routing_table_is_fresh(
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/io/_pool.py", line 721, in ensure_routing_table_is_fresh
    self.update_routing_table(
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/io/_pool.py", line 648, in update_routing_table
    if self._update_routing_table_from(
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/io/_pool.py", line 596, in _update_routing_table_from
    new_routing_table = self.fetch_routing_table(
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/io/_pool.py", line 534, in fetch_routing_table
    new_routing_info = self.fetch_routing_info(
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/io/_pool.py", line 504, in fetch_routing_info
    cx = self._acquire(address, deadline, None)
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/io/_pool.py", line 221, in _acquire
    return connection_creator()
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/io/_pool.py", line 138, in connection_creator
    connection = self.opener(
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/io/_pool.py", line 441, in opener
    return Bolt.open(
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/io/_bolt.py", line 377, in open
    connection.hello()
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/io/_bolt4.py", line 450, in hello
    check_supported_server_product(self.server_info.agent)
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/io/_common.py", line 283, in check_supported_server_product
    raise UnsupportedServerProduct(agent)
neo4j.exceptions.UnsupportedServerProduct: None

The last three lines indicate that the problem seems to be a simple validation, which we can confirm by looking up the offending function in the Neo4j driver’s source code:

def check_supported_server_product(agent):
    """ Checks that a server product is supported by the driver by
    looking at the server agent string.
    :param agent: server agent string to check for validity
    :raises UnsupportedServerProduct: if the product is not supported
    """
    if not agent.startswith("Neo4j/"):
        raise UnsupportedServerProduct(agent)

What would happen if we simply disable this check? Let’s find out.

Tweaking the Neo4j Driver to Work with Memgraph

First, let’s clone the Neo4j driver’s repo:

git clone https://github.com/neo4j/neo4j-python-driver.git

A quick search shows that there are two places where the server product check is done:

There are two equivalent check_supported_server_product() functions in _neo4j/_async/io/_common.py and _neo4j/_sync/io/_common.py.

We can disable the validation by replacing the implementation of each function with just pass:

def check_supported_server_product(agent):
    pass

Next, we build this modified version of the Neo4j driver as follows:

python3 setup.py sdist

This creates a file called neo4j-5.2.dev0.tar.gz in a dist subfolder. Take note of the path of this file.

Back in the folder with our Python test code (where we were attempting to communicate with Memgraph), install the package we just built:

$ pip3 install /home/daniel/Desktop/neo4j-python-driver/dist/neo4j-5.2.dev0.tar.gz
Processing /home/daniel/Desktop/neo4j-python-driver/dist/neo4j-5.2.dev0.tar.gz
Requirement already satisfied: pytz in /home/daniel/.local/lib/python3.8/site-packages (from neo4j==5.2.dev0) (2022.1)
Building wheels for collected packages: neo4j
  Building wheel for neo4j (setup.py) ... done
  Created wheel for neo4j: filename=neo4j-5.2.dev0-py3-none-any.whl size=244857 sha256=ec2951ea1fecf2ae1aacced4d93c66b1b5d90bc3710746ff3814b9b62a96a9af
  Stored in directory: /home/daniel/.cache/pip/wheels/0d/4c/55/2486d65ebf98105bc54a490ebd91cea4ba538268a32ffc91f0
Successfully built neo4j
Installing collected packages: neo4j
  Attempting uninstall: neo4j
    Found existing installation: neo4j 5.2.1
    Uninstalling neo4j-5.2.1:
      Successfully uninstalled neo4j-5.2.1
Successfully installed neo4j-5.2.dev0

Run the Python code again…

$ python3 main.py
Unable to retrieve routing information
Transaction failed and will be retried in 0.9256931081701124s (Unable to retrieve routing information)
Unable to retrieve routing information
Transaction failed and will be retried in 2.0779915720272504s (Unable to retrieve routing information)

We still have a failure, but this is a simple connectivity issue that is easily fixed by changing the scheme in the URI from neo4j to bolt:

driver = GraphDatabase.driver("bolt://localhost:7687",
                              auth=("neo4j", "password"),)

Running it again, we see that it now works!

$ python3 main.py
Guinevere
Lancelot
Merlin

We can also view the created data in Memgraph Lab to double-check that it really worked:

Querying the data in Memgraph Lab, we see that the example nodes were created.

Conclusion

In this article, we’ve confirmed that, at a basic level, the only thing preventing the Neo4j Bolt Driver for Python from being used with Memgraph is a simple check against a response from the server. We saw that queries could be executed once this check was disabled.

As a result, it’s not clear why Memgraph built their own Python clients instead of simply addressing this check (e.g. by sending the same response as Neo4j, or forking the driver and eliminating the check as I did). I will refrain from speculating on possible reasons, but I found this interesting to investigate and hope it saves time for other people in the same situation.

P.S.: There’s an Easier Way

This section was added on 21st November 2022.

I learned from the Memgraph team that they do provide a way to deal with the server check – it’s just not documented at the time of writing this article. Basically, all you have to do is run Memgraph using a --bolt-server-name-for-init switch that sets the missing server response. So if you run Memgraph in Docker, you’d need to run it as follows:

sudo docker run --rm -it -p 7687:7687 -p 3000:3000 -e MEMGRAPH="--bolt-server-name-for-init=Neo4j/" memgraph/memgraph-platform

If you run the example code with the bolt:// scheme using the unmodified Neo4j Bolt Driver for Python, it should work just as well.

Update 19th September 2023: as of Memgraph v2.11, --bolt-server-name-for-init has a default value compatible with the Neo4j Bolt Driver, and therefore no longer needs to be provided.