Using the Neo4j Bolt Driver for Python with Memgraph

Memgraph is a relatively young graph database. It supports the Cypher query language and the Bolt protocol – just like Neo4j – therefore it is usually possible to use Neo4j client libraries (called “drivers”) with Memgraph. In fact, according to the Memgraph Drivers documentation, using the Neo4j drivers is the recommended way to communicate with Memgraph from several languages like Go, Java and C#.

Memgraph Python Client Libraries

For Python, there are actually a number of options to choose from:

  • pymgclient: I discovered this recently and haven’t used it, but it seems to be a lower-level client that works well.
  • gqlalchemy: Memgraph’s recommended client for Python, which uses pymgclient underneath. Perhaps named after Python ORM sqlalchemy, it provides three different ways to query Memgraph, none of which I found to be very practical:
    • Basic execute() and execute_and_fetch(): this seems simple enough, but I haven’t found any way to pass parameters to queries, making it useless for my use case.
    • OGM: This is a graph equivalent of an ORM. It’s no secret that ORMs are one of the things I avoid like the plague – I’ve already written some of my thoughts on the subject in “ADO .NET Part 1: Introduction“, and time-permitting it will also be the subject of a future article. In a nutshell: I just want to write Cypher queries and execute them, not have to translate them to some library’s arbitrary API.
    • Query builder: A fluent query builder, similar in approach to what Elasticsearch provides for .NET. I’m not a fan for the same reasons that apply to ORMs (see previous point above).
  • Neo4j Bolt Driver for Python: This doesn’t work with Memgraph out of the box, but we’ll talk more about this.

It’s unfortunate that the Neo4j Bolt Driver for Python doesn’t work with Memgraph by default, because if you already have Python code that works with Neo4j, you could otherwise use Memgraph as a drop-in replacement for Neo4j with minimal changes (e.g. fixing incompatible Cypher).

For the rest of this article, I will be focusing on the Neo4j Bolt Driver for Python, to understand why we can’t use it with Memgraph and explain how to get around the problem.

Update 21st November 2022: TL;DR: if you need a quick solution, go to the end of this article.

Why the Neo4j Driver Fails with Memgraph

Let’s make a first attempt to use the Neo4j Bolt Driver for Python with Memgraph.

First, we need to have an instance of Memgraph running. The easiest way is to run it with Docker, e.g. as follows (assuming you’re on Linux):

sudo docker run --rm -it -p 7687:7687 -p 3000:3000 memgraph/memgraph-platform

If this works, it will start a Memgraph shell, and you can also access Memgraph Lab (Memgraph’s web user interface) by visiting http://localhost:3000/.

Memgraph shell after running it with Docker, and Memgraph Lab (UI) open in Firefox in the background.

Next, create a folder for your Python code. Run the following to install the Neo4j driver:

pip3 install neo4j

At the time of writing this article, the version of the Neo4j Python driver is 5.2.1. With earlier versions, it’s possible you might run into errors such as:

neobolt.exceptions.SecurityError: Failed to establish secure connection to ‘[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1131)’

In this case, update the driver as follows:

pip3 install neo4j --upgrade

At this point, we can steal some example code from the Neo4j Bolt Driver for Python, as follows, and put it in a file called main.py:

from neo4j import GraphDatabase

driver = GraphDatabase.driver("neo4j://localhost:7687",
                              auth=("neo4j", "password"))

def add_friend(tx, name, friend_name):
    tx.run("MERGE (a:Person {name: $name}) "
           "MERGE (a)-[:KNOWS]->(friend:Person {name: $friend_name})",
           name=name, friend_name=friend_name)

def print_friends(tx, name):
    query = ("MATCH (a:Person)-[:KNOWS]->(friend) WHERE a.name = $name "
             "RETURN friend.name ORDER BY friend.name")
    for record in tx.run(query, name=name):
        print(record["friend.name"])

with driver.session(database="neo4j") as session:
    session.execute_write(add_friend, "Arthur", "Guinevere")
    session.execute_write(add_friend, "Arthur", "Lancelot")
    session.execute_write(add_friend, "Arthur", "Merlin")
    session.execute_read(print_friends, "Arthur")

driver.close()

Once we run this with python3 main.py, we get a nice big error:

$ python3 main.py
Traceback (most recent call last):
  File "main.py", line 18, in <module>
    session.execute_write(add_friend, "Arthur", "Guinevere")
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/work/session.py", line 712, in execute_write
    return self._run_transaction(
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/work/session.py", line 484, in _run_transaction
    self._open_transaction(
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/work/session.py", line 396, in _open_transaction
    self._connect(access_mode=access_mode)
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/work/session.py", line 123, in _connect
    super()._connect(access_mode, **access_kwargs)
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/work/workspace.py", line 198, in _connect
    self._connection = self._pool.acquire(**acquire_kwargs_)
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/io/_pool.py", line 778, in acquire
    self.ensure_routing_table_is_fresh(
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/io/_pool.py", line 721, in ensure_routing_table_is_fresh
    self.update_routing_table(
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/io/_pool.py", line 648, in update_routing_table
    if self._update_routing_table_from(
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/io/_pool.py", line 596, in _update_routing_table_from
    new_routing_table = self.fetch_routing_table(
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/io/_pool.py", line 534, in fetch_routing_table
    new_routing_info = self.fetch_routing_info(
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/io/_pool.py", line 504, in fetch_routing_info
    cx = self._acquire(address, deadline, None)
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/io/_pool.py", line 221, in _acquire
    return connection_creator()
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/io/_pool.py", line 138, in connection_creator
    connection = self.opener(
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/io/_pool.py", line 441, in opener
    return Bolt.open(
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/io/_bolt.py", line 377, in open
    connection.hello()
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/io/_bolt4.py", line 450, in hello
    check_supported_server_product(self.server_info.agent)
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/io/_common.py", line 283, in check_supported_server_product
    raise UnsupportedServerProduct(agent)
neo4j.exceptions.UnsupportedServerProduct: None

The last three lines indicate that the problem seems to be a simple validation, which we can confirm by looking up the offending function in the Neo4j driver’s source code:

def check_supported_server_product(agent):
    """ Checks that a server product is supported by the driver by
    looking at the server agent string.
    :param agent: server agent string to check for validity
    :raises UnsupportedServerProduct: if the product is not supported
    """
    if not agent.startswith("Neo4j/"):
        raise UnsupportedServerProduct(agent)

What would happen if we simply disable this check? Let’s find out.

Tweaking the Neo4j Driver to Work with Memgraph

First, let’s clone the Neo4j driver’s repo:

git clone https://github.com/neo4j/neo4j-python-driver.git

A quick search shows that there are two places where the server product check is done:

There are two equivalent check_supported_server_product() functions in _neo4j/_async/io/_common.py and _neo4j/_sync/io/_common.py.

We can disable the validation by replacing the implementation of each function with just pass:

def check_supported_server_product(agent):
    pass

Next, we build this modified version of the Neo4j driver as follows:

python3 setup.py sdist

This creates a file called neo4j-5.2.dev0.tar.gz in a dist subfolder. Take note of the path of this file.

Back in the folder with our Python test code (where we were attempting to communicate with Memgraph), install the package we just built:

$ pip3 install /home/daniel/Desktop/neo4j-python-driver/dist/neo4j-5.2.dev0.tar.gz
Processing /home/daniel/Desktop/neo4j-python-driver/dist/neo4j-5.2.dev0.tar.gz
Requirement already satisfied: pytz in /home/daniel/.local/lib/python3.8/site-packages (from neo4j==5.2.dev0) (2022.1)
Building wheels for collected packages: neo4j
  Building wheel for neo4j (setup.py) ... done
  Created wheel for neo4j: filename=neo4j-5.2.dev0-py3-none-any.whl size=244857 sha256=ec2951ea1fecf2ae1aacced4d93c66b1b5d90bc3710746ff3814b9b62a96a9af
  Stored in directory: /home/daniel/.cache/pip/wheels/0d/4c/55/2486d65ebf98105bc54a490ebd91cea4ba538268a32ffc91f0
Successfully built neo4j
Installing collected packages: neo4j
  Attempting uninstall: neo4j
    Found existing installation: neo4j 5.2.1
    Uninstalling neo4j-5.2.1:
      Successfully uninstalled neo4j-5.2.1
Successfully installed neo4j-5.2.dev0

Run the Python code again…

$ python3 main.py
Unable to retrieve routing information
Transaction failed and will be retried in 0.9256931081701124s (Unable to retrieve routing information)
Unable to retrieve routing information
Transaction failed and will be retried in 2.0779915720272504s (Unable to retrieve routing information)

We still have a failure, but this is a simple connectivity issue that is easily fixed by changing the scheme in the URI from neo4j to bolt:

driver = GraphDatabase.driver("bolt://localhost:7687",
                              auth=("neo4j", "password"),)

Running it again, we see that it now works!

$ python3 main.py
Guinevere
Lancelot
Merlin

We can also view the created data in Memgraph Lab to double-check that it really worked:

Querying the data in Memgraph Lab, we see that the example nodes were created.

Conclusion

In this article, we’ve confirmed that, at a basic level, the only thing preventing the Neo4j Bolt Driver for Python from being used with Memgraph is a simple check against a response from the server. We saw that queries could be executed once this check was disabled.

As a result, it’s not clear why Memgraph built their own Python clients instead of simply addressing this check (e.g. by sending the same response as Neo4j, or forking the driver and eliminating the check as I did). I will refrain from speculating on possible reasons, but I found this interesting to investigate and hope it saves time for other people in the same situation.

P.S.: There’s an Easier Way

This section was added on 21st November 2022.

I learned from the Memgraph team that they do provide a way to deal with the server check – it’s just not documented at the time of writing this article. Basically, all you have to do is run Memgraph using a --bolt-server-name-for-init switch that sets the missing server response. So if you run Memgraph in Docker, you’d need to run it as follows:

sudo docker run --rm -it -p 7687:7687 -p 3000:3000 -e MEMGRAPH="--bolt-server-name-for-init=Neo4j/" memgraph/memgraph-platform

If you run the example code with the bolt:// scheme using the unmodified Neo4j Bolt Driver for Python, it should work just as well.

Update 19th September 2023: as of Memgraph v2.11, --bolt-server-name-for-init has a default value compatible with the Neo4j Bolt Driver, and therefore no longer needs to be provided.

From .NET to GoLang: Here We Go Again

Although I’ve been working with Go for less than a year, there are a number of things I’ve learned or noticed since I wrote “From .NET to GoLang: Where Did Everything Go?“, my critique of the Go programming language and ecosystem, three and a half months ago.

As a result, I’d like to use this followup article to discuss some new things that bother me as well as revisit points I already mentioned in the original article.

This is also the 300th article published at Gigi Labs. Hooray!

Productivity

I ended “From .NET to GoLang: Where Did Everything Go?” on a somewhat negative note:

“Coming from C#, learning my way around Go has been a fun diversion, but also something of a disappointment. Given the simplistic, verbose and sometimes confusing language, and the limited standard library, there’s no reason I can think of to choose Go for a new project over something richer and more mature like C#, which nowadays runs on any platform.”

As a result, it is somewhat surprising that I actually feel a lot more productive with Go than I ever did with C#, despite C# being the superior language in my opinion. I think the main reason for this is that .NET software tends to be extremely overcomplicated, owing to a combination of excessive OOP, layers of abstraction, design patterns, and obscenely exploited third party libraries (see “Pitfalls of AutoMapper“) that overzealous software engineers anxious to prove their mettle tend to sprinkle all over the place. On the other hand, Go’s simplicity – the very thing I was complaining about – does not provide many opportunities for this kind of madness.

Stability

I also mentioned that Go’s development seems to be somewhat slow, with particular reference to generics having taken 12 years to appear in the language. But this is also in a way a good thing. Go is a very stable language and upgrading to newer versions is a breeze. I can’t say the same about C# any more, given that every year it’s become a hassle to keep up with the latest unnecessary language ‘features’ as well as breaking changes to ASP .NET Core.

Exception Handling

I talked about Go’s lack of exception handling and the need to have heaps of if statements to check errors returned by function calls.

Go does in fact have a mechanism similar to exception handling, in the form of panic and recover. But since there are no exception objects, this mechanism operates with basic types, just like how the error interface returns a string and does not otherwise carry structured error information. This is unfortunate because it means it’s impossible for Go to have first chance exceptions, which make the debugger pause as soon as an exception is thrown – an essential troubleshooting tool that I sorely miss from .NET.

Defer

The same blog post I linked about panic and recover also talks about defer, which is the closest thing Go has to .NET’s using block (to clean up resources that need disposing). In “Scope Bound Resource Management in C#“, I discuss many of the things you can do by using and abusing the cleanup mechanisms in C++ and C#, and you can do most if not all of the same things in Go with defer.

Exported Names

Go code is organised in packages (like namespaces), but it doesn’t have public or private accessors. Variables and functions can be shared between packages if they start with a capital letter.

Wait, what? Let’s see this in action. I made a really simple program with two packages. I call a function from the money package from main():

Make() can be called from main() because it starts with a capital letter.

But if we change the name to start with a lowercase letter, it’s no longer visible:

make(), on the other hand, can only be used by code inside the money package.

I’m not at all a fan of such magic conventions, since they are not obvious at all to newcomers (unlike a keyword such as public would be), in the same way that I’m also not a fan of convention over configuration (used for a long time in ASP .NET Web API, MVC and Core) and content before chrome (used by the disastrous Windows 8 “Metro” theme).

LINQ and Sets

In “From .NET to GoLang: Where Did Everything Go?“, I complained about the standard library’s lack of support for common data structures such as sets, as well as the lack of anything like LINQ, making Go rely a lot on loops and elementary data structures.

However, in both cases, the community has stepped in and filled the gaps. I recently wrote the article “GoLang Set Data Structure“, which describes how to use the third party golang-set library. There is also a go-linq library on GitHub which I haven’t tried yet. Although it’s likely very useful, it’s somewhat unfortunate that Go’s syntax makes its usage a lot less elegant than in C#.

Slices

Go has arrays and slices (the latter being an array-like data structure). You can get a sub-range (also called “slice”) of an array, slice or string just like you can in Python:

package main

import "fmt"

func main() {
	numbers := []int{1, 2, 3, 4, 5, 6}
	jonathan := "Jonathan"

	fmt.Println(numbers[1:3]) // [2 3]
	fmt.Println(jonathan[:3]) // Jon
}

Go, however, doesn’t allow negative indices:

package main

import "fmt"

func main() {
	numbers := []int{1, 2, 3, 4, 5, 6}

	fmt.Println(numbers[1:-3]) // compiler error
}

Whereas with Python you can use negative indices to denote an index starting from the end (handy when you want to e.g. trim off the last character of a string):

>>> numbers = [1, 2, 3, 4, 5, 6]
>>> numbers[1:-3]
[2, 3]

Achieving the same thing with Go is a little more verbose, which is not very surprising at this point:

package main

import "fmt"

func main() {
	numbers := []int{1, 2, 3, 4, 5, 6}

	fmt.Println(numbers[1 : len(numbers)-3]) // [2 3]
}

Else Braces

I did mention in the original article that Go is a little fussy about braces used with an if statement, both in terms of their necessity and placement. Where it gets really annoying, though, is when it fails to build because you put the else on the next line, like this:

package main

import "fmt"

func main() {
	if ("sky" == "blue") {
		fmt.Println("Not in Ireland")
	}
	else { // compiler error
		fmt.Println("Probably Ireland")
	}
}

The pedantic bastard requires you to put the else on the same line as the preceding brace, like this:

package main

import "fmt"

func main() {
	if "sky" == "blue" {
		fmt.Println("Not in Ireland")
	} else { // compiler is happy
		fmt.Println("Probably Ireland")
	}
}

Whatever makes you happy, dear Go compiler.

Syntax (Revisited)

I wrote at length in the original article about how I find Go’s variable declaration syntax to be very confusing at best. There’s an article on the Go blog explaining the reasoning behind Go’s Declaration Syntax, which is informative but frankly not very convincing.

I also wrote about my aversion to the := syntax. My feelings are not just a matter of style but also come from having been screwed a couple of times by this fancy operator. Let’s see a simple if somewhat absurd example:

package main

import "fmt"

func uptown() (string, error) {
	return "Uptown Funk!", nil
}

func main() {
	lyrics := "Eye of the Tiger"

	if "sky" != "fuchsia" {
		lyrics, err := uptown()
		if err != nil {
			// nobody cares, Bruno
		}
		fmt.Println(lyrics)
	}

	fmt.Println(lyrics)
}

What do you think the output will be? OK, I’ll tell you. It’s:

Uptown Funk!
Eye of the Tiger

I used the := operator because the err variable is new and needs declaring. But because of that same operator, the lyrics variable inside the if statement is an entirely new variable that shadows the one outside. So once we’re back outside the if statement, thinking we updated the value of the lyrics variable, we find that it still has its original value.

I’m not even sure what is the best way to deal with this situation, though an easy way to update the outer lyrics variable is to use an intermediate variable to capture the output of uptown() and then assign that to lyrics:

package main

import "fmt"

func uptown() (string, error) {
	return "Uptown Funk!", nil
}

func main() {
	lyrics := "Eye of the Tiger"

	if "sky" != "fuchsia" {
		innerLyrics, err := uptown()
		if err != nil {
			// nobody cares, Bruno
		}
		fmt.Println(innerLyrics)
		lyrics = innerLyrics
	}

	fmt.Println(lyrics)
}

Addressable Values

Let’s say I have a map whose value is a struct. For instance:

package main

type Person struct {
	Hobby string
}

func main() {
	people := map[string]Person{"James": {Hobby: "Drinking"}}

	people[i].Hobby = "Farting"
}

I create a map with a single value which maps the key “James” to a struct, and then I try to update a property on that struct. Seems simple enough? And yet, it doesn’t compile:

You cannot simply assign a value to a struct’s property in a map in Go. Sorry.

What?! Why? It seems like this boils down to this concept of Addressable values in Go, WHICH FRANKLY I DON’T CARE ABOUT BECAUSE NO OTHER LANGUAGE EVER HAD A PROBLEM UPDATING A PROPERTY LIKE THIS!

To update that value, as far as I can tell, you instead have to take the detour of replacing the entire object in the map:

func main() {
	people := map[string]Person{"James": {Hobby: "Drinking"}}

	updatedJames := Person{Hobby: "Drinking"}
	people["James"] = updatedJames
}

Dates & Times

In “GoLang Timestamps: A Cross-Platform Nightmare“, I wrote about a very specific problem I encountered with Go time. But why don’t we talk about something more common? Parsing timestamps.

Let’s try to parse a date/time in a way that any reasonable language would allow, i.e. using placeholders like “yyyy-MM-dd“:

You cannot simply parse a date in an intuitive manner either.

No, Go doesn’t use such placeholders. Why would it? Instead, it uses this crazy mnemonic device to represent the various parts of the date:

Mon Jan 2 15:04:05 MST 2006

If you look closely, you’ll notice that these are actually the numbers 1 through 7:

  • Jan (1) is the month
  • 2 is the day
  • 15 (3) is the hour
  • 4 is the minutes
  • 5 is the seconds
  • 2006 (6) is the year
  • MST (-7) is the time zone relative to UTC

So, the equivalent of parsing as “yyyy-MM-dd” would be something like:

You have to use the weird mnemonic thingy. But you have to chop off the extra text too.

Oops, that didn’t work either. I have to chop off the extra bits for it to work:

package main

import (
	"fmt"
	"time"
)

func main() {
	fmt.Println(time.Now()) // 2022-11-16 22:23:39.354436468 +0100 CET m=+0.000014317
	parsedTime, err := time.Parse("2006-01-02", "2022-11-16")
	if err != nil {
		fmt.Println("Failed to parse time: ", err)
	} else {
		fmt.Println("Parsed time: ", parsedTime)
	}
}
Finally. That was a little bit complicated for such a basic operation.

Fortunately, there are a few predefined layouts you can use, such as time.RFC3339. But the ubiquitous RFC 8601 is nowhere to be found, so you’ll likely have to beg Stack Overflow for that.

Conclusion

It’s certainly been educational, entertaining and occasionally frustrating to learn about Go’s quirks. And considering I haven’t been working with it that long, I’m sure there will be lots more to come!

Surprising Swiss Trivia

In my previous article, “Surviving in Canton Zurich“, I wrote a lot about essential things you need to know if you want to live in Switzerland, or at least specifically in Canton Zurich. Here, on the other hand, I’ll be covering some little details that I wasn’t expecting to find in Switzerland, and which I consider fun and interesting to know.

Romansh

Switzerland is known mainly for three languages: German, French and Italian. However, there is a fourth official language, Romansh, a minority language spoken in the Canton of the Grisons.

Place Names

One of the views from Rapperswil.

Speaking of languages, names of many Swiss German places don’t seem to use umlauts (e.g. ä, ö, ü) or ess-tsets (ß), and instead use their simpler equivalents (ae, oe, ue and ss). This is evident in examples such as Uetliberg, Oerlikon, Bahnhofstrasse (or any other street name), but is not a general rule (Hardbrücke being a notable counter-example).

Another interesting pattern is the number of places ending in -ikon (e.g. Oerlikon, Dietlikon, Pfaffikon, Wiedikon to name a few) and -wil (e.g. Thalwil, Oetwil am See, Wädenswil, etc). The former seems to be a contraction of “-inghofen”, which roughly means “the people of the farm of”. -wil, on the other hand, means “hamlet” or “village”.

The similarities between names of towns and villages aren’t restricted to suffixes. There are a lot of names that are confusingly similar, such as Dietikon and Dietlikon, or Uerikon and Uetikon. You’ll also find the same names in different cantons, such as Egg (Zurich and Schwyz) or Pfäffikon (Zurich and Schwyz), in which case the abbreviation of the canton usually follows (e.g. Pfäffikon ZH). It seems like the Swiss aren’t very creative when it comes to naming things, although from an English-speaking perspective, Switzerland has its fair share of odd names (Rapperswil anyone?).

V-Zug

Switzerland is known for many things: banks, watches, trains, chocolate and more. Not as well-known among visitors is V-Zug, a brand of modern high-quality household appliances. V-Zug makes everything from dishwashers to tumble dryers, and it enjoys a very good reputation. When an apartment is advertised as having V-Zug appliances, it basically means that they’re the best on the market.

Village Life

Cows relax in the Swiss countryside.

Many Swiss villages are extremely well-equipped. You can live in a village of less than ten thousand people and have lots of amenities (e.g. multiple supermarkets, a school, a sports complex, a church, etc), live in a modern apartment with lots of high-tech appliances and Gigabit internet, and have great transport connectivity that can take you to the nearest city in 20-30 minutes (thanks also to the small size of Swiss cities).

Splitting Trains

Speaking of transport, you already know that Switzerland is known for its trains among other things. But did you know that some of the trains split in two? If you chance upon one of them, you had better board on the right half if you want to reach your destination on time!

Shops Close on Sundays

The Swiss take their work/life balance very seriously, so availability of shops and services is limited over the weekend. In the Zurich area, all shops (including supermarkets) close on Sundays except at the Zurich Hauptbahnhof (central station). Most offices and services, including banks, are also closed on Saturdays. So if you need someone to fix something in your apartment, no matter the emergency, you might have to wait until Monday.

Tree on the Roof

A tree on the roof of a newly built house.

If you see a tree on top of the roof of a newly built house, that’s a Swiss tradition. The builders put the tree up there to celebrate completion of the project and to indicate that the place is now fit for people to live in.