I’ve recently written about how to use Cartography to collect infrastructural data from both AWS and Okta into a Neo4j graph database for security analysis.
Neo4j is a long-standing player in the graph database market, with a robust product, great documentation, and a massive following. However, its long legacy is in a way also a disadvantage, as it can be costly, slow, and resource-hungry (due in no small part to its reliance on the JVM). Sometimes people would like to use an alternative for any of these reasons.
Memgraph, on the other hand, is a relatively young graph database, and certainly not as fully-featured as Neo4j. A key difference is that it is written in C++, meaning it’s designed to be faster and more lightweight than Neo4j (whether it lives up to this is something you’ll need to evaluate for your own use cases). Memgraph also made a very wise decision to support the Bolt protocol and the Cypher language – both of which Neo4j uses – meaning that it’s compatible with existing Neo4j clients and queries. Although there are variations in Cypher dialect, the incompatibilities are few, and moving from Neo4j to Memgraph is significantly less painful than, say, transitioning to a graph database that uses Gremlin as its query language.
At the time of writing this article, Cartography requires Neo4j 4.x, and does not work with Memgraph. However, I’m going to show you how to make at least part of it (the Okta intel module) work with minor alterations to the Cartography codebase. This serves as a demonstration of how to get started migrating an existing application from Neo4j to Memgraph.
Running Memgraph
Before we start looking at Cartography, let’s run an instance of Memgraph. To do this, we’ll take a tip from my earlier article, “Using the Neo4j Bolt Driver for Python with Memgraph“, and run it under Docker as follows (drop the sudo if you’re on Mac or Windows):
sudo docker run --rm -it -p 7687:7687 -p 3000:3000 -e MEMGRAPH="--bolt-server-name-for-init=Neo4j/" memgraph/memgraph-platform
That --bolt-server-name-for-init=Neo4j/ is a first critical step in Neo4j compatibility. As explained in that same article, the Neo4j Bolt Driver (i.e. client) for Python (which Cartography uses) checks whether the server sends an “agent” value that starts with “Neo4j/”. By setting this, Memgraph is effectively posing as a Neo4j server, and the Neo4j Bolt Driver for Python can’t tell the difference.
Update 19th September 2023: as of Memgraph v2.11, --bolt-server-name-for-init has a default value compatible with the Neo4j Bolt Driver, and therefore no longer needs to be provided.
If it’s successful, you should see output such as the following:
Cloning the Cartography Repo
The next thing to do is grab a copy of the Cartography source code from the Cartography GitHub repo:
git clone https://github.com/lyft/cartography.git
Next, run the following command to install the necessary dependencies:
pip3 install -e .
Note: in the past, I’ve usually had to upgrade the Neo4j Bolt Driver for Python to 5.2.1 to get anything working, but as I try this again, it seems to work even with the default 4.4.x that Cartography uses. If you have problems, try changing setup.py to require neo4j>=5.2.1 and run the above command again.
Creating a Launch Configuration in Visual Studio Code
In order to run Cartography from its source code, you could run it directly from the terminal, for instance:
cd cartography/cartography
python3 __main__.py
However, as I’ve recently been using Visual Studio Code for all my polyglot software development needs, I find it much more convenient to set up a launch configuration that allows me to easily debug Cartography and pass whatever command-line arguments and environment variables I want.
The following launch.json is handy to run Cartography with an Okta configuration as described in “Getting Started with Cartography for Okta“:
{
"version": "0.2.0",
"configurations": [
{
"name": "Run Cartography",
"type": "python",
"request": "launch",
"program": "cartography/__main__.py",
"console": "integratedTerminal",
"justMyCode": true,
"args": [
"--neo4j-user",
"ignore",
"--neo4j-password-env-var",
"NEO4J_PASS",
"--okta-org-id",
"dev-xxxxxxxx",
"--okta-api-key-env-var",
"OKTA_API_TOKEN"
],
"env": {
"NEO4J_PASS": "ignore",
"OKTA_API_TOKEN": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
}
}
]
}
You might notice we’re telling Cartography to connect to Neo4j (Memgraph actually) with username and password both set to “ignore”. The reason for this is that while the Community Edition of Memgraph does not require (or support) authentication, the Neo4j Bolt Driver for Python (i.e. Neo4j client) does require a username and password to be provided. So, as a second critical compatibility step, we pass any arbitrary value for the Neo4j username and password so long as they are not left empty.
As for the Okta configuration, remember to replace the Organisation ID and API Token with real ones.
Incompatible Index Creation Cypher
Pressing F5, we can now run Cartography from inside Visual Studio Code, and we immediately run into the first problem:
Memgraph is choking on the index creation step in indexes.cypher (in VS Code, use Ctrl+P / Command+P to quickly locate the file) because the index creation syntax is one aspect of Memgraph’s Cypher implementation that is not compatible with that of Neo4j. If we take the first line in the file, the Neo4j-compatible syntax is:
CREATE INDEX IF NOT EXISTS FOR (n:AWSConfigurationRecorder) ON (n.id);
…whereas the equivalent on Memgraph would be:
CREATE INDEX ON :AWSConfigurationRecorder(id);
Note: in Memgraph, the “IF NOT EXISTS” bit is implicit: an index is created if it doesn’t exist; if it does, the operation is a no-op that does not cause any error.
Fortunately, this syntactic difference is easily resolved by replacing (using VS Code search & replace syntax, with regex enabled) this:
CREATE INDEX IF NOT EXISTS FOR \(n:(.*?)\) ON \(n.(.*?)\);
…with this:
CREATE INDEX ON :$1($2);
Tip: although not in scope here, you’ll need to make a similar change also in querybuilder.py and tx.py if you also want to get other intel modules (e.g. AWS) working.
Neo4j Result Consumption
After fixing the index creation syntax and rerunning Cartography, we run into another problem:
I’m told that consume() is used to fix a problem in which Neo4j connections hang in situations where internal buffers fill up, although the Cartography team is re-evaluating whether this is necessary. In practice, I have seen that removing this doesn’t seem to cause problems with datasets I’ve tested with, although your mileage may vary. Let’s fix this problem by removing usage of consume() in statement.py.
First, we drop the .consume() at the end of line 76 inside the run() function:
def run(self, session: neo4j.Session) -> None:
"""
Run the statement. This will execute the query against the graph.
"""
if self.iterative:
self._run_iterative(session)
else:
session.write_transaction(self._run_noniterative)
logger.info(f"Completed {self.parent_job_name} statement #{self.parent_job_sequence_num}")
Then, in the _run_iterative() function, we remove the entire while loop (lines 120-128) except for line 121, which we de-indent:
# while True:
result: neo4j.Result = session.write_transaction(self._run_noniterative)
# Exit if we have finished processing all items
# if not result.consume().counters.contains_updates:
# # Ensure network buffers are cleared
# result.consume()
# break
# result.consume()
When we run it again, it should finish the run without problems and return control of the terminal with the prompt showing:
...
INFO:cartography.sync:Finishing sync stage 'duo'
INFO:cartography.sync:Starting sync stage 'analysis'
INFO:cartography.intel.analysis:Skipping analysis because no job path was provided.
INFO:cartography.sync:Finishing sync stage 'analysis'
INFO:cartography.sync:Finishing sync with update tag '1689401212'
daniel@andromeda:~/git/cartography$
Querying the Graph
The terminal we’re using to run Memgraph has the mgconsole client running (that’s the memgraph> prompt you see in the earlier screenshot), meaning we can try running queries directly there. For starters, we can try the ubiquitous “get everything” Cypher query:
memgraph> match (n) return n;
Note: if you get a “mg_raw_transport_send: Broken pipe”, just run the query again and it should reconnect.
This gives us some data back:
mgconsole.As you can see, this is not great to visualise results. Fortunately, Memgraph has its own web client (similar to Neo4j Browser) called Memgraph Lab, that you can access on http://localhost:3000/:
On the Quick Connect page, click the “Connect now” button. Then, switch to the “Query Execution” page using the left navigation sidebar, and you can run queries and view results more comfortably:
Unlike Neo4j Browser, Memgraph Lab does not return relationships by default when you run this query. If you want to see them as well, you can run this instead:
match (a)-[r]->(b)
return a, r, b
If the graph looks too cluttered, just drag the nodes around to rearrange them in a way that is more pleasant.
More Cartography with Memgraph
Cartography is a huge project that gathers data from a variety of data sources including AWS, Azure, GitHub, Okta, and others.
I’ve intentionally only covered the Okta intel module in this article because it’s small in scope and easy to digest. To use Cartography with other data sources, additional effort is required to address other problems with incompatible Cypher queries. For instance, at the time of writing this article, there are at least 9 outstanding issues that need to be fixed before Cartography can be used with Memgraph for AWS (that’s quite impressive considering that the AWS intel module is the biggest). Other intel modules may have other problems that need solving; nobody has explored them with Memgraph yet.
Summary
In this article, I’ve shown how one could go about taking an existing application that depends on Neo4j and migrating it to Memgraph. I’ve used Cartography with its Okta intel module to keep things relatively straightforward. The steps involved include:
- Running Memgraph with
--bolt-server-name-for-init=Neo4j/ - Using the same Bolt-compatible Neo4j client, providing arbitrary Neo4j username and password values
- Fixing any incompatible Neo4j client code (in this case,
consume()), if applicable - Adjusting any incompatible Cypher queries
