Tag Archives: Python

Using the Neo4j Bolt Driver for Python with Memgraph

Memgraph is a relatively young graph database. It supports the Cypher query language and the Bolt protocol – just like Neo4j – therefore it is usually possible to use Neo4j client libraries (called “drivers”) with Memgraph. In fact, according to the Memgraph Drivers documentation, using the Neo4j drivers is the recommended way to communicate with Memgraph from several languages like Go, Java and C#.

Memgraph Python Client Libraries

For Python, there are actually a number of options to choose from:

  • pymgclient: I discovered this recently and haven’t used it, but it seems to be a lower-level client that works well.
  • gqlalchemy: Memgraph’s recommended client for Python, which uses pymgclient underneath. Perhaps named after Python ORM sqlalchemy, it provides three different ways to query Memgraph, none of which I found to be very practical:
    • Basic execute() and execute_and_fetch(): this seems simple enough, but I haven’t found any way to pass parameters to queries, making it useless for my use case.
    • OGM: This is a graph equivalent of an ORM. It’s no secret that ORMs are one of the things I avoid like the plague – I’ve already written some of my thoughts on the subject in “ADO .NET Part 1: Introduction“, and time-permitting it will also be the subject of a future article. In a nutshell: I just want to write Cypher queries and execute them, not have to translate them to some library’s arbitrary API.
    • Query builder: A fluent query builder, similar in approach to what Elasticsearch provides for .NET. I’m not a fan for the same reasons that apply to ORMs (see previous point above).
  • Neo4j Bolt Driver for Python: This doesn’t work with Memgraph out of the box, but we’ll talk more about this.

It’s unfortunate that the Neo4j Bolt Driver for Python doesn’t work with Memgraph by default, because if you already have Python code that works with Neo4j, you could otherwise use Memgraph as a drop-in replacement for Neo4j with minimal changes (e.g. fixing incompatible Cypher).

For the rest of this article, I will be focusing on the Neo4j Bolt Driver for Python, to understand why we can’t use it with Memgraph and explain how to get around the problem.

Update 21st November 2022: TL;DR: if you need a quick solution, go to the end of this article.

Why the Neo4j Driver Fails with Memgraph

Let’s make a first attempt to use the Neo4j Bolt Driver for Python with Memgraph.

First, we need to have an instance of Memgraph running. The easiest way is to run it with Docker, e.g. as follows (assuming you’re on Linux):

sudo docker run --rm -it -p 7687:7687 -p 3000:3000 memgraph/memgraph-platform

If this works, it will start a Memgraph shell, and you can also access Memgraph Lab (Memgraph’s web user interface) by visiting http://localhost:3000/.

Memgraph shell after running it with Docker, and Memgraph Lab (UI) open in Firefox in the background.

Next, create a folder for your Python code. Run the following to install the Neo4j driver:

pip3 install neo4j

At the time of writing this article, the version of the Neo4j Python driver is 5.2.1. With earlier versions, it’s possible you might run into errors such as:

neobolt.exceptions.SecurityError: Failed to establish secure connection to ‘[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1131)’

In this case, update the driver as follows:

pip3 install neo4j --upgrade

At this point, we can steal some example code from the Neo4j Bolt Driver for Python, as follows, and put it in a file called main.py:

from neo4j import GraphDatabase

driver = GraphDatabase.driver("neo4j://localhost:7687",
                              auth=("neo4j", "password"))

def add_friend(tx, name, friend_name):
    tx.run("MERGE (a:Person {name: $name}) "
           "MERGE (a)-[:KNOWS]->(friend:Person {name: $friend_name})",
           name=name, friend_name=friend_name)

def print_friends(tx, name):
    query = ("MATCH (a:Person)-[:KNOWS]->(friend) WHERE a.name = $name "
             "RETURN friend.name ORDER BY friend.name")
    for record in tx.run(query, name=name):
        print(record["friend.name"])

with driver.session(database="neo4j") as session:
    session.execute_write(add_friend, "Arthur", "Guinevere")
    session.execute_write(add_friend, "Arthur", "Lancelot")
    session.execute_write(add_friend, "Arthur", "Merlin")
    session.execute_read(print_friends, "Arthur")

driver.close()

Once we run this with python3 main.py, we get a nice big error:

$ python3 main.py
Traceback (most recent call last):
  File "main.py", line 18, in <module>
    session.execute_write(add_friend, "Arthur", "Guinevere")
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/work/session.py", line 712, in execute_write
    return self._run_transaction(
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/work/session.py", line 484, in _run_transaction
    self._open_transaction(
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/work/session.py", line 396, in _open_transaction
    self._connect(access_mode=access_mode)
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/work/session.py", line 123, in _connect
    super()._connect(access_mode, **access_kwargs)
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/work/workspace.py", line 198, in _connect
    self._connection = self._pool.acquire(**acquire_kwargs_)
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/io/_pool.py", line 778, in acquire
    self.ensure_routing_table_is_fresh(
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/io/_pool.py", line 721, in ensure_routing_table_is_fresh
    self.update_routing_table(
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/io/_pool.py", line 648, in update_routing_table
    if self._update_routing_table_from(
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/io/_pool.py", line 596, in _update_routing_table_from
    new_routing_table = self.fetch_routing_table(
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/io/_pool.py", line 534, in fetch_routing_table
    new_routing_info = self.fetch_routing_info(
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/io/_pool.py", line 504, in fetch_routing_info
    cx = self._acquire(address, deadline, None)
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/io/_pool.py", line 221, in _acquire
    return connection_creator()
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/io/_pool.py", line 138, in connection_creator
    connection = self.opener(
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/io/_pool.py", line 441, in opener
    return Bolt.open(
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/io/_bolt.py", line 377, in open
    connection.hello()
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/io/_bolt4.py", line 450, in hello
    check_supported_server_product(self.server_info.agent)
  File "/home/daniel/.local/lib/python3.8/site-packages/neo4j/_sync/io/_common.py", line 283, in check_supported_server_product
    raise UnsupportedServerProduct(agent)
neo4j.exceptions.UnsupportedServerProduct: None

The last three lines indicate that the problem seems to be a simple validation, which we can confirm by looking up the offending function in the Neo4j driver’s source code:

def check_supported_server_product(agent):
    """ Checks that a server product is supported by the driver by
    looking at the server agent string.
    :param agent: server agent string to check for validity
    :raises UnsupportedServerProduct: if the product is not supported
    """
    if not agent.startswith("Neo4j/"):
        raise UnsupportedServerProduct(agent)

What would happen if we simply disable this check? Let’s find out.

Tweaking the Neo4j Driver to Work with Memgraph

First, let’s clone the Neo4j driver’s repo:

git clone https://github.com/neo4j/neo4j-python-driver.git

A quick search shows that there are two places where the server product check is done:

There are two equivalent check_supported_server_product() functions in _neo4j/_async/io/_common.py and _neo4j/_sync/io/_common.py.

We can disable the validation by replacing the implementation of each function with just pass:

def check_supported_server_product(agent):
    pass

Next, we build this modified version of the Neo4j driver as follows:

python3 setup.py sdist

This creates a file called neo4j-5.2.dev0.tar.gz in a dist subfolder. Take note of the path of this file.

Back in the folder with our Python test code (where we were attempting to communicate with Memgraph), install the package we just built:

$ pip3 install /home/daniel/Desktop/neo4j-python-driver/dist/neo4j-5.2.dev0.tar.gz
Processing /home/daniel/Desktop/neo4j-python-driver/dist/neo4j-5.2.dev0.tar.gz
Requirement already satisfied: pytz in /home/daniel/.local/lib/python3.8/site-packages (from neo4j==5.2.dev0) (2022.1)
Building wheels for collected packages: neo4j
  Building wheel for neo4j (setup.py) ... done
  Created wheel for neo4j: filename=neo4j-5.2.dev0-py3-none-any.whl size=244857 sha256=ec2951ea1fecf2ae1aacced4d93c66b1b5d90bc3710746ff3814b9b62a96a9af
  Stored in directory: /home/daniel/.cache/pip/wheels/0d/4c/55/2486d65ebf98105bc54a490ebd91cea4ba538268a32ffc91f0
Successfully built neo4j
Installing collected packages: neo4j
  Attempting uninstall: neo4j
    Found existing installation: neo4j 5.2.1
    Uninstalling neo4j-5.2.1:
      Successfully uninstalled neo4j-5.2.1
Successfully installed neo4j-5.2.dev0

Run the Python code again…

$ python3 main.py
Unable to retrieve routing information
Transaction failed and will be retried in 0.9256931081701124s (Unable to retrieve routing information)
Unable to retrieve routing information
Transaction failed and will be retried in 2.0779915720272504s (Unable to retrieve routing information)

We still have a failure, but this is a simple connectivity issue that is easily fixed by changing the scheme in the URI from neo4j to bolt:

driver = GraphDatabase.driver("bolt://localhost:7687",
                              auth=("neo4j", "password"),)

Running it again, we see that it now works!

$ python3 main.py
Guinevere
Lancelot
Merlin

We can also view the created data in Memgraph Lab to double-check that it really worked:

Querying the data in Memgraph Lab, we see that the example nodes were created.

Conclusion

In this article, we’ve confirmed that, at a basic level, the only thing preventing the Neo4j Bolt Driver for Python from being used with Memgraph is a simple check against a response from the server. We saw that queries could be executed once this check was disabled.

As a result, it’s not clear why Memgraph built their own Python clients instead of simply addressing this check (e.g. by sending the same response as Neo4j, or forking the driver and eliminating the check as I did). I will refrain from speculating on possible reasons, but I found this interesting to investigate and hope it saves time for other people in the same situation.

P.S.: There’s an Easier Way

This section was added on 21st November 2022.

I learned from the Memgraph team that they do provide a way to deal with the server check – it’s just not documented at the time of writing this article. Basically, all you have to do is run Memgraph using a --bolt-server-name-for-init switch that sets the missing server response. So if you run Memgraph in Docker, you’d need to run it as follows:

sudo docker run --rm -it -p 7687:7687 -p 3000:3000 -e MEMGRAPH="--bolt-server-name-for-init=Neo4j/" memgraph/memgraph-platform

If you run the example code with the bolt:// scheme using the unmodified Neo4j Bolt Driver for Python, it should work just as well.

Update 19th September 2023: as of Memgraph v2.11, --bolt-server-name-for-init has a default value compatible with the Neo4j Bolt Driver, and therefore no longer needs to be provided.

Calculating Circle Area the Monte Carlo Way

Ten years ago, I was in the middle of my MSc degree, titled “Distributed High-Fidelity Graphics Using P2P”. My research revolved around ray tracing – a family of techniques used to produce highly photorealistic images by simulating various aspects of Physics, especially the way light interacts with the various surfaces and materials it comes in contact with.

Ray tracing is very complex and computationally intensive, and it would take an entire book to write anything substantial about it. But, at the heart of modern ray tracing is an approach called Monte Carlo simulation, or stochastic simulation. It’s a fancy way of saying: this problem is too complex and I don’t have an easy way to calculate a solution, so I’ll take a number of random samples and approximate it instead.

In this article, I’ll illustrate the concept by calculating the area of a circle. Since we have an easy formula to calculate the area of a circle (πr2), there’s normally no need to do this via Monte Carlo simulation. However, it’s an easy way to explain the concept, which you can then use to solve more complex problems.

Approximating the Area of a Circle in Theory

Let’s assume that we don’t know how to calculate the area of a circle – either because a formula does not exist, or because it is too complex to derive.

However, we do have a way to determine whether a point lies inside a circle. If the point is represented by the coordinates (x, y), then the point lies inside the circle of radius r if (x2 + y2) <= r2, assuming that the circle’s centre is located at (0, 0).

A visualisation of a Monte Carlo approximation of a circle.

In this case, we can take a number of random points in space and compare how many fall inside and outside of the circle. You can see a simple visualisation of this above. Imagine we were throwing darts at a dartboard while blindfolded. The more darts we throw, the more we are filling in the circle, and the ratio of darts inside vs outside the dartboard tells us the overall area we covered.

It’s a little counterintuitive to imagine how a random process leads towards a deterministic solution. In fact, it’s not really deterministic, and the actual result changes with every run; but the approximation does work very well, and the more samples you take, the closer you get to the real solution. Let’s test this out in practice.

Approximating the Area of a Circle in Python 3.8

The code to implement the Monte Carlo approximation of the area of a circle is quite straightforward, as you can see below.

import random

def get_random_coord(radius):
    return random.uniform(-radius, radius)

def calculate_area_monte_carlo(radius):

    total_in_circle = 0

    for i in range(0, 10000000):
        (x, y) = get_random_coord(radius), get_random_coord(radius)
        in_circle = x ** 2 + y ** 2 <= radius ** 2
        
        if in_circle:
            total_in_circle += 1

        if i in [9999, 99999, 150000, 999999, 1500000, 9999999]:
            approximate_area = (2 * radius) ** 2 * total_in_circle / i
            print(f"Area ({i} samples): {approximate_area}")

Essentially, all I’m doing is:

  • Defining a helper function get_random_coord() to generate a random coordinate, just so that I don’t have to repeat the same code twice.
  • Generating an (x, y) coordinate pair within the square containing the circle.
  • Determining whether this (x, y) falls within the circle, using the aforementioned formula (x2 + y2) <= r2, in which case I increment total_in_circle.
  • At specific (arbitrary) intervals (numbers of samples), I calculate the approximate area of the circle and print it out to show how the calculated area gets more accurate, the more samples you take.

The approximate area is calculated as a ratio of samples in circle (total_in_circle) vs total samples both inside and outside of the circle (i). However, we have to adjust that by the area of a square, which is 2r2, as explained in this Stack Overflow answer.

In reality, we do know a formula to accurately calculate the area of a circle (πr2), so let’s toss that in as well in order to compare how accurate the results are:

import random
import math

def get_random_coord(radius):
    return random.uniform(-radius, radius)

def calculate_area_monte_carlo(radius):

    total_in_circle = 0

    for i in range(0, 10000000):
        (x, y) = get_random_coord(radius), get_random_coord(radius)
        in_circle = x ** 2 + y ** 2 <= radius ** 2
        
        if in_circle:
            total_in_circle += 1

        if i in [9999, 99999, 150000, 999999, 1500000, 9999999]:
            approximate_area = (2 * radius) ** 2 * total_in_circle / i
            print(f"Area ({i} samples): {approximate_area}")

def calculate_area_formula(radius):
    area = math.pi * radius ** 2
    print("Area (reference):", area)

def main():
    radius = 400
    calculate_area_monte_carlo(radius)
    calculate_area_formula(radius)

if __name__ == "__main__":
    main()

We can now run this program, and after a little patience, we get some results:

$ python3 main2.py
Area (9999 samples): 501874.1874187419
Area (99999 samples): 503793.8379383794
Area (150000 samples): 503940.26666666666
Area (999999 samples): 502511.2225112225
Area (1500000 samples): 502637.2266666667
Area (9999999 samples): 502743.410274341
Area (reference): 502654.8245743669

If we run it again, the results are a little different, but the trend is still there:

$ python3 main2.py
Area (9999 samples): 506930.69306930696
Area (99999 samples): 501861.0186101861
Area (150000 samples): 502101.3333333333
Area (999999 samples): 502941.30294130294
Area (1500000 samples): 502856.1066666667
Area (9999999 samples): 502779.44227794424
Area (reference): 502654.8245743669

This is the essence of the Monte Carlo approach: the more random samples you take, the more you’re filling in the problem space. However, this comes at a cost: the more samples you take, the more time it takes to compute. With enough samples, the error eventually becomes small enough that the approximation is indistinguishable from an accurate solution. Let’s visualise this.

Visualising the Area of a Circle Approximation

We can use the PIL library to quickly generate progressive images of the circle coverage. All we need is to add a few lines to our existing code:

import random
import math
from PIL import Image

def get_random_coord(radius):
    return random.uniform(-radius, radius)

def calculate_area_monte_carlo(radius):
    diameter = radius * 2
    image = Image.new("RGB", (diameter, diameter), (255, 255, 255))
    in_colour = (255, 0, 0) # red
    out_colour = (0, 0, 255) # blue

    total_in_circle = 0

    for i in range(0, 10000000):
        (x, y) = get_random_coord(radius), get_random_coord(radius)
        in_circle = x ** 2 + y ** 2 <= radius ** 2
        
        if in_circle:
            total_in_circle += 1
            colour = in_colour
        else:
            colour = out_colour

        image.putpixel((math.floor(x + radius), math.floor(y + radius)), colour)

        if i in [9999, 99999, 150000, 999999, 1500000, 9999999]:
            approximate_area = (2 * radius) ** 2 * total_in_circle / i
            print(f"Area ({i} samples): {approximate_area}")
            image.save(f'circle_{i}.png')

def calculate_area_formula(radius):
    area = math.pi * radius ** 2
    print("Area (reference):", area)

def main():
    radius = 400
    calculate_area_monte_carlo(radius)
    calculate_area_formula(radius)

if __name__ == "__main__":
    main()

What we’re doing here is:

  • Creating an image at the beginning.
  • For each sample, we colour the corresponding pixel either red (if it’s in the circle) or blue (if it’s outside the circle). Note that we have to add the radius to x and y because in the image, (0, 0) is not the centre of the circle, but the upper-left corner of the image.
  • Saving an image whenever we print an approximate area.

Earlier, we saw how the numbers seemed to be converging towards the accurate solution. If we now run the updated code above, we can verify the progressive coverage of the circle from actual images (click to enlarge):

Applications

As I said earlier, the Monte Carlo approach is not necessary for problems where we have a simple formula, like calculating the area of a circle. However, not all problems have a simple formula.

If you’ve studied calculus, you’ll know that you can integrate a function to calculate the area under its graph. You can do that for a simple function like 5x2, but there are more complex functions for which it is impossible to derive an analytical solution. Even if you never studied calculus, you can imagine how hard it must be to calculate the area of an irregular shape, such as your favourite politician. Randomly throwing darts at them and counting how many hit might be one way of working that out.

And if you ever work with ray tracing, you’ll eventually realise that ray tracing is really just another integration problem – one where you have to calculate the area (or contribution of light towards a surface) of something very irregular. A Monte Carlo approximation is a very effective – if computationally expensive – way of solving this otherwise intractable problem.

Retrieving Stock Prices using AWS Lambda

AWS Lambda functions are great for simple logic running periodically (among other things). In this article, we’ll create a simple AWS Lambda function in Python that retrieves stock prices from a REST API every minute. Let’s get straight to it!

Create a Lambda Function

First, we need to create a function. Follow the instructions illustrated below to do this.

From the AWS Console dashboard, locate the Lambda service. You can also do this via the Services drop-down panel at the top, or from your recently visited services (if you’ve already been using Lambda).
Once you are in the Lambda service, create a new function by clicking on the “Create a function” button as shown above.
Choose a name for the Lambda function, and also the runtime. We’re using Python 3.7 (which is the latest supported Python version in AWS Lambda at the time of writing this) for this example, but other options are available (e.g. Node.js, NET Core, Go, etc). Leave everything else as is and hit the Create function button.
Once the Lambda is created, you are taken to the new function itself. A green status message at the top indicates that it has been created successfully.

Editing the Lambda’s Code

The function’s configuration screen can seem quite confusing at first, but all you need to do is scroll down to get to the code editor. While there are a few different ways to add code to your Lambda, using the provided editor (which is the default option for Python) is the easiest.

Replace the default code in the editor with the following, and hit the Save button at the top-right.

from urllib.request import urlopen
from contextlib import closing
import json

def lambda_handler(event, context):
    with closing(urlopen("https://financialmodelingprep.com/api/v3/stock/real-time-price/GOOGL")) as responseData:
        jsonData = responseData.read()
        deserialisedData = json.loads(jsonData)
        price = deserialisedData['price']
        print(price);
    return price

Here we are simply retrieving Google’s stock price using the Financial Modeling Prep Stock Realtime Price API, which is open and doesn’t require any authentication.

Next to the Save button at the top-right, there’s a Test button. Click it, and the following screen comes up.

The Configure test event screen. Just enter a name and hit Create.

Just enter a name (e.g. “Test”) and hit the Create button further below. We’re not using the input JSON data, so you can just ignore it.

Next, click the Test button at the top-right again, and your Lambda function will be executed:

After clicking Test again, the lambda is executed and the results are shown below the code.

The results are shown below the code, and these include various metadata (such as a Request ID and execution time) as well as Google’s stock price of 1082.38, which we retrieved from the REST API and logged using the print statement in the code.

Running Periodically

We now have a working Lambda function, but so far we have to invoke it manually every time. Let’s set it up so that it runs every minute.

At the top of the screen, click CloudWatch Events on the left to add a CloudWatch trigger.

Scroll back to the top, and you’ll see a placeholder telling you to “Add triggers from the list on the left“. Following that advice, click on “CloudWatch Events” to the left.

A CloudWatch Event trigger is added to the function.

This has the effect of adding “CloudWatch Events” as a trigger in the slot where the placeholder text was, but what you might not notice at first is that the lower part of the page changes from the code editor to a “Configure triggers panel“. This can be quite confusing for those new to AWS Lambda who might not intuit right away that clicking on the boxes will affect the content in some other part of the page.

By scrolling down, we can configure the new trigger.

Here we use a Schedule expression of rate(1 minute) to make the function run every minute.

Filling in most of the settings (e.g. choosing a name) is easy, bearing in mind that there are some restrictions (e.g. some characters, such as spaces, are restricted in the name).

The only tricky part is where we specify how frequently we want the function to be executed. For this, we can use cron or rate expressions (refer to AWS documentation: Schedule Expressions Using Rate or Cron). By using an expression of rate(1 minute), we configure the function to run every minute, which is the smallest supported interval.

Once this is all set up, click the Add button to set up the trigger. Then, don’t forget to click the Save button at the top-right of the page to apply the changes to the Lambda function.

Checking Output in CloudWatch

After waiting a few minutes for the function to run a few times, we can go into CloudWatch and check the output of each execution.

CloudWatch logs.

From the AWS Services, locate CloudWatch. Go into Logs from the left menu, and locate the log group for our Lambda function (in this case it’s /aws/lambda/StockChecker).

Select the most recent log stream (the one at the top), and if you scroll to the end, you should see logs showing the function’s execution every minute, as well as whatever we’re writing to standard output (in this case, Google’s stock price).

CloudWatch logs show that the Lambda function is executing every minute.

We can see that the function is executing every minute, and we’re logging a stock price each time. The US stock market is closed right now, and that’s why the stock price is always the same (you’d expect it to change frequently when the market is active).

Conclusion

At this point, we have a simple, working AWS Lambda function (written in Python) that runs every minute and retrieves Google’s stock price. To keep things simple, we’re just writing it to standard output, which means we can see the value in CloudWatch – but we could also expand the code to build something useful from this.

Object Oriented Programming in Python

This article is a practical overview of Object Oriented Programming (OOP) in Python. It explains why OOP is useful, aside from how it’s done. This should be useful to both people who don’t know what OOP is, and experienced developers transitioning from other languages.

I am not a professional Python developer, and I am currently re-learning the language after not having used it for 8 years. So keep that in mind as you read, and feel free to offer feedback that can improve the quality of this article. Just be nice. 🙂

Due to the eternal divide between Python 2 and 3, I have to state that I’m using Python 3.6.4 here. Why Python 3? Because it makes no difference to me. When you are just learning and don’t have any requirements for maintaining backwards compatibility, you can afford to use the latest and greatest.

Introduction

In his hysterical rant on the web and OOP (in which he says the word “bizarre” enough times to qualify as a cover of OMC’s song), Zed Shaw cites OOP being “difficult to teach” as one of its major flaws.

Image credit: taken from here.

That’s a bold claim coming from someone who wrote in his own book:

“Search online for “object-oriented programming” and try to overflow your brain with what you read. Don’t worry if it makes absolutely no sense to you. Half of that stuff makes no sense to me either.” — Learn Python the Hard Way, Third Edition. Zed A. Shaw. 2014.

There are many things in computing that are hard to teach. I don’t think that Object Oriented Programming is one of them.

Motivation

In order to understand why OOP is useful, we’ll start off by not using it, and observe the problems we encounter. To do this, we need a proper example. People often teach OOP in terms of animals or cars, but I think games make more fun and interesting examples.

Screenshot from Dark Sun: Shattered Lands (1993)

A player-controlled character in a game typically has a number of attributes (e.g. name, hit points, etc). In order to group the attributes for our character, we need some kind of record or structure, which in C-style languages would be a struct. We don’t have that in Python, but we can use dictionaries instead.

talorus = {
    'name': 'Talorus',
    'hitpoints': 30,
    'dead': False,
    'inventory': []
}

Once we have a way to hold related data, we’ll want to perform some kind of operations on it.

def rename(character, newName):
    character['name'] = newName

def sufferDamage(character, damage):
    character['hitpoints'] -= damage
    if (character['hitpoints'] <= 0):
        character['dead'] = True

def receiveItem(character, item):
    character['inventory'].append(item)

Here’s some example usage:

You’ll notice a common theme across these functions. In all cases, we’re passing our character as the first parameter, and then using some of its attributes within the body of each function. We’re not using OOP yet, but we can already see a natural progression towards the character object being a first class citizen.

However, our current approach has a number of flaws. One of these is that it is easy for any code, anywhere, to tamper with our dictionary’s state.

Our logic from the sufferDamage() function specifies that characters die only if they run out of hitpoints, so how is our character dead with 26 hitpoints?

Being able to tamper with an object’s state without restriction is a bad thing: it is a violation of encapsulation, which is one of the three pillars of OOP (along with inheritance and polymorphism). We’ll discuss these later.

Classes and Objects

A class is just an abstract template for a type of object. For instance:

class Troll:
    pass

We’re declaring a Troll class, and using the pass keyword to indicate that there’s nothing in it for the time being. Once we have this class, then we can create concrete instances:

tom = Troll()
bert = Troll()
bill = Troll()

In Python, we create instances of a class (i.e. objects) by calling the class name as if it were a function.

An object may have any number of attributes (data members), just like the elements in a dictionary, but accessed using dot notation. Since Python is a dynamic language, it poses no restriction on the attributes that a class must have. We can add and remove attributes on the fly:

A class may define functions (called methods) that operate on an instance of the class:

class Character:

    def setName(self, newName):
        self.name = newName

This might look a bit weird, so let’s see some example usage and then discuss what we’re doing here:

The structure of the method might be familiar from the earlier section where we emulated OOP with dictionaries. In this case, we are similarly passing in the object itself as the first parameter, named self by convention. This extra parameter is required by Python. Through self, we can then access attributes of the class using dot notation.

What might look really strange here is that although setName() takes two parameters, we’re calling it with one. That’s because the self parameter is passed in implicitly when you call a method.

Constructors

A class may define a special method called __init__() which serves as the class’s constructor. It is usually used to initialise the object’s attributes, and may optionally take parameters which must be supplied when the object is instantiated:

class Character:

    def __init__(self, name, hitPoints):
        self.name = name
        self.hitPoints = hitPoints
        self.dead = False
        self.inventory = []

    def setName(self, newName):
        name = newName

Class-Level Variables

Screenshot from Ravenloft: Stone Prophet (1995)

A class may define variables within its scope:

class Monster:

    totalMonsters = 0

    def __init__(self, name, immortal):
        self.immortal = immortal
        Monster.totalMonsters += 1

Such class-level variables are not attributes of individual objects. They are shared across all instances of the class, just like static member variables in other languages. The distinction should be clear when you see that you access object attributes using self and class attributes using the name of the class itself. In this example, the shared totalMonsters counter is incremented every time a new monster is created:

Composition

Screenshot from Dark Sun: Shattered Lands (1993)

In the real world, complex objects are made up (composed) of other objects. The classic example is that a car has an engine (among other parts), but I prefer to stick to the game example. So let’s say we develop our inventory beyond a simple list, and make it into its own class:

class Inventory:

    def __init__(self):
        self.items = []

    def add(self, item):
        self.items.append(item)

    def has(self, item):
        return item in self.items

While this is a trivial implementation, it can be extended to support more complex operations.

We can now change our Character class to contain the new Inventory class:

class Character:

    def __init__(self, name, hitPoints):
        self.name = name
        self.hitPoints = hitPoints
        self.dead = False
        self.inventory = Inventory()

    def setName(self, newName):
        name = newName

Composition is used to model a has-a relationship (e.g. Character has an Inventory). As you can see, it’s nothing special. It’s merely a case of a class (e.g. Character) having an attribute whose type is also a class (e.g. Inventory).

Inheritance

Screenshot from Ultima 9: Ascension (1999)

A sword is a very common weapon in games. We can represent a simple sword by the following class:

class Sword:

    def __init__(self):
        self.damage = 10

    def attack(self, target):
        print('%d damage done to %s' % (self.damage, target))

Here’s an example usage:

However, there isn’t just one type of sword across all games in existence. Many games have magical swords with all sorts of positive (and negative) effects. One example is a fire sword. It does extra fire damage.

class FireSword:

    def __init__(self):
        self.damage = 10
        self.fireDamage = 5

    def attack(self, target):
        print('%d damage done to %s' % (self.damage, target))
        print('%d extra fire damage done to %s' % (self.fireDamage, target))

As you can see, there’s a lot of repetition here. If we also add classes for lightning swords, poison daggers etc, do we really want to duplicate this code and have to maintain it in several different places?

Fortunately, OOP allows us to create classes that inherit from others.

class FireSword (Sword):
    pass

The above code states that FireSword is-a Sword, and as a result, it inherits all of Sword‘s attributes and methods:

However, while we are reusing Sword‘s implementation for FireSword, we don’t yet have the extra functionality (i.e. extra fire damage) that makes it a fire sword, as we had in the original example. In order to do that, we must override Sword‘s methods to provide the extra functionality.

class FireSword (Sword):

    def __init__(self):
        super().__init__()
        self.fireDamage = 5

    def attack(self, target):
        super().attack(target)
        print('%d extra fire damage done to %s' % (self.fireDamage, target))

Here’s an example usage:

By calling super(), we’re calling the Sword class’s implementation before doing the extra logic specific to FireSword. In OOP terminology, Sword is the base class, parent class or superclass, and FireSword is the derived class or child class.

When you request an attribute or call a method on a derived class, Python will first look for an implementation in the derived class, and if it’s not there, it will look it up in the base class. This mechanism is what enables inheritance. However, it is also possible to have a method in the derived class to replace or extend the equivalent method in the base class, as we have seen above.

In other OOP languages, methods must usually be marked as virtual to allow them to be overridden. This is not necessary in Python.

“For C++ programmers: all methods in Python are effectively virtual.” — The Python Tutorial – Classes

Python allows a class to inherit from more than one base class. This is known as multiple inheritance, and is strongly discouraged because it makes classes extremely hard to work with. More modern OOP languages such as Java and C# expressly forbid multiple inheritance.

As a humorous aside, if you have a copy of Zed Shaw’s “Learn Python the Hard Way” book, you might want to read his section on “Inheritance vs Composition” for laughs. Shaw wastes almost a whole page with a silly story about a forest and an evil queen, which are supposed to be analogies for inheritance and multiple inheritance. His argument is that inheritance is bad because multiple inheritance is troublesome. That’s a bit like saying we should ban fire because some idiot got burned.

“In object-oriented programming, inheritance is the evil forest. Experienced programmers know to avoid this evil because they know that deep inside the dark forest of inheritance is the evil queen, multiple inheritance. She likes to eat software and programmers with her massive complexity teeth, chewing on the flesh of the fallen. But the forest is so powerful and so tempting that nearly every programmer has to go into it and try to make it out alive with the evil queen’s head before they can call themselves real programmers. You just can’t resist the inheritance forest’s pull, so you go in. After the adventure, you learn to just stay out of that stupid forest and bring an army if you are ever forced to go in again.” — Learn Python the Hard Way, Third Edition. Zed A. Shaw. 2014.

Shaw suggests that inheritance should be avoided, and composition should be used instead. For him, the choice between “inheritance versus composition comes down to an attempt to solve the problem of reusable code”. Unfortunately, he misses the point entirely. The main benefit of OOP is to model objects and their relationships. Inheritance models an is-a relationship, whereas composition models a has-a relationship. Code reuse is a practical benefit of both, but does not make them interchangeable.

Encapsulation

In the Motivation section towards the beginning of this article, we saw how emulating OOP with dictionaries results in a situation where the internal state of our classes can be tampered with. Let’s revisit that example, but with OOP:

class Character:

    def __init__(self, name, hitPoints):
        self.name = name
        self.hitPoints = hitPoints
        self.dead = False

    def sufferDamage(self, damage):
        self.hitPoints -= damage
        if (self.hitPoints <= 0):
            self.dead = True

Unfortunately, OOP in Python doesn’t do much to protect our internal state, and we can still tamper with it without restriction:

Other OOP languages usually have private, protected and public access modifiers to control access to internal data members of the class; these are enforced by the language. There is none of this in Python. The only thing you can do is follow a convention where private attributes are prefixed by an underscore, and hope that people play fair. It doesn’t stop people from accessing internal state.

Hiding the internal state of a class is called encapsulation. One strong reason why it is important is, as we’ve just seen, to ensure the consistency of that internal state (dead with 255 hit points? huh?). Another reason is to be able to modify the way that state works, without external code being affected.

So right now, we have an attribute called dead (or _dead, if we’re making it private by convention). Let’s add a method that exposes it:

class Character:

    def __init__(self, name, hitPoints):
        self._name = name
        self._hitPoints = hitPoints
        self._dead = False

    def sufferDamage(self, damage):
        self._hitPoints -= damage
        if (self._hitPoints <= 0):
            self._dead = True

    def isDead(self):
        return self._dead

Code external to this class may now check whether the character is dead by calling the isDead() method, and should not access _dead directly:

xanathar.isDead()

This extra method gives us a lot of flexibility because external code does not get to see how we store our internal state. We could, for instance, replace our _dead attribute with a computation based on _hitPoints, and the external code would never know the difference:

    def isDead(self):
        return self._hitPoints <= 0

So while in Python you can’t force external code not to touch a class’s internal state (as other OOP languages usually do), it is good practice to hide internal state using the available conventions, and expose only what needs to be exposed.

Polymorphism

Image credit: screenshot of Ultima 7: The Black Gate (1992) using Exult, taken from Let’s Play Archive entry

Typically, a person in a game can talk:

class Person:

    def Talk(self):
        print('Hello!')

Sometimes, though, an item can also talk.

class BlackSword:

    def Talk(self):
        print('Which of my powers dost thou seek to use?')

Animals, too, may surprise you with their gift of speech.

class SherryTheMouse:

    def Talk(self):
        print('Do you have any cheese?')

So here we have three completely unrelated classes, but they all have the same ability: we can call the Talk() method. When different objects exhibit similar behaviour, and thus we can work with them in a consistent manner, it’s called Polymorphism.

This is useful, for instance, when iterating over different kinds of objects in a loop:

This is unusual in the world of OOP, but since Python uses duck typing, it’s enough that two classes have the same method signature so that you can use them in the same way. In more strongly-typed OOP languages such as C# or Java, the classes would need to have something in common for you to do this (e.g. they implement the same interface, or they share a common base class).

Generics

This section is for developers coming from OOP in other languages. If you’re new to OOP, you may skip it.

Sometimes, you want to make a class have the same behaviour with different data types. For instance, you create a class representing a stack, and it should work the same regardless of whether it’s a stack of integers or of strings.

C++ provides this through templates, and C# and Java provide generics. These are a way to generalise the class implementation across dependent types, while still enforcing type safety.

Since Python is a dynamic language and it does not care what types you use, generics are not necessary. Your stack (or whatever) class will work just as will with integers, strings, or Animals (although I don’t recommend putting elephants at the top of the stack).

Summary

In this article, we’ve covered the basics of OOP in Python.

  • Even if you’re not currently doing OOP, you’ll notice that groups of variables and functions will tend to relate to the same entity. There is a natural tendency towards OOP.
  • Classes are groups of attributes and functions (methods). They provide a template but are not concrete.
  • Objects are concrete instances of classes. Person is a class. Joe is an object.
  • A constructor allows you to initialise attributes and pass in any parameters at instantiation time.
  • Class-level variables are shared across all instances of that class.
  • Composition is when a class contains other classes. It expresses a has-a relationship.
  • Inheritance expresses an is-a relationship. If FireSword is-a Sword, then FireSword inherits all of Sword’s attributes and methods, and may override those methods to provide more specialised variants.
  • Encapsulation is hiding internal attributes of a class so that external code can’t change them, and so that internal code can be changed without affecting external code. This is not enforced by the language but is upheld by convention.
  • Polymorphism is when different objects behave in a similar way. In Python, it works as a result of duck typing.
  • Generics aren’t necessary in a language without type safety.

This material includes basic concepts and language syntax, but is merely a starting point.

Mastering OOP is a matter of understanding that it is all about abstraction, and learning to work with abstractions in a way that is beneficial to software (e.g. models problem domains, maximises code reuse, reduces coupling, increases maintainability etc). The three pillars of OOP (inheritance, encapsulation and polymorphism) are basic building blocks of such abstraction. Various design patterns have emerged which demonstrate OOP abstractions put to good use.