Cameron MacLeod

How to sort parent nodes before child nodes? - Topological sort

2023-08-06T00:00:00+01:00

Dependencies between things are common, but they can be tricky to manage in code. You might think of using a graph to model your system, but how can you ensure that thing A happens before thing B? This is where the idea of a topological sort comes in.

This article is aimed at programmers who want to use topological sorting. I won't explain the maths but I will give code examples. There are also links to help you find an implementation for your language. The examples will be in Python, but the concepts should apply generally.

Why would you need a topological sort?
What is a topological sort?
Cycles and topological sorts
How do you use a topological sort?
Multi-threading and TopologicalSorter
Conclusion

Why would you need a topological sort?#

A package manager is a piece of software that installs other software. For example, a user could ask a package manager to install a text editor. It would then be the package manager's job to know where to find the text editor and how to install it.

Software packages almost always have dependencies. To install a text editor, you might first need to install a UI library and that UI library might require a C++ compiler. This dependency graph would look like the following:

This example is simple, but in the general case each package might have many dependencies and packages may share dependencies. This quickly becomes a tangled mess, and making sure that dependencies are installed before the software that needs them becomes difficult.

You can phrase this problem more generally. Given a graph, with dependencies between parent nodes and child nodes, how do you produce a list of nodes such that parent nodes come before their child nodes?

You can sort nodes of a graph in this way using a topological sort.

What is a topological sort?#

A topological sort is a fancy name for:

Taking a graph that has parent and child nodes (a directed graph)
Making a list of nodes in that graph such that:
- Each parent node will always appear before its children
- Each child node will come after its parents in the list

So taking a graph like the following:

A topological sort might look like:

Package	Depends on
1. gcc	N/A
2. Network library	gcc
3. UI library	gcc
4. Python	gcc
5. Language server	gcc, Python
6. Text editor	Network library, UI library, Language server

In the above ordering, a package always comes after its dependencies.

In the case of a package manager, this would mean that topological sorting produces an installation order that ensures dependencies are always installed before the software that depends on them. This is great, because there will never be missing dependency errors!

Why is it called a topological sort?

hand-waving begins

There is a branch of maths called topology which deals with shape and structure. Graph theory originally came out of topology.

Dependencies give structure to a graph, and topology is structure. Sorting a graph by its dependencies, therefore, can be considered a topological sort.

There are a lot of Wikipedia pages with topology in their name in this area, so it feels like the name intuitively fits, but I can't give you a more concrete answer unfortunately. Further reading below!

hand-waving ends, back to the coding

Cycles and topological sorts#

You can't apply a topological sort to a graph with a cycle in it. A cycle is where two or more nodes depend on each other. For example, A depends on B, but B also depends on A. In this case you can't produce a topological sort, because both A, B and B, A have a child node before a parent node.

What can you do about cycles? Fundamentally, cycles cannot be topologically sorted. That said, there are a few things you could do depending on what problem you are solving:

Manually remove the cycle

If you are building a graph by hand, then the easiest way to fix a problem like this is to manually remove the cycle. This can be done by removing conflicting edges.

For example, in the A->B->A graph above, you could remove A's dependency on B, or B's dependency on A.

Throw an error to the user/caller of your code

If the graph is being passed from outside your code, then it might be acceptable to throw an error in case of a cycle.

Algorithmically remove cycles

The problem of removing cycles from a graph while keeping the maximum number of edges is called the feedback arc set. There are algorithms for the feedback arc set, but they are not for the faint of heart! Proceed with caution:

How do you use a topological sort?#

For those who just want Python code, I've compiled it in this gist.

The below examples are all in Python, however, here are some links for libraries in other languages. I have not used any of the below, but hopefully they can provide a starting point for your research.

JavaScript: toposort - npm
Java: TopologicalOrderIterator - JGraphT
Rust: petagraph::algo::toposort

Thankfully Python (from version 3.9) comes with topological sort in the standard library! The wonderful graphlib provides everything you need.

Going back to the package manager example from earlier, you could model a package and its dependencies with the below class:

from dataclasses import dataclass    
from typing import List    


@dataclass          
class Package:      
    name: str                   
    depends_on: List[str]

The above code defines a class Package that has two fields, name and depends_on. name is the name of the package and depends_on is a list of other package names that this package depends on. The fields have type hints on them for readability.

@dataclass is a really powerful decorator that creates a lot of boilerplate methods, such as __init__ for free. Thanks to @dataclass, you can use Package like so:

text_editor = Package("editor", ["ui_lib"])
ui_lib = Package("ui_lib", ["gcc"])
gcc = Package("gcc", [])

This will create three packages. A text editor which depends on a UI library which in turn depends on GCC:

After creating the packages, the next job is to create a TopologicalSorter. A TopologicalSorter has methods for topologically sorting a graph. The below code creates one and tells it about the packages:

ts = TopologicalSorter()
ts.add(text_editor.name, *text_editor.depends_on)
ts.add(ui_lib.name, *ui_lib.depends_on)
ts.add(gcc.name, *gcc.depends_on)

In the above snippet, the add method takes a node as the first argument, and its dependencies as the following arguments. The code uses an asterisk * to unpack the depends_on fields into separate arguments. If you've not seen this syntax before, check out the Python docs.

There are a couple of things to note in the above code snippet. Firstly, all arguments provided to add represent nodes and must be hashable. This means that strings and tuples are valid nodes, but lists are not.

Secondly, you may notice that the text_editor node was added before the nodes it depends on. This is because TopologicalSorter is very forgiving and allows you to add nodes in any order. From the docs:

If a node that has not been provided before is included among predecessors it will be automatically added to the graph with no predecessors of its own.

The next step is to call static_order on the TopologicalSorter to produce a topological sort of the package graph:

for node in ts.static_order():
    print(node)

This will print:

gcc
ui_lib
editor

Great! This prints packages in the correct installation order. GCC can be installed first because it has no dependencies, and installing it unblocks installing the UI library which, in turn, unblocks installing the text editor.

static_order() returns an iterator, which is why the code uses a for loop to get the resulting nodes. Iterators are usually more efficient than lists, but if you wanted the result in a list, you could do the following:

nodes = list(ts.static_order())

# nodes = ["gcc", "ui_lib", "editor"]

For a more complex example, imagine the text editor had more dependencies:

In the dependency graph above, there are a lot more packages, and some of them even share dependencies! Luckily, TopologicalSorter will handle them without a problem:

text_editor = Package("editor", ["ui_lib", "lang_server", "network_lib"])
lang_server = Package("lang_server", ["gcc", "python"])
network_lib = Package("network_lib", ["gcc"])
ui_lib = Package("ui_lib", ["gcc"])
python = Package("python", ["gcc"])
gcc = Package("gcc", [])

ts = TopologicalSorter()
ts.add(text_editor.name, *text_editor.depends_on)
... # more add()s here
ts.add(gcc.name, *gcc.depends_on)

for node in ts.static_order():
    print(node)

This will print:

gcc
network_lib
ui_lib
python
lang_server
editor

TopologicalSorter is a powerful tool indeed! However, what happens when you accidentally pass it a cycle? The below code example creates a cycle to demonstrate what happens:

text_editor = Package("editor", ["lang_server"])
lang_server = Package("lang_server", ["editor"])

ts = TopologicalSorter()
ts.add(text_editor.name, *text_editor.depends_on)
ts.add(lang_server.name, *lang_server.depends_on)

for node in ts.static_order(): # raises CycleError
    print(node)

When the code calls static_order, it raises a CycleError:

Traceback (most recent call last):
 File "/home/cameron/src/scratch/cycle-toposort.py", line 21, in <module>
   for node in ts.static_order():
 File "/usr/lib/python3.9/graphlib.py", line 242, in static_order
   self.prepare()
 File "/usr/lib/python3.9/graphlib.py", line 104, in prepare
   raise CycleError(f"nodes are in a cycle", cycle)
graphlib.CycleError: ('nodes are in a cycle', ['editor', 'lang_server', 'editor'])

Take a look at the section above on dealing with cycles for pointers on handling this error.

Multi-threading and TopologicalSorter#

Calling static_order, as in the examples above, produces a list that works well with single-threaded code. However, it doesn't tell you which nodes can be processed at the same time.

To take the package manager example, a naive implementation using static_order may end up installing a package at the same time as its dependency. This could cause problems.

A naive implementation using two threads and static_order would break on the simple dependencies example from earlier. It would try to install gcc at the same time as the UI library, which would fail as the UI library depends on gcc:

Processing multiple nodes at once is useful as it can speed up your code, but is incompatible with static_order(). You can still use graphlib in multi-threaded code, though. Below are steps from the docs to use TopologicalSorter in a parallel way:

Create an instance of the TopologicalSorter with an optional initial graph.

Add additional nodes to the graph.

Call prepare() on the graph.

While is_active() is True, iterate over the nodes returned by get_ready() and process them. Call done() on each node as it finishes processing.

You've already seen how to do steps 1 and 2, that's the same as in the earlier code. To recap, here's how to add nodes to the TopologicalSorter:

ts = TopologicalSorter()
ts.add(text_editor.name, *text_editor.depends_on)
...

After creating a TopologicalSorter and adding nodes to it, step 3 is to call prepare() on it. prepare() marks the graph as ready for processing and checks for cycles:

try:
  ts.prepare()
except graphlib.CycleError:
  # do something with the error
  ...

This is where a graphlib.CycleError may be raised, so the code uses a try/except to deal with the error. After calling prepare(), you can't add any more nodes to the graph.

Before writing the code to get packages from TopologicalSorter, there needs to be a way of simulating package installation. Since installing packages takes time, the code can simulate it with time.sleep:

import time
from queue import Queue

installed_packages_queue = Queue()

def install_package(package):
    print(f"* Installing package {package}...")
    time.sleep(1)
    installed_packages_queue.put(package)

The above code defines a function install_package that sleeps for a second and then puts the package that it just 'installed' onto a queue.

The queue is used to communicate to the TopologicalSorter that packages depending on the now installed package can be processed. A queue allows the code to wait for any package to be installed even if multiple threads are installing packages.

Step 4 is where things get a little more complicated as it introduces three new functions, is_active(), get_ready() and done(). Let's take a look at the code and then step through it to see what it does:

from concurrent.futures import ThreadPoolExecutor

...

with ThreadPoolExecutor() as executor:
    while ts.is_active():
        ready_packages = ts.get_ready()
        print(f"Ready to install {ready_packages}")
        executor.map(install_package, ready_packages)

        installed_package = installed_packages_queue.get()
        ts.done(installed_package)
        print(f"Installed package {installed_package}")

ThreadPoolExecutor is a convenient way to work with threads. You can submit jobs to a ThreadPoolExecutor and have them automatically run in different threads. It simplifies a lot of the boilerplate and clean-up that typically comes with threaded code.

The code creates a ThreadPoolExecutor which will get cleaned up automatically once execution leaves the with block. ThreadPoolExecutor is a context manager:

with ThreadPoolExecutor() as executor:
    ...

The code then enters into the loop that the graphlib docs described:

While is_active() is True, iterate over the nodes returned by get_ready() and process them. Call done() on each node as it finishes processing.

is_active() reports whether there are nodes left to process. Until all the nodes have been returned by get_ready(), is_active() will return True. The code uses is_active() to stop once all packages are installed.

get_ready() gets a tuple of nodes that can be processed right now. A node can be processed if all of its parents have been processed or if it has no parents:

...
while ts.is_active():
    ready_packages = ts.get_ready()
    print(f"Ready to install {ready_packages}")
    executor.map(install_package, ready_packages)

The above code snippet gets a list of currently ready packages and then submits them as jobs to the ThreadPoolExecutor using map(func, iterable). This creates a new install_package job for each package that get_ready returns. Note that map is non-blocking.

The code then blocks on a package being put onto installed_packages_queue. The .get() call will return after the first install_package returns:

...

installed_package = installed_packages_queue.get()
ts.done(installed_package)
print(f"Installed package {installed_package}")

done(package) marks a node as processed. This enables get_ready() to return any child nodes of the processed node. In the above code, done marks a package as installed, which allows get_ready() to return packages that depend on the installed package.

With all these pieces in place, here's the English version of the above code:

While there are packages left to install
1. Get the packages that can currently be installed
2. Start an installation thread for each package
3. Wait for a package to be installed by listening on installed_packages_queue
4. Mark that package as done in the TopologicalSorter
5. Go round again to see whether any more package can now be installed

It took me a little while to wrap my head around this example, so don't be afraid to re-read the above a couple of times! It's worth noting that if your use case works with single-threaded processing, then static_order is an easier option.

Running the code in the attached gist produces the following output:

Conclusion#

Topological sorting is a really powerful tool. It allows you to order nodes in a graph by their dependencies, making sure that parent nodes come before their children.

In this article, you learned:

What a topological sort is and what it's useful for
How to use graphlib.TopologicalSorter in the Python standard library
How to deal with cycles in graphs when topologically sorting them

All the code in this article is available in a GitHub Gist.

abracadabra: How does Shazam work?

2022-02-19T00:00:00+00:00

Your phone's ability to identify any song it listens to is pure technological magic. In this article, I'll show you how one of the most popular apps, Shazam, does it. The founders of Shazam released a paper in 2003 documenting how it works, and I have been working on an implementation of that paper, abracadabra.

Where the paper doesn't explain something, I will fill in the gaps with how abracadabra approaches it. I've also included links to the corresponding part of the abracadabra codebase in relevant sections so you can follow along in Python if you prefer.

The state of the art has moved on since this paper, and Shazam has probably evolved its algorithm. However, the core principles of audio identification systems haven't changed, and the accuracy you can obtain using the original Shazam method is impressive.

To get the most out of this article, you should understand:

Quick links

What is Shazam?
Why is song recognition hard anyway?
System overview
Calculating a spectrogram
- The Fourier transform
- Spectrograms
Fingerprinting
Matching
Conclusion
Enter abracadabra
Further reading

What is Shazam?#

Shazam is an app that identifies songs that are playing around you. You open the app while music is playing, and Shazam will record a few seconds of audio which it uses to search its database. Once it identifies the song that's playing, it will display the result on screen.

Shazam recognising a song

Before Shazam was an app, it was a phone number. To identify a song, you would ring up the number and hold your phone's microphone to the music. After 30 seconds, Shazam would hang up and then text you details on the song you were listening to. If you were using a mobile phone back in 2002, you'll understand that the quality of phone calls back then made this a challenging task!

Why is song recognition hard anyway?#

If you haven't done much signal processing before, it may not be obvious why this is a difficult problem to solve. To help give you an idea, take a look at the following audio:

The above graph shows what Chris Cornell's "Like a Stone" looks like when stored in a computer. Now take a look at the following section of the track:

If you wanted to tell whether this section of audio came from the track above, you could use a brute-force method. For example, you could slide the section of audio along the track and see if it matches at any point:

Matching a section of track by sliding it

This would be a bit slow, but it would work. Now imagine that you didn't know which track this audio came from, and you had a database of 10 million songs to search. This would take a lot longer!

What's worse, when you move from this toy example to samples that are recorded through a microphone you introduce background noise, frequency effects, amplitude changes and more. All of these can change the shape of the audio significantly. The sliding method just doesn't work that well for this problem.

Thankfully, Shazam's approach is a lot smarter than that. In the next section, you'll see the high-level overview of how this works.

System overview#

If Shazam doesn't take the sliding approach we described above, what does it do? Take a look at the following high-level diagram:

The first thing you will notice is that the diagram is split up into register and recognise flows. The register flow remembers a song to enable it to be recognised in the future. The recognise flow identifies a short section of audio.

Registering a song and identifying some audio share a lot of commonality. The following sections will go into more detail, but both flows have the following steps:

Calculate the spectrogram of the song/audio. This is a graph of frequency against time. We'll talk more about spectrograms later.
Find peaks in that spectrogram. These represent the loudest frequencies in the audio and will help us build a fingerprint.
Hash these peaks. In short, this means pairing peaks up to make a better fingerprint.

After calculating these hashes, the register flow will store them in the database. The recognise flow will compare them to hashes already in the database to identify which song is playing through the matching step.

In the next few sections, you'll learn more about each of these steps.

Calculating a spectrogram#

The first step for both flows is to obtain a spectrogram of the audio being registered or recognised. To understand spectrograms, you first have to understand Fourier transforms.

The Fourier transform#

A Fourier transform takes some audio and tells you which frequencies are present in that audio. For example, if you took a 20 Hertz sine wave and used the Fourier transform on it, you would see a big spike around 20 Hertz (Hz):

In the above image, you can see a large spike around 20Hz and nothing at other frequencies. Sine waves are often called pure tones because of this property, since they only contain a single frequency.

The result of a Fourier transform is called a frequency spectrum. We say that when you take the Fourier transform of a signal, you move it from the time domain into the frequency domain. These are fancy terms for describing whether time or frequency is along the bottom of a graph. In mathematical terms, the domain is more or less the X-axis of a graph.

The Y-axis of the frequency spectrum represents the strength of each frequency component. If a frequency component is stronger, then it will be more audible in the time-domain signal.

If you were to add a 50Hz sine wave at half the strength to that 20Hz sine wave, the resulting frequency spectrum would show a spike at 20Hz and a smaller spike at 50Hz:

As you can see, adding multiple audio waves together combines the frequencies present in them. In fact, all audio signals can be reconstructed from waves like this. For more, take a look at 3Blue1Brown's video on the Fourier transform.

One great property of the frequency domain is that it often helps us to see things that aren't clear in the time domain. For example, if you take the signal with two frequencies from before and add noise to it, in the time domain it looks visually very different. However, in the frequency domain, the two spikes are still very clear:

In the frequency domain graph on the right, you can still clearly see the spikes for the main component frequencies. It would be harder in the time domain to see what frequency sine waves went into the signal.

Up until now, our examples have only contained one or two frequencies, but what happens if you put a more complex signal through the Fourier transform? Let's take a look at our section of audio from Like a Stone:

Real audio files like the one above contain lots of different frequencies. This is a good thing, as it means that the frequencies present can uniquely identify songs.

Spectrograms#

abracadabra implementation

If you run a Fourier transform over an entire song, then you will see the strength of the frequencies present over the whole song. However, the frequencies that are present change over time. To better represent the frequencies changing over time, we need to split the song into small sections before taking the Fourier transform. This is called taking a spectrogram.

Here's a simplified animation of how spectrograms work:

Explanation of the spectrogram process

In the above animation, you can see that the song is first split into small sections. Next, we use the Fourier transform to calculate the frequency spectrum of each of these sections. When you put all these frequency spectrums together, you get a spectrogram.

To make this concrete, let's take a look at the spectrogram of Like a Stone:

Even though the spectrogram looks 2-dimensional, it's actually a 3D graph with the following axes:

Time (X-axis)
Frequency (Y-axis)
Strength (Z-axis/colour)

The Z-axis is represented by colour in the spectrogram above. Bright green shows a high magnitude for a particular frequency component and dark blue shows a low magnitude.

Looking at the spectrogram above, you can see that the brightest spots (strongest frequencies) almost exclusively occur below 5000Hz. This is quite common with music, for example most pianos have a frequency range of 27Hz-4186Hz.

The frequencies present in a track contain a lot of identifying information, and calculating the spectrogram allows us access to that information. In the next section, you'll learn how we turn all that information into a unique fingerprint for the track.

Fingerprinting#

Just as a fingerprint uniquely identifies a person, we can extract a unique fingerprint for some audio from its spectrogram.

These audio fingerprints rely on finding peaks in the spectrogram. These peaks are the loudest frequencies at some time in the song. Because they are loud, it's likely they'll survive when subjected to noise or other distortions.

In the next section, you'll read some more about the motivation behind using spectrogram peaks to build fingerprints.

Why is the fingerprint based on spectrogram peaks?#

A spectrogram peak is a frequency that is loud at some point in an audio signal. You can recognise these on a spectrogram since they will be the brightest points.

In music, these would represent the loudest notes. For example, during a guitar solo, the notes that the guitar is playing might become spectrogram peaks since they would likely be the loudest notes at that time.

A spectrogram peak is the point least likely to be affected by noise. Noise has to be louder than the spectrogram peak to make it unrecognisable and the spectrogram peaks are the loudest frequency components in the track.

To make this visual, take a look at our earlier example of a Fourier transformed signal that had noise added to it. When noise is added, the frequency peaks still retain their rough shape.

Another advantage of using spectrogram peaks to fingerprint audio is that they cut down the amount of data we have to store. Storing only the loudest frequency components means we don't have to store everything else. This speeds up searching fingerprints since there is less data to look through.

Before we can use frequency peaks in our fingerprint though, we have to find them. In the next section, you'll learn how.

Finding peaks#

abracadabra implementation

As discussed in the previous section, the peaks of a spectrogram represent the strongest frequencies in a signal. For frequency peaks to be usable in an audio fingerprint, it's important that they are evenly spaced through the spectrogram.

It's important the peaks are evenly spaced in time, so the system can recognise any section of the song. For example, if all the peaks were at the start of the song, then the fingerprint wouldn't cover later sections:

In the image above, all the peaks (white crosses) are clustered at the start of the song. This means that the system can't recognise any sample from the rest of the song.

It's also important that the peaks are evenly spaced in frequency, so the system can deal with noise and frequency distortion. Sometimes noise will be very loud and concentrated at a specific frequency range, for example a car horn in the background:

Peaks clustered in a frequency band affected by noise

In the above animation, the peaks are well-spaced in time, but are clustered into a small frequency band. When a loud noise is introduced, for example a car horn, it can make an entire section of song unrecognisable by changing which peaks are selected.

To find spectrogram peaks while keeping them well-spaced, we can borrow a technique from image processing known as a maximum filter. The process looks something like the following:

Use the maximum filter to highlight peaks in the spectrogram.
Locate the highlighted peaks by comparing to our original spectrogram.
(Optional) Discard some peaks.

Let's run through these steps one-by-one. First, let's take a look at how the maximum filter works:

Step 1: Maximum filter

A maximum filter emphasises the peaks in an image. It does this by looking in a neighbourhood around each pixel for the maximum value and then setting the pixel to that local maximum. The following animation shows a maximum filter that looks at a 3x3 neighbourhood around each pixel:

Animation of a maximum filter on a simple image

As you can see in the above animation, the maximum filter takes each pixel of an image in turn and finds the maximum in a region surrounding it. The filtered pixel is then set to that local maximum. This has the effect of expanding each local peak to its surrounding area.

Running a maximum filter on Like a Stone's spectrogram gives the following result:

The maximum-filtered spectrogram looks like a lower-resolution version of the original spectrogram. This is because the peaks in the signal have expanded and taken over the other pixels. Each box with the same colour in the filtered image corresponds to a local peak in the original image.

The maximum filter has a parameter that controls the size of the box to use when finding the local maxima. When you set this parameter to make a smaller box, you end up getting more peaks. Similarly, by setting this parameter larger you get fewer peaks.

Step 2: Recover original peaks

The maximum filter doesn't do all the work for us. The filter has emphasised the local peaks, but it hasn't found their locations. To find the peak locations, we need to find the points that have equal values in the original spectrogram and the filtered spectrogram.

The idea behind this trick is that all the non-peak points in the spectrogram have been replaced by their local peaks, so their values have changed. The only points whose values haven't changed are the peaks.

Below is a zoomed in section of the spectrogram above. The points where the values are equal in the filtered and original spectrograms are highlighted:

As you can see in the images above, the highlighted points where the two spectrograms are equal correspond to the local peaks of that part of the image.

Plotting all of the peaks together produces something called a constellation map. Here's the constellation map for Like a Stone:

These graphs are called constellation maps since they look a bit like an image of the night sky. Who said computer science couldn't be romantic?

Step 3: (Optional) Discard peaks

Once we have a constellation map of peaks, the next step is to potentially discard some. The size of our fingerprint is dependent on the number of peaks that we use in it. Keeping fingerprints small matters when you are storing millions of songs in your database.

However, by reducing the number of peaks we use, we reduce the accuracy of our system. Fewer peaks in a fingerprint mean fewer chances to match a sample to the correct song.

There are a couple of options for reducing the number of peaks in our fingerprint:

Take the top N peaks. N should be proportional to the length of audio that you are fingerprinting to avoid over-representing shorter songs.
Take all peaks above a certain threshold. This doesn't guarantee you a certain fingerprint size per time like the other method, but may give more accurate results.

We have almost finished constructing our fingerprint, the next step is to produce a set of hashes from our peaks.

Hashing#

abracadabra implementation

To motivate hashing, imagine that our fingerprint was just a collection of spectrogram peaks. Each peak's frequency would be represented by a certain number of bits, for example 10. With 10 bits of information, we can represent 2^10=1024 individual frequencies. With thousands of these points per track, we quickly get a lot of repeats.

Uniqueness is important for a fingerprint, since it makes searching a lot faster and helps to recognise more songs. Shazam's solution to the problem of uniqueness is to create hashes from pairs of peaks:

The diagram above shows a zoomed in portion of a spectrogram. Each circle represents a peak and the dashed line box represents a hash. You can see that a hash is formed of two peaks. The information that is recorded for each hash is the frequency of each peak, f_A and f_B, and the time delta between them, 𝚫T.

The advantage of pairing points up is that two paired points are much more unique than a single point. Looking at it mathematically, if each point has 10 bits of frequency information, and the time delta between the two points could be represented by 10 bits, then you have 30 bits of information. 2^30=1073741824 which is significantly larger than 1024 possibilities for a single point.

Shazam creates pairs using the following algorithm:

Pick a point. This will be called the anchor point.
Calculate a target zone of the spectrogram for the anchor point.
For every point in the target zone, create a pair with the anchor point.

You can see this algorithm illustrated in the following animation:

Animation of pairing points

Choosing a target zone isn't described in the Shazam paper, but the images the paper contains show it as starting slightly ahead of time of the anchor point and being centred on the anchor point's frequency.

Once a pair has been created, it is stored as a hash in the database with the following information:

			Other information
Point A freq (f_A)	Point B freq (f_B)	Time delta (𝚫T)	Point A time	Track ID

The first three columns (f_A, f_B and 𝚫T) make up the hash. The "Other information" is used to locate the hash at a specific time in a song. This will be used in matching later.

All of the hashes for a particular track make up the fingerprint. In the next section, you'll read about how Shazam matches these fingerprints.

Matching#

Given a collection of fingerprints in a database, how does Shazam figure out which one a given audio sample matches? This is where the matching part of the system comes in. Recall the system diagram from earlier:

The recognise and register flows both generate fingerprints. The difference lies in what they do with them. While the register flow stores fingerprints away for future matching, the recognise flow has to match its fingerprint with what is already in the database.

The matching algorithm contains the following steps:

Retrieve all hashes from the database that match the sample's fingerprint.
Group these hashes by song.
For each song, figure out if the hashes line up.
Choose the track with the most lined up hashes.

We'll look at each of these steps in turn.

Step 1: Retrieve matching hashes

abracadabra implementation

The first step is to find every hash in the database that matches a hash in the fingerprint we just created. Even though a hash is a 3-tuple of (f_A, f_B, 𝚫T), abracadabra stores this as hash(f_A, f_B, 𝚫T) where hash() is a hash function that returns a single value. This way you only have to search for a single value per hash instead of three.

Step 2: Group hashes by song

Recall the format of an individual hash in the database:

			Other information
Point A freq (f_A)	Point B freq (f_B)	Time delta (𝚫T)	Point A time	Track ID

Thanks to the track ID that we associated with each hash, we can group the hashes by track. This allows us to score each potentially matching track.

Step 3: Figure out if hashes line up

abracadabra implementation

If a sample matches a song, then the hashes present in that sample should line up nicely with the hashes in some section of that song. The diagram below illustrates what this would look like:

In the above diagram, a sample has been lined up with the section of the original song that it came from. The blue points represent the anchor points of the hashes.

While the above diagram shows the perfect scenario, there is a chance that the matching hashes from the database don't line up perfectly. For example, noise could have introduced peaks in the sample that resemble peaks at a different point in the song. This can lead to the following scenario:

In the above diagram, the red circles represent hashes that match to points in the song outside the section the sample came from. In this situation, it's harder to see that the sample is a perfect match for the song.

What's worse, sometimes hashes can match to the wrong song! This is where checking that the hashes line up comes in.

To explain how we can check whether the hashes line up in code, let's look at an example. Let's imagine that we've got a list of matching hashes from the database and grouped them by track. For a given track, we can then check the time that the hash occurs in the original track against the time that the hash occurs in the sample.

Sample time	Track time	Track time - Sample time
3	13	10
4	14	10
7	20	13
5	15	10
6	12	6
1	11	10

In the above table, you can see that all the matches with a Track time - Sample time of 10 have been highlighted. These are the true matches, while the other two rows are false matches. To see this is the case, let's look at a similar diagram to the ones we saw before:

The above diagram contains the same hashes from the previous table. As you can see, the true matches have a Track time - Sample time that is equal to how far into the track time that the sample starts.

To see how we turn this into a score for the track, let's make this data into a histogram. A histogram is a fancy name for a bar chart. We're going to plot each Track time - Sample time against the number of times it occurs:

Each bar in the histogram above is referred to as a bin. To score a song on how good a match it is for an audio sample, we just need to take the largest bin. Songs that aren't good matches will have low values in all bins, whereas a song that's a good match will have a large spike in one of the bins.

This way we can compare a sample to all the songs with matching hashes in our database and score each of them. The song with the highest score is likely to be the correct result.

You might wonder why we don't just go for the song that matches the largest number of hashes as it would be much simpler to implement. The problem with this approach is that not all songs are the same length. Longer songs are likely to get more matches than shorter songs and when some Spotify tracks are over 4 hours long this can really bias your results!

Conclusion#

Well done for making it this far, that was a long journey! Over the course of this article, you've learned how Shazam extracts fingerprints from audio, and how it matches these fingerprints to those that it has already registered in its database.

To summarise, Shazam does the following to register a song:

Calculates a spectrogram of a song
Extracts peaks from that spectrogram
Pairs those peaks up into hashes
Stores the collection of hashes for a song as a fingerprint

Shazam does the following to recognise an audio sample:

Calculates a fingerprint of the audio sample
Finds the hashes that match that fingerprint in the database
For each potential song match:
- Calculate Track time - Sample time for each matching hash
- Group those values into a histogram
- Take the largest bin in this histogram as the score for the song
Return the song with the highest score

Enter abracadabra#

I learned everything written here over the process of writing abracadabra, my implementation of this paper.

If you are interested in seeing what this might look like in code, please take a look! Everything is open source and I've done my best to document the project. abracadabra can also be used as a library in other projects, so please feel free to re-mix and build something cool. If you do use it, I'd love to hear about it.

What a computer science degree looks like in 2020

2020-05-25T00:00:00+01:00

Lots of people who are learning to code ask the question "Should I get a CS degree or do a bootcamp?", or "Is a degree necessary?". I have just finished my degree (not graduated yet), and while I can't give all the answers to these questions, I can share my experience.

In this post I will go into detail about what my degree contained. You can use this information to decide whether university is for you, make your own curriculum, or even just see how the syllabus has changed since you went.

I don't want to tell you to get a degree, and I equally don't want to tell you to not get one. It is a personal choice. However, on the whole I'm very pleased that I went to university. I learned a lot of things that I wouldn't have otherwise, but more importantly, I met lots of great people and had experiences that I couldn't have had in any other setting. The piece of paper that I will get once I graduate is important for the job I chose, but it won't be important for everyone on my course.

Below you will find a list of every module I took in the course of my degree and their content. Not every degree will look like this, I had a lot of elective modules and some universities are more practical than others, but this should give you an idea of what university can offer.

The courses are broken down by year and semester. Each course has the following structure.

Name and link to full course description

Some information about the course, including searchable terms where appropriate and the main textbook used.

Number of credits to give you an idea of how much depth each course has.

This is a Bachelor of Science (Hons) degree from the University of Edinburgh. It is under the Scottish university system, so it is four years long. Each year is composed of two semesters and you need to take 120 credits of courses across these semesters.

Disclaimer: This is not a comprehensive guide to the degree program at Edinburgh, if you are looking for that, go to DRPS (Degree Regulations and Programmes of Study). I have tried not to include my opinions of the courses, since it's the syllabus that's important.

With that out the way, here's the full list.

Year 1

Year 1 had 40 credits free for optional modules. The number of free credits in first year may be higher in Scotland than elsewhere due to Scottish universities doing a four year bachelors degree.

Semester 1

Computation and Logic

Propositional and first-order logic
Finite state machines
Set theory

Textbook: Mathematical Methods in Linuguistics

10 credits

Functional Programming

Introductory course on Haskell. Gets as far as monads, but doesn't really address them that thoroughly.

Textbook: Learn you a Haskell for Great Good!

10 credits

Introduction to Linear Algebra

Complex numbers
Vectors
Systems of linear equations
Matrices
Eigenvalues/eigenvectors

Textbook: 'Linear Algebra, A Modern Introduction' by David Poole

20 credits

Semester 2

Object-oriented Programming

Honestly, I didn't really pay attention to this course. It taught introductory Java.

Textbook: 'The Java Tutorial: A Short Course on the Basics' - Raymond Gallardo et al.

10 credits

Data and Analysis

SQL/Basic databases
XML
Text corpora

Textbook: Unknown. The lecturer handed out notes for this course

10 credits

Calculus and its Applications

Introductory calculus course. It covered single variable integration, differentiation and series.

Textbook: 'Essential Calculus: Early Transcendentals' - James Stewart

20 credits

Full Year

Russian Studies (optional)

Russian course for beginners. One of the reasons that I went to university in Scotland was so that I could do languages alongside my degree. Scotland has more of a "liberal arts" system than the rest of the UK.

40 credits

Year 2

Year 2 only had 20 credits free. Bayes Theorem was a big theme in year 2.

Semester 1

Processing Formal and Natural Languages

Natural language processing
Hidden Markov Models
Grammars

Textbook: Speech and Language Processing, Jurafsky, D and J. H. Martin

20 credits

Introduction to Computer Systems

Instruction sets and assembly
C programming
Basic processor and digital logic design

Textbook: Computer Organisation and Design, D.A. Patterson and J.L. Hennessy

10 credits

Introduction to Software Engineering

Software lifecycle
UML
Design patterns
Intro to testing
Version control

Textbook: Software Engineering, Ian Sommerville

10 credits

Discrete Mathematics and Mathematical Reasoning

Number systems
Proofs by induction
Counting (combinatorics)
Graphs
Trees
Discrete probabilities

Textbook: Discrete Mathematics and its Applications, Kenneth Rosen

20 credits

Semester 2

Algorithms, Data Structures, Learning

This was actually two courses combined. Firstly the algorithms and data structures component contained:

Asymptotic notation and algorithms (Big O etc)
Data structures
Sorting algorithms
Graphs

Textbook: Data Structures and Algorithms in Java, Goodrich, M T and Tamassia

The other half of the course was focused on machine learning.

Dimensionality in data (PCA etc)
Naive Bayes
Clustering algorithms
Gaussians
Neural networks

Textbook: A First Course in Machine Learning - second edition, Simon Rogers and Mark Girolami

20 credits

Probability with Applications

Combinatorics
Bayes theorem
Discrete random variable distributions
Continuous random variable distributions
Jointly distributed random variables
Covariance and correlation
Discrete Markov chains
Birth and death processes

Textbook: A First Course in Probability, Sheldon Ross

20 credits

Reasoning and Agents (optional)

Intelligent agents
Problem solving through search
Adversarial search
Knowledge bases and inference
First order logic
Planning
Bayes Theorem
Markov decision processes

Textbook: Artificial Intelligence: A Modern Approach, Russell R & Norvig P

20 credits

Signals and Communications Systems (optional)

Types of signals (continuous vs discrete, periodic vs aperiodic etc)
Power and energy
Fourier analysis
Convolution
Nyquist's sampling theorem
Modulation techniques (OOK, FSK, PSK)
Multiplexing (FDM and TDM)
Information theory

Textbook: Digital Signal Processing, John G. Proakis & Dimitris G. Manolakis

10 credits

Year 3

Year 3 was the first honours year, and so courses start to get more specific in comparison to the foundational courses of years 1 and 2. It had 50 free credits

Semester 1

Professional Issues

This course was focused on "soft" skills such as ethical, legal and financial issues for software development businesses.

Textbook: Professional Issues in Information Technology, Frank Bott

10 credits

Informatics Large Practical

This was a project course that required students to build an Android game. There was a specification, but some room for creativity. My code from this course is on GitHub

No textbook

20 credits

Computer Design (optional)

Course on digital logic and processor design.

Combinatorial logic
Sequential logic
Processor design
Memory system design
I/O controller design
Synchronisation issues
RISC

Textbook: 'Computer Organization', V. C. Hamacher, Z. G. Vranesic & S. G. Zaky

20 credits

Introduction to Vision and Robotics (optional)

Robotic actuators and sensors
Control theory
Image thresholding, filtering and classification
Active vision and attention

Textbook: Russell & Norvig Chapters 24 & 25 in Artificial Intelligence: A Modern Approach

10 credits

Semester 2

System Design Project

This was a group project to build a lego robot with the theme of "assistive technology". My team chose to build a chess-playing robot to enable disabled players to compete in tournaments more easily. See the Deeper Blue GitHub organisation for more details.

No textbook

20 credits

Computer Security

Network security
Usable security (the human factor)
Symmetric and asymmetric ciphers
MACs and hash functions
Digital signatures
SSL/TLS
TOR
OS Security
Web security

Textbook: Introduction to Computer Security, Michael Goodrich and Robert Tamassia Pearson

20 credits

Operating Systems (optional)

Process management
Resource allocation and deadlocks
The Kernel
Memory management (virtual memory, paging etc)
Scheduling
File management

Textbook: 'Operating Systems, Internals and Design Principles', W. Stallings

20 credits

Year 4

Year 4 really frees up with 80 credits for optional courses. Of course, most of these credits have to be spent in the area of computer science unlike years 1 and 2.

Semester 1

Computer Graphics (optional)

This was a coursework-only course which is unusual. Most courses have at least 60% of their mark come from an exam. There were three courseworks.

Compositing using Blender, PBRT and Gimp
Build a raytracer
Implement a new sampling method and material in PBRT

Textbook: 'Physically Based Rendering', Matt Pharr, Greg Humphreys & Wenzel Jakob

10 credits

Extreme Computing (optional)

Course on cloud computing and web-scale technologies.

Distributed file systems
Virtualisation
Fault tolerance
Hadoop/MapReduce

Textbook: Data Intensive Text Processing with MapReduce, Jimmy Linn & Chris Dyer

10 credits

Speech Processing (optional)

Speech signals (source-filter model)
Text processing (POS tagging, expanding abbreviations etc)
Pronunciation (letter-to-sound models, prosody prediction)
Waveform generation in speech synthesis
Speech recognition features
Hidden Markov models

Textbook: Speech and Language Processing, Jurafsky, D and J. H. Martin

20 credits

Semester 2

Computer Communications and Networks (optional)

The Internet layer model
UDP and TCP
IP
Routing algorithms
Link layer protocols
Software defined networking

Textbook: Computer Networking: A Top-Down Approach, J. F. Kurose and K. W. Ross

20 credits

Automatic Speech Recognition (optional)

Gaussian mixture models
EM algorithm
Hidden Markov models
Speech signal analysis and features
Context dependent modelling
Hybrid HMM/DNN systems
Large vocabulary ASR
TDNN and LSTM architectures
Speaker adaptation
Multi-lingual speech recognition
Speaker verification and diarization

Textbook: Speech and Language Processing, Jurafsky, D and J. H. Martin

10 credits

Software Testing (optional)

Unit testing
Combinatorial testing
Structural testing
Coverage
Mutation testing
TDD
Regression testing

Textbook: Software Testing and Analysis: Process, Principles and Techniques, Pezze and Young

10 credits

Full Year

Honours Project

This is the course that represents our undergraduate dissertation. At the end of third year, you have to produce a preference list of five topics from around 300 or you can propose your own. You then get assigned a topic and a supervisor and work on it through fourth year. Mine was on Kubernetes and mobile networks. You can read it here.

40 credits

Build a snake game on the BBC micro:bit

2020-05-18T00:00:00+01:00

By the end of this tutorial, you'll have built your very own game and learned not only about game development, but Python and the BBC micro:bit too. What's more, you don't even need to own a micro:bit to follow along!

Below is a preview of what we'll be building.

Snake game running on the BBC micro:bit

Pre-requisites#

This tutorial will assume that you know what Python is, and that you know some basic programming concepts. But don't worry, if you consider yourself a beginner then this tutorial is for you. In fact, I will try to over-explain things and provide solutions to every exercise, so there's nothing to fear!

With that out the way, let's get started.

Quick links

What is a micro:bit?#

A micro:bit is a tiny computer that you can program in Python. It comes with an LED grid (for a screen), an accelerometer (to detect motion), electrical outputs (for controlling motors, lights etc) and a bunch of other cool features. They're also pretty cheap (£20/$20 at time of writing on Amazon), and programming them is as simple as dragging a file to a USB stick.

If you don't already have one, don't worry! There are simulators available online and you'll be able to follow along with this whole tutorial.

Setting up environment#

Whether you have a micro:bit or not, the first step is to open the create.withcode editor. When you do, you should see a screen like the one below.

Most of the screen (1) is taken up with the editor. This is where you'll put your code. In the lower-right corner (2) is a button to run your code which looks like a green triangle. Alternatively, you can run your code by pressing Ctrl+Enter.

Your editor will auto-backup your code to your computer. You can access these backups by clicking the "+" button in the lower-right corner then clicking the button that looks like a clock.

Running your code will produce a pop-up that looks like the below.

Your code will be running on the virtual micro:bit you can see, and there are tabs along the top for your to control various inputs to the device, which will become important later.

If you are working with a physical micro:bit, then the "Download HEX" button in the top-left of the pop-up will allow you to download the file you need to program your micro:bit. You can drag this file to the "MICROBIT" device in your file manager to program it.

Before we start looking at some code, let's take a quick detour to look at the game of Snake.

Snake#

Fun fact - if you go to Google and search for "snake", a snake game should appear. Just in case it doesn't appear for you, you can play it here. Below you can see it in action.

Google Snake game demo

This is a good example of a snake game. It has all the necessary components:

A grid on which the snake moves.
- The grid has cells, which can be identified by their X and Y co-ordinates.
A snake.
- The snake occupies some cells of the grid (you can think of it as a list of co-ordinates).
- The snake has a direction (up, down, left, right).
Some randomly generated food on a cell.
An end condition.
- In the video this happens when the snake crashes into itself, losing the game.

So in theory, if we can build all these components, then we should have a snake game!

Skeleton code#

To make it easier to get started, we'll be working from a skeleton file. Find it at this link and copy the contents into the create.withcode editor.

Let's take a look at each part of the code to figure out what it does.

from microbit import *

class Snake:
...

At the very top of the file is an import. An import allows us to use code someone else has written in our own script. In this case, the other code we're using is the microbit library, which allows us to control various components of the micro:bit including the display and accelerometer.

Skipping the class definition for now, at the bottom of the file is our game loop.

while True:
    game.handle_input()
    game.update()
    game.draw()
    sleep(500)

The game loop is the entry point for the game (the first bit of code to be executed). It defines a series of actions that we do repeatedly in order to make the game work. In this case, we get some input from the user, update the game state accordingly and then re-draw the screen. Updating the screen twice a second feels about right, and so at the end we sleep for 500 milliseconds (half a second).

The functions called in the game loop are part of the Snake class, which is defined above the game loop.

class Snake:
    """ This class contains the functions that operate
        on our game as well as the state of the game.
        It's a handy way to link the two.
    """

    def __init__(self):
        ...

    def handle_input(self):
        ...

    def update(self):
        ...

    def draw(self):
        ...

game = Snake()

A class is a handy way of keeping some state, like the position of our snake, alongside the functions that operate on it. Our class has a few different functions that we'll have to write.

__init__(self): This is a special Python function that is called when an "instance" of a class is created. We'll see an example of creating an instance later on. The function will initialise our variables ready for the game to run. For example, we need an initial position for the snake and a direction it will first be heading.

You may notice that it takes an argument, self. This argument is a Python requirement for functions that are part of a class, and represents the instance of the class. This means we can use it to access the state that's part of the class.
handle_input(self): This function gets input from the user and uses it to change the direction of the snake. We'll be using the accelerometer for this which means that the user will be tilting the device to make the snake move. Again, this function takes self since Python requires it to.
update(self): Once we have the direction that the user wants the snake to move, we can update the game state. This means moving the snake, "eating" food and ending the game.
draw(self): The draw function allows us to see the game state on the screen. In this function we'll be lighting up LEDs to represent the snake and the food.

Below the Snake class definition, we create an instance of it and assign it to the variable game. If you're not yet comfortable with classes and instances, take a look at the expandable panel below.

Classes and instances

A class is a way of linking functions to variables, but you can also think of it as defining the 'type' of something. Let's use cars as an analogy:

class Car:
    def __init__(self, make, model, colour):
        self.make = make
        self.model = model
        self.colour = colour

    def print_info(self):
        print(f"This is a {self.colour} {self.make} {self.model}")

astra = Car("Vauxhall", "Astra", "white")
golf = Car("Volkswagen", "Golf", "blue")

astra.print_info()
# Prints "This is a white Vauxhall Astra"
golf.print_info()
# Prints "This is a blue Volkswagen Golf"

In this example, we've defined a `Car` _class_ that contains the make, model and colour of a car. The class represents the concept of a car, but note that it does not represent any actual cars, that's what instances are for. Below the class definition, we create two _instances_ of `Car`. One that represents an Astra, and one that represents a Golf. An instance represents a specific thing, instead of the concept of a thing. So our `astra` variable does not represent the idea of a car, but instead a specific white Vauxhall Astra. One thing you may notice about the class is that both of its functions take `self` as the first argument. This represents the instance of the class that the function is being called on. When we call `astra.print_info()`, Python passes the `astra` instance as the first argument to the `print_info()` function. One other thing that often confuses people with classes is the `__init__` method. It's a special method in Python that gets called when you create an instance of a class. When we create an instance, you can think of the following happening. 1. We call `astra = Car("Vauxhall", "Astra", "white")` 2. Python creates an empty `Car`, let's call it `temp_car`. 3. Python runs `astra = Car.__init__(temp_car, "Vauxhall", "Astra", "white")` 4. `astra` now has the correct variables and `temp_car` is discarded. For more discussion of classes, take a look a [Jeff Knupp's blog post](https://jeffknupp.com/blog/2014/06/18/improve-your-python-python-classes-and-object-oriented-programming/).

Now that we have a template, our first step will be to initialise the game state.

Creating the game state#

Before we can intialise the game state, we need to learn a little bit about Python and the micro:bit. Feel free to skip these sections if you have done this before.

Python lists#

Lists are a way to store multiple values in the same variable in Python. If you haven't come across them before there's a more detailed introduction here. You may be able to get by with the cheat sheet below though.

# define a list
>>> my_list = [1, 2, 3]
# access elements of a list (start counting at 0)
>>> my_list[0]
1
# -1 is the last element of the list
>>> my_list[-1]
3
# Add an item to a list
>>> my_list.append(4)
>>> my_list
[1, 2, 3, 4]
# Remove an item from a list
>>> my_list.pop(0)
1
>>> my_list
[2, 3, 4]
# list of lists
>>> list_of_coords = [[1, 2], [1, 3]]
>>> list_of_coords[0]
[1, 2]
>>> list_of_coords[0][1]
2

The micro:bit co-ordinate system#

On the front of a micro:bit is a grid of LEDs. When drawing something on the screen, we need a way to describe individual LEDs within this grid so that we know which one to light up. This is where the co-ordinate system comes in.

Display co-ordinates on the micro:bit work a little like the graphs you may have plotted in school. There's an X axis that goes from left to right and a Y axis that goes from top to bottom. The main difference from the graphs in school is that the Y axis increases going down (towards the pins) whereas in school it would have increased going up.

In the image above, point A is at the co-ordinates [3, 0]. The left number in these co-ordinates is the position along the X axis and the right number is the position along the Y axis. Similarly, point B is at co-ordinates [1, 2].

Your turn 1: Filling in the state#

If you now feel comfortable with these concepts, you'll notice that the __init__ function has some commented out code. When you uncomment that, it looks like the below. Feel free to have a go at completing it and we'll look at the solution in the next section.

# direction is a string with up, down, left or right
self.direction =
# snake is a list of the pixels that the snake is at
self.snake = [[2, 2]]
# food is the co-ords of the current food
self.food =
# whether or not to end the game (boolean)
self.end =

Solution

Let's talk through the solution line by line.

self.direction = "up"

We defined `self.direction` to be a string and I've chosen to make the snake go up at first. If you've put any other direction that's fine too.

self.food = [0, 2]

I've put my food in the middle of the left-most column to start with. If you've put it elsewhere, that's fine as long as you've defined valid co-ordinates that aren't the same as the snake's starting position.

self.end = False

This variable tells us whether the game has finished or not. It wouldn't be very helpful to set it to `True` before it's even started!

Drawing to the screen#

We have some initial state, but if you ran this code right now, you wouldn't see much happening. That's because we're not actually drawing anything to the screen.

So what do we need to draw on the screen? Thinking back to our Google Snake example, the snake, food and grid were visible. On the micro:bit the grid is already built for us since the LEDs form a grid naturally. That leaves the snake and the food to be drawn.

Drawing the food#

Ideally, then, we need a function to light up an individual LED using its co-ordinates. The snake and the food are both stored using co-ordinates, so if we can draw these on one at a time then we'll have completed our draw function.

The micro:bit offers us a number of different ways to draw to the screen, including one that does what we need. display.set_pixel(x, y, value) will set the pixel at [x, y] to the level of brightness value. value can be between 0 and 9, where 0 is off and 9 is fully on.

Now that we have a function to turn on a pixel, we can start to fill in our draw function.

def draw(self):
    display.clear()
    display.set_pixel(self.food[0], self.food[1], 5)
    # TODO: draw snake

The first thing we do in this function is display.clear(). This sets all the LEDs off, which is important because this draw function will be called repeatedly and we want to draw something different each time. If we didn't clear the display, then the game state from the last iteration of the game loop would still be there.

Next we use the self.food co-ordinates to draw the food. We use the intensity 5 because that will help us to distinguish it from the snake.

Before we draw the snake, let's take a look at for loops in Python. Again, if you've done this before, feel free to skip it.

Python `for` loops#

for loops are great for when you want to do the same thing to multiple elements. In Python they're written like this:

for variable_name in iterable_thing:
    do_things(variable_name)

iterable_thing is something with multiple elements, for example a list, tuple or even a range(start, stop). variable_name is the name that you assign to the element that you are currently working on in the loop. To make this example more concrete, here's how you would print the numbers from one to five.

>>> for i in range(1, 6):
>>>     print(i)
1
2
3
4
5
# alternatively
>>> nums = [1, 2, 3, 4, 5]
>>> for num in nums:
>>>     print(num)
1
2
3
4
5

Your turn 2: Drawing the snake#

Now you know how to light up individual pixels, and how to use a for loop to go over a list, you should be able to draw the snake to the screen. You'll want to draw the snake with intensity 9 so that it looks different from the food we drew earlier. As before, the solution is below.

Solution

def draw(self):
    display.clear()
    display.set_pixel(self.food[0], self.food[1], 5)
    for part in self.snake:
      display.set_pixel(part[0], part[1], 9)

Since `self.snake` is a list of co-ordinates, we can use it in a `for` loop to get every co-ordinate that's part of the snake. We then use the `set_pixel` function to draw each part to the screen.

Running our code#

At this point, we should see some output, so let's go ahead and run our code. Whether or not you have a physical micro:bit, press Ctrl+Enter in the editor to bring up the micro:bit pop up. You can also click the cross in the corner and then click the green play button.

If you want to run the code on your physical micro:bit, click the "Download HEX" button in the top left corner. You can refer to the Setting up environment section if you can't find it.

You should see something that looks like the below. If so, well done! If not, go back and check the solutions match your code exactly. Whitespace matters in Python as well, so check that carefully.

Our snake is in the centre of the screen as a bright dot, and our food is at the left of the screen as a less bright dot. Yours might be in different positions depending on what you chose in Creating the game state. The next step is to make the snake move.

Moving the snake#

To understand how to move the snake, let's take a look at an example.

Animation showing how the snake moves.

The animation shows a snake before and after moving down one cell. At first glance, moving the snake in this situation looks like it would be complicated, especially since it moves around a corner. Luckily for us though, it only consists of two actions.

Calculate the new head
Cut the tail off

So our first step will be to calculate the new head. We can do that with the help of if statements.

Python `if` statements#

Sometimes we want to choose what to do based on a condition. For example, you would want to wear a coat if it was raining outside, but if it was sunny you would put on sunglasses. You can do this in Python using if statements.

# rain is a boolean variable (True or False)
if rain:
    put_on_coat()
else:
    # it must be sunny
    put_on_sunglasses()
# we'll go outside whether it's raining or sunny
go_outside()

You can even choose between more than two options. Sticking with the weather example, maybe it could be windy, rainy, or sunny. Being the responsible person that you are, you have clothes for each different possibility.

# in this case, weather is a string
if weather == "rain":
    put_on_raincoat()
elif weather == "wind":
    put_on_jumper()
elif weather == "sunny":
    put_on_sunglasses()
go_outside()

Your turn 3: Calculating the new head#

We're going to be working in the update() function here, since that updates the game state before we draw the screen.

There are a few steps in calculating the new head of the snake.

Get current head of the snake (last item of the list)
Calculate a new head based on self.direction.
Add this head to the end of the list (.append(item) function)

The most difficult step here will be figuring out what to do for each direction. You might find this easier if you draw it out on a co-ordinate grid like the one in the micro:bit co-ordinate system section.

Important - If you create the new head from the old head (e.g. new_head = self.snake[-1]), you will need to copy it. Otherwise you will have a really annoying bug. You can copy a list like so:

# call list() to copy the old head
new_head = list(self.snake[-1])

What is the really annoying bug? (optional)

If you don't copy the head of the snake, then you will modify the existing head as well as creating a new one. Let's illustrate this with an example: Let's imagine a snake at `[[1, 1], [2, 1], [2, 2]]` where `[2, 2]` is the head of the snake. Also let's say that the snake is moving south. ![Snake at `[[1, 1], [2, 1], [2, 2]]`](/images/microbitsnake/bugex1.png) When we update the snake, we will take the head and update it to `[2, 3]` since it is heading downwards. Ordinarily this would mean the snake looks like this: ![Snake at `[[2, 1], [2, 2], [2, 3]]`](/images/microbitsnake/bugex2.png) However, since we didn't copy the head, we'll end up modifying the existing one **and** appending it, so our snake will be at `[[2, 1], [2, 3], [2, 3]]`: ![Snake at `[[2, 1], [2, 3], [2, 3]]`](/images/microbitsnake/bugex3.png) This is because lists in Python are _mutable_, which means that you can modify their values. When we type `new_head = self.snake[-1]` we're not creating a new list called `new_head`, we're assigning a new name to `self.snake[-1]`. Some objects in Python, for example tuples, are _immutable_ which means their values cannot be changed. If we used tuples for the co-ordinates we would not have this problem. However, if we used tuples, the code wouldn't be as simple as incrementing or decrementing a co-ordinate, and it would introduce another concept to a beginner-friendly tutorial, something I'd like to keep to a minimum.

Solution

def update(self):
    new_head = list(self.snake[-1])
    if self.direction == "up":
        # Y decreases as you go up the screen
        new_head[1] -= 1
    elif self.direction == "down":
        new_head[1] += 1
    elif self.direction == "left":
        # X decreases as you go left
        new_head[0] -= 1
    elif self.direction == "right":
        new_head[0] +=1
    # put the head on the end of the list
    self.snake.append(new_head)

Firstly we copy the old head of the snake, remember `-1` is the index of the last element of a list. Next we check what the current direction is, and increment or decrement the appropriate co-ordinate. Finally, we append the new head to the end of the snake list.

Your turn 4: Removing the tail#

Remember the .pop(idx) method on lists from before? You can remove an item from a list by calling this function with the index of the item that you want to remove.

>>> my_list = [1, 2, 3]
>>> my_list.pop(0)
1
>>> my_list
[2, 3]

Your task is to remove the tail of the snake. Here's a hint, if we add the new head to the end of the snake list, then the tail will be at the start of it.

Solution

The full `update` function should now look like this:

def update(self):
    new_head = list(self.snake[-1])
    if self.direction == "up":
        new_head[1] -= 1
    elif self.direction == "down":
        new_head[1] += 1
    elif self.direction == "left":
        new_head[0] -= 1
    elif self.direction == "right":
        new_head[0] +=1
    self.snake.append(new_head)
    # cut the tail of the snake
    self.snake.pop(0)

We're now ready to run this again, and our snake should now move!

Demonstration of the wrapping error with the snake moving code.

Except, we get an error. The error on the simulator isn't very helpful, but on the micro:bit it makes more sense.

ValueError: index out of bounds

What's happening here is that our new_head is going out of bounds, which means it's becoming a value that can't be displayed on the screen. If you remember our co-ordinates diagram, the Y co-ordinate is getting smaller until it gets to -1. When we then try and call display.set_pixel with -1 as an input, it gives us an error, since there's no LED at [2, -1].

We could fix this by ending the game if the snake goes out of bounds, but on a screen this small that wouldn't be much fun. Instead, let's fix it by wrapping our snake around the board.

Wrapping#

Wrapping just means that if the snake hits one side of the screen, then it should appear on the other side of the screen.

Demonstration of wrapping the snake to the other side of the screen.

In the above example, the snake is moving up the screen. After the snake's head reaches [3, 0] it goes to [3, 4] and appears at the bottom of the screen. The snake continues to move upwards, just starting from the bottom of the screen now. We can implement this behaviour by checking the bounds of the new_head.

Your turn 5: Wrap the screen#

The bounds of our screen are 0-4 in both the X and Y axes. So you'll want to check if the X co-ordinate or the Y co-ordinate are less than/greater than these boundaries. If they are, reset them to the other edge of the screen.

Again, for this task we'll be working in our update method.

Solution

def update(self):
    ... # the code from above
    elif self.direction == "right":
        new_head[0] +=1

    # X co-ordinate
    if new_head[0] > 4:
        new_head[0] = 0
    elif new_head[0] < 0:
        new_head[0] = 4
    # Y co-ordinate
    if new_head[1] > 4:
        new_head[1] = 0
    elif new_head[1] < 0:
        new_head[1] = 4

    self.snake.append(new_head)
    ...

We add a new `if` statement for both the X co-ordinate and the Y co-ordinate. In it, we check if they are above 4 or below 0 since those are the maximum co-ordinates of our display. This would also work if you only have a single `if/elif` block for both the X and Y since we'll only be changing one at a time.

When we add this code to our game, it should look like the below.

The game successfully wrapping the snake round the screen.

Getting input#

At this point, we have an image on the screen and the snake moves! The user can't yet control the snake though, so let's add that in.

The micro:bit has a few different ways of getting input from the user and the world around it. Most visibly, it has two buttons on the front that the user can press. While we could use these for input, four directions doesn't map into two buttons in an intuitive way. Other inputs the micro:bit has include an accelerometer for detecting tilt and motion, a compass for direction relative to the earth, pins on the bottom for electrical input and a temperature sensor. Out of these, the accelerometer can provide the simplest method of control.

Accelerometers#

We won't try to understand how accelerometers work here, that would take a post of its own. However, we can understand what an accelerometer can do for us. If you've never heard of an accelerometer before, it's the bit of your phone that detects orientation and makes your videos landscape when you tilt your phone.

In short, an accelerometer can measure the tilt of a device, and we can use this for our game. To see how this will work, take a look at the video below.

Using the accelerometer to move a dot around the micro:bit screen

The dot follows the tilt of the device here. When we tilt the device away from us, the dot heads up the screen (away from us). Tilting the device to the left, right or towards us have similar efects.

Simulator only: How to test with the accelerometer#

Controlling the micro:bit accelerometer with the mouse in the create.withcode editor

The micro:bit pop-up in the simulator has some tabs along the top for controlling the various components of the micro:bit. One of these tabs controls the accelerometer (seen in the video above). In the tab you will see three sliders, one for each axis, and a checkbox labelled "Move accelerometer with mouse". Go ahead and tick that checkbox.

With the checkbox ticked, you'll notice that moving your mouse over the micro:bit's screen will change the accelerometer values in the sliders. As you head towards the lower-right corner of the micro:bit the values will increase, and as you head towards the upper-left corner they will decrease.

Your turn 6: Updating direction#

We're working in the handle_input function for this section.

The accelerometer will return you values in the X and Y axes using the following functions. We don't care about the Z axis since our screen is 2D.

# These functions return values in the range -2000 to 2000
accelerometer.get_x()
accelerometer.get_y()

The X axis increases going left to right, and the Y axis increases from top to bottom, just like the screen co-ordinates.

We'll calculate the direction the user wants to go in by taking the biggest value from X and Y and then seeing if it's positive or negative (see diagram above). Careful though! X and Y can both be negative, so you can't do a simple X > Y. To see why, imagine that X was 500 and Y was -1000. Even though Y is a bigger number than X, X > Y == True because X is positive.

You may find the absolute function, abs(x), useful.

# abs (the absolute function) will remove a minus sign if present
>>> abs(-30) == 30
True
# ... and leave the value alone if not
>>> abs(30) == 30
True

Solution

def handle_input(self):
    # get accelerometer values
    x = accelerometer.get_x()
    y = accelerometer.get_y()
    # compare the magnitude of X and Y
    if abs(x) > abs(y):
        # X is bigger than Y (maybe negative)
        if x < 0:
            self.direction = "left"
        else:
            self.direction = "right"
    else:
        # Y is bigger than X (maybe negative)
        if y < 0:
            self.direction = "up"
        else:
            self.direction = "down"

First we get the raw values from the accelerometer for X and Y, and then we compare their magnitudes using `abs()`. If X is bigger than Y, then we see whether X is negative (left) or positive (right). We do the equivalent checks when Y is bigger than X.

The user can now tilt the micro:bit to make the snake move! You may notice while playing with this that the snake will never grow, so now it's time to figure out how food works.

Food#

When the snake approaches some food, we want the snake to grow and the food to appear again in another location. Ideally, the process would look like the following animation.

Animation showing the snake eating food and growing

Building our food mechanic can be split into two steps. Detecting and reacting to food being eaten, and then generating new food.

Your turn 7: Eating food#

The snake "eats" the food when its head is in the same cell as the food. When it eats the food, it should grow by one cell which can be done by not cutting the tail from the snake.

In the case that the snake is eating the food, you should leave some space in your code to generate the new food.

Hint: In Python you can compare two lists just with list1 == list2.

Solution

def update(self):
    ...
    self.snake.append(new_head)
    if new_head == self.food:
        # generate new food
        pass
    else:
        # cut the tail of the snake
        self.snake.pop(0)

We compare the new head and the food. If they are the same then we leave some space in the code to generate the new food. We cut the tail in the else block, which means that we only cut it when the snake isn't eating food.

Random numbers#

We want to generate new food randomly so that the game is interesting. We can do this using the random module in Python. Here's an example that guesses (poorly) how old you are.

# import the function randint from the module random
from random import randint

# generate a random integer between 0 and 100 (inclusive)
age = randint(0, 100)

print(f"I bet you are {age} years old")

In this program we import then call the randint(start, stop) function. This will give us back a number between 1 and 100. We then print it out using an f-string. Side note: if you haven't come across f-strings before, check out that link. They are very cool.

We'll be using this randint function to generate random co-ordinates.

Your turn 8: Generating food#

To generate new food, you'll need to generate an X co-ordinate and a Y co-ordinate. Remember, the axes run from 0 to 4.

One edge case you'll have to think about is what happens if the co-ordinates you generate are covered by the snake. The solution I'll recommend is to keep re-generating the co-ordinates until they are not in the snake, but feel free to come up with your own solutions here. You can check if the co-ordinates are inside the snake using the in keyword (co_ords in self.snake) and a while loop.

Solution

def update(self):
    ...
    if new_head == self.food:
        self.food = [randint(0, 4), randint(0, 4)]
        while self.food in self.snake:
            self.food = [randint(0, 4), randint(0, 4)]
    ...

We use a `while` loop to check if the food we've generated is inside the snake or not. We keep generating the food until it's visible on the screen.

Is this really a valid solution?

In case you're having trouble believing that re-generating the food is a valid solution, let's take a look at some probabilities. We can model generating the food successfully as a binomially distributed random variable. If the snake is of length two (the smallest case after eating food), then the probability of generating the food in an empty space is 23/25. This means that we have a 92% chance of success on the first go and a 99% chance of succeeding within two tries. The expected number of generations before our first success is just over 1. Taking the worst case where the snake is length 24 after eating food, the probability of generating the food correctly is 1/25. This means that the expected number of trials before our first success is 25. In fact 63% of the time it will take 25 tries or fewer and 99% of the time it will take fewer than 113. These numbers are very small for a computer, and we won't notice much delay in the snake's motion.

Our snake game is almost finished! The only step left is figuring out how to end the game.

Ending the game#

The game of snake ends in two situations. Firstly, if you crash into yourself (or the wall, in some versions) then you lose. Secondly, if you fill the entire grid with your snake then you win.

We'll worry about losing the game for now, and I'll leave winning the game as an exercise for you.

Loop control#

We've seen while loops a couple of times so far. Most recently we used them to generate food, but at the very start we saw how our game loop is made of a while loop.

In Python, you have the ability to run a loop forever with while True. The complementary feature is that you have the ability to cut a loop short early using the break statement. Let's take a quick look at how it works.

In the below example, we wait for the micro:bit's button A to be pressed then show a happy face.

# import all the microbit code
from microbit import *

while True:
    # check if button A was pressed since we last called this function
    if button_a.was_pressed():
        # if so, exit the loop
        break
    # wait for 100 milliseconds
    sleep(100)

# show a happy face
display.show(Image.HAPPY)

Your turn 9: Ending the game#

There are three steps to implementing losing the game.

Detect if you've crashed into yourself using the in keyword. This will be part of the update function.
Set self.end to True if the player has lost.
Use break in the game loop if the player has lost. self.end will be game.end inside the game loop.

You could also display a sad face, or scroll some text if the player loses. Get creative!

Solution

In our update function:

def update(self):
    ...
    # after we calculate new_head
    if new_head in self.snake:
        self.end = True
    self.snake.append(new_head)
    ...

We've just added an `if` statement here to check whether the new head is already part of the snake. In our game loop:

while True:
    ...
    game.update()
    # now game.end will be set
    if game.end:
        display.show(Image.SAD)
        break
    game.draw()
    ...

We add an if statement to check whether `game.end` was set to `True` in `update` and if so we break. I've also added a sad face in there to make it more obvious that the player has lost, but that is optional.

Final notes#

Well done on making it this far! You've just built your very own game, and can be really proud of yourself. I hope you've enjoyed the experience and that it's inspired you to keep going with micro:bit, programming or game development. If you want some more material for learning, then check out the Useful links below. If you'd like to check your solution, I've put the final code for this tutorial on GitHub.

This tutorial was based off a workshop that I wrote while part of the Embedded and Robotics Society (EaRS) at Edinburgh. It was one of my favourite workshops there, and it's been fun to re-visit it and turn it into a full tutorial. If you like this, then they have more resources on their website so you should check them out.

If you found any issues in this tutorial, have ideas for new tutorials you'd like to see, or have a question then feel free to contact me. I'd love to hear from you.

Extensions#

Just because you finished this tutorial doesn't mean you have to stop working on the game. There are a number of extensions you could make to it, and you may even think of some of your own.

Implement a winning condition.
Stop the player from going in a direction opposite to the current one they're going in. This annoyed me while playing the game, since it means you lost when you go back into yourself, and most snake implementations prevent you from being able to do this.
Expand the game grid. Maybe you could have a grid that is bigger than the screen, and display a part of it at a time. You could show this by moving the food instead of the snake, or by flashing the screen when you drew a different area.

Useful links#

Copyright#

Pretty micro:bit SVG images were modified from the Micro:bit Educational Foundation's GitHub repo under the CC BY-NC-SA 4.0 license. Those images on this page are therefore licensed under the same terms.

All other content on this page is licensed under the CC BY 4.0 license.

The create.withcode editor hosted on my site is courtesy of Pete Dring of blog.withcode.uk. It has been slightly modified, hence this tutorial not using the official site.

Disappearing documentation (a Kubernetes war story)

2020-05-03T00:00:00+01:00

I like a good war story, and I like them even more when I get to tell them. While writing my dissertation, I ran into a case of disappearing documentation. Here's what happened.

As part of my undergraduate dissertation, I was working with Kubernetes, and more specifically, the controllers within Kubernetes. A controller can be thought of as a generalised thermostat. Just as a thermostat maintains temperature in a house, a controller maintains state in a cluster.

A controller has some desired state, some world state, and a way of affecting the world state (an actuator). In the example of the thermostat, the world state would be the current temperature, the desired state would be the user-set temperature, and the actuator would be a heating system.

Kubernetes has lots of controllers, but the one relevant for this story is the Horizontal Pod Autoscaler. Its job is to check the average CPU used by a set of pods. If the usage is higher or lower than the target usage, then it will create new pods or delete them appropriately. A pod in Kubernetes is a 'logical host', the equivalent of a virtual machine.

Controllers in Kubernetes are implemented as control loops. This means that every N seconds they will collect information about the state of the world, act on it if necessary and then go back to sleep for the next N seconds.

For this part of my dissertation, I needed to measure the time it takes for each iteration of the autoscaler control loop to run. This was to test whether it was correlated with the load on the Kubernetes cluster, something that an attacker may be interested in. The loop execution time, in theory, should be influenced by how much work the autoscaler (and, more generally, the master node) is doing.

I planned to do this from within the cluster, and the flow roughly went as follows:

Record the startup time of a worker pod.
That pod busy loops with high CPU usage.
A short while later, the first iteration of the autoscaler loop runs and decides that more pods need to be scheduled.
A new worker pod is brought up and the startup time recorded.
The two worker pods busy loop with low CPU usage.
The autoscaler runs again but this time decides it needs to destroy a pod.
A worker pod receives SIGTERM and records the time.
The process starts again.

The time between the startups and terminations of the nodes (in bold above) should be correlated with how long it takes the autoscaler control loop to run. However, a problem with this flow is that the controller has a stabilization window to prevent pods from being created and deleted too often. The window ensures that after making a change, Kubernetes won't make another change until the stabilization window has expired. Luckily this can be set, and this is where the disappearing documentation comes in.

The documentation for the Horizontal Pod Autoscaler (HPA) contains a section on the stabilization window above, which included the following text at the time I was looking at it:

Starting from v1.17 the downscale stabilization window can be set on a per-HPA basis by setting the behavior.scaleDown.stabilizationWindowSeconds field in the v2beta2 API. See Support for configurable scaling behavior.

This was what I was looking for, so I set this field in my HPA deployment file and... got an error. I double and triple checked the code I was using to make sure it matched the spec on that page, but every time I tried to kubectl apply this file, I got the error below.

$ kubectl create -f some-autoscale.yaml
error validating "some-autoscale.yaml": error validating data: ValidationError(HorizontalPodAutoscaler.spec): unknown field "behavior" in io.k8s.api.autoscaling.v2beta2.HorizontalPodAutoscalerSpec

Just to make this even more confusing, the Kubernetes API docs agreed with the error that the behavior field didn't exist. At this point, I had exhausted all other options and had to turn to StackOverflow for help. At 7PM I posted a question, Kubernetes unknown field "behavior", and at 11PM I received an answer. The answerer said that I was reading the docs wrong and that the field didn't exist in the first place. Ready to drag this unsuspecting samaritan into my confusion, I went to sleep and planned to reply the following day.

When I checked the docs the following day, the quote from above was missing. After considering whether I'd imagined it all, I discovered that the Kubernetes docs were in a public GitHub repository that was linked from the bottom of each page. It turned out that a couple of hours before I had posted my question, the text about the behavior object had been removed in a PR. The behavior object was meant to be introduced in V1.18 instead of V1.17 that was current at the time. The reason I hadn't seen the updated docs was probably due to my browser caching the old version since I'd visited that page a lot.

Finding out that I wasn't imagining things and that the docs were wrong was very satisfying. Everything had fallen nicely into place. I answered the Stack Overflow question with a shortened version of this story and carried on with my dissertation.

Now that 1.18 is out, this will no longer be a problem, but I thought it would be an interesting story to share. This was only a small part of my dissertation, which was on mobile networks and Kubernetes. It's now finished (exhausted hooray). If you're interested, you can read it here.

Why I chose product management over software development

2020-03-03T00:00:00+00:00

I'm about to finish my degree and have been fortunate enough to receive two graduate job offers. One was an associate product manager position at Google, and the other was software engineering at Bloomberg. Both companies and jobs excited me, which left me with a difficult choice. In this post, I'll explain why I chose product management, even after years of wanting to be a software developer.

The factors that were important to my decision won't be the same for everyone, and this post is not a recommendation of one career path over the other. Choosing a job is deeply personal, and both career paths could potentially be right for any given individual. However, I do hope that this post helps someone else if confronted with a similar decision.

Background#

At the time of this decision, I:

was a final year Computer Science student.
had completed a few software development internships (including an enjoyable one at Bloomberg).
had applied mostly to software jobs and only to a single product management position (the Google one).

I had applied to the Google APM program on a whim and only because I spotted it while looking for software engineering roles. Of course, I liked the sound of the position but had never really thought about working in product before. I wasn't that clear on what product meant.

Throughout the interview process and up until I signed the contract, I did my best to understand product management. I think that I ended up with a reasonably clear idea of what it is, but of course, I have never actually done it. After my research, the points below are the biggest reasons that justified my decision.

Generalism#

I have always identified with the label 'generalist' for a couple of reasons. Firstly, it means that I enjoy novelty, something that shows up most in my hobbies. I rarely stick with one for longer than a few months. Whether or not this is a character flaw (something I'm still unsure about), it's a part of me that I have to take into account when making big decisions.

Another reason that I identify with this label is a fear of specialisation. The idea of becoming a professional expert in a single field or skill scares me. I worry that it would be boring. I'm not saying that I don't want to be an expert in anything, more that I'm afraid of becoming a one-trick pony.

Software development has the potential to provide new challenges each day. I enjoy writing software, and hopefully always will, but I worry about getting pigeonholed as one type of developer. While development would tick the generalism box at the start of my career, I'm not sure that it still would as my career progressed.

Product management, in itself, would be new for me. From everything I've read, the job involves a lot of context switching and many different skills. Most importantly, it doesn't look like this generalist quality of the job disappears as you move up the career ladder.

Communication#

Communication is getting an idea out of your head and into someone else's, but this can cover a wide range of skills due to the many ways we convey information. Writing, presenting, face-to-face conversation, education and visual design can all fall under the communication umbrella.

As I've gotten older, I have placed more importance on communication, mostly as a result of enjoying related activities. Writing for this blog, teaching, giving presentations and finding sponsorship for a hackathon have all brought me a lot of satisfaction, and all of them have been communication heavy. Improving my communication is something I find difficult but pleasantly so.

Product management relies heavily on communication. Attempting to align people from different backgrounds requires understanding all perspectives and then translating them into the language of other people around you. I find this challenge exciting.

Rounding my skill-set#

I have been programming for a long time and over that time have acquired a decent level of technical skill. While there is still a lot of room for me to grow as a developer, there are many other skills that I have never had the chance to focus on improving. Becoming a product manager would allow me to focus on different skills than those I have been able to so far.

Product managers sit in the middle of developers, designers and business people, allowing them to learn a little about each of these areas. This learning would be alongside getting more in-depth knowledge about product management. Product management should broaden my knowledge base in a way few other positions could.

The Google effect#

It might reduce my credibility in some circles, but "working for Google" was also attractive to me. When I first started programming, I idolised Google. Their products were everywhere, and the narrative they cultivated around their working culture seemed exciting and fun, especially for someone who knew little about the working world.

Of course, after having a few jobs and being exposed to more perspectives, my initial infatuation faded. Every year of university, however, I applied religiously to Google for internships. As time went on, some of my friends started getting in, but I still never even received an interview.

So finally getting in after all this time did feel a little unbelievable and I had to be very careful not to accept the offer on that basis alone. I stand by this excitement forming part of my final decision, but it was important to me that it wasn't the whole reason for making my choice.

Conclusion#

I justified my final decision with the points above, but it was still painful to reject the software engineering position. The decision process was long and involved a series of pros/cons lists and talking to many people before I settled on an answer. Alongside friends and family, one of my interviewers from Google also offered to answer any questions I had, which was really helpful.

I'm pleased to say that I'm very sure about my decision, and I feel confident that I made the right choice for me. Of course, it's not possible to know that one job or the other is better overall, because that would require predicting not just one future, but two. That said, I have no regrets.

My start date is in August, and I'm super excited.

Easy Python speed wins with functools.lru_cache

2019-06-10T00:00:00+01:00

Recently, I was reading an interesting article on some under-used Python features. In the article, the author mentioned that from Python version 3.2, the standard library came with a built in decorator functools.lru_cache which I found exciting as it has the potential to speed up a lot of applications with very little effort.

That's great and all, you may be thinking, but what is it? Well, the decorator provides access to a ready-built cache that uses the Least Recently Used (LRU) replacement strategy, hence the name lru_cache. Of course, that sentence probably sounds a little intimidating, so let's break it down.

What is a cache?#

A cache is a place that is quick to access where you store things that are otherwise slow to access. To demonstrate this, let's take your web browser as an example.

Getting a web page from the internet can take up to a few seconds, even on a fast internet connection. In computer time this is an eternity. To solve this, browsers store the web pages you've already visited in a cache on your computer which can be thousands of times faster to access.

Using a cache, the steps to download a webpage are as follows:

Check the local cache for the page. If it's there, return that.
Go find the web page on the internet and download it from there.
Store that web page in the cache to make it faster to access in future.

While this doesn't make things faster the first time you visit a web page, often you'll find yourself visiting a page more than once (think Facebook, or your email) and every subsequent visit will be faster.

Web browsers aren't the only place caches are used. They are used everywhere from servers to computer hardware between the CPU and your hard disk/SSD. Getting things from a cache is quick, and so when you are getting something more than once, it can speed up a program a lot.

What does LRU mean?#

A cache can only ever store a finite amount of things, and often is much smaller than whatever it is caching (for example, your hard drive is much smaller than the internet). This means that sometimes you will need to swap something that is already in the cache out for something else that you want to put in the cache. The way you decide what to take out is called a replacement strategy.

That's where LRU comes in. LRU stands for Least Recently Used and is a commonly used replacement strategy for caches.

Why does a replacement strategy matter?

A cache performs really well when it contains the thing you are trying to access, and not so well when it doesn't. The percentage of times that the cache contains the item you are looking for is called the hit rate. The primary factor in hit rate (apart from cache size) is replacement strategy. Think about it this way: Using the browser example, if your most accessed site was `www.facebook.com` and your replacement strategy was to get rid of the most accessed site, then you are going to have a low hit rate. However if it was LRU, the hit rate would be much better.

The idea behind Least Rececntly Used replacement is that if you haven't accessed something in a while, you probably won't any time soon. To use the strategy, you just get rid of the item that was used longest ago when the cache is full.

In the above diagram each item in the cache has an associated access time. LRU chooses the item at 2:55PM to be replaced since it was accessed longest ago. If there were two objects with the same access time, then LRU would pick one at random.

It turns out that there is an optimal strategy for choosing what to replace in a cache and that is to get rid of the thing that won't be used for longest. This is called Bélády's optimal algorithm but unfortunately requires knowing the future. Thankfully, in many situations LRU provides near optimal performance .

How do I use it?#

functools.lru_cache is a decorator, so you can just place it on top of your function:

import functools

@functools.lru_cache(maxsize=128)
def fib(n):
  if n < 2:
    return 1
  return fib(n-1) + fib(n-2)

The Fibonacci example is really commonly used here because the speed-up is so dramatic for so little effort. Running this on my machine, I got the following results for with and without cache versions of this function.

$ python3 -m timeit -s 'from fib_test import fib' 'fib(30)'
10 loops, best of 3: 282 msec per loop
$ python3 -m timeit -s 'from fib_test import fib_cache' 'fib_cache(30)'
10000000 loops, best of 3: 0.0791 usec per loop

That's a 3,565,107x speed increase for a single line of code.

Of course, I think it can be hard to see how you'd actually use this in practice, since it's quite rare to need to calculate the Fibonacci series. Going back to our example with web pages, we can take the slightly more realistic example of caching rendered templates.

In server development, usually individual pages are stored as templates that have placeholder variables. For example, the following is a template for a page that displays the results of various football matches for a given day.

<html>
  <body>
    <h1>Matches for {{day}}</h1>
    <table id="matches">
      <tr>
        <td>Home team</td>
        <td>Away team</td>
        <td>Score</td>
      </tr>
      {% for match in matches %}
      <tr>
        <td>{{match["home"]}}</td>
        <td>{{match["away"]}}</td>
        <td>{{match["home_goals"]}} - {{match["away_goals"]}}</td>
      </tr>
      {% endfor %}
    </table>
  </body>
</html>

When the template is rendered, it looks like the below:

This is a prime target for caching because the results for each day won't change and it's likely that there will be multiple hits on each day. Below is a Flask app that serves this template. I've introduced a 50ms delay to simulate getting the match dictionary over a network/from a large database.

import json
import time
from flask import Flask, render_template

app = Flask(__name__)

with open('match.json','r') as f:
    match_dict = json.load(f)

def get_matches(day):
    # simulate network/database delay
    time.sleep(0.05)
    return match_dict[day]

@app.route('/matches/<day>')
def show_matches(day):
    matches = get_matches(day)
    return render_template('matches.html', matches=matches, day=day)

if __name__ == "__main__":
    app.run()

Using requests to get three match days without caching takes on average 171ms running locally on my computer. This isn't bad, but we can do better, even considering the artificial delay.

@app.route('/matches/<day>')
@functools.lru_cache(maxsize=4)
def show_matches(day):
    matches = get_matches(day)
    return render_template('matches.html', matches=matches, day=day)

I've set maxsize=4 in this example, because my test script only gets the same three days and it's best to set a power of two. Using this makes the average come down to 13.7ms over 10 loops.

Anything else I should know?#

The Python docs are pretty good, but there are a few things worth highlighting.

Built in functions#

The decorator comes with some built-in functions that you may find useful. cache_info() will help you figure out how big maxsize should be by giving you information on hits, misses and the current size of the cache. cache_clear() will delete all elements in the cache.

Sometimes you shouldn't use a cache#

In general a cache can only be used when:

The data doesn't change for the lifetime of the cache.
The function will always return the same value for the same arguments (so time and random don't make sense to cache).
The function has no side effects. If the cache is hit, then the function never gets called, so make sure you're not changing any state in it.
The function doesn't return distinct mutable objects. For example, functions that return lists are a bad idea to cache since the reference to the list will be cached, not the list contents.

Whilst it's not suitable for every situation, caching can be a super simple way to gain a large performance boost, and functools.lru_cache makes it even easier to use. If you're interested to learn more then check out some of the links below.

More powerful caching library

Python 3.7 functools source

Better parameter validation in Flask with marshmallow

2019-04-25T00:00:00+01:00

Recently I've had two Flask projects with endpoints that take lots of parameters. While working on the first project, I noticed that I was writing a lot of code for validation in each method, and it ended up looking ugly and probably full of bugs. When I started the second project, I thought that there had to be a way to fix this, and it turns out that there was!

To illustrate what I'm talking about, imagine you need to implement the following endpoint for a note taking app.

/api/note - POST

Parameters:
 - title (str) No longer than 60 characters
 - note (str) No longer than 1000 characters
 - user_id (int) No smaller than 1
 - time_created (datetime) Not in the future

In Flask without a library for validation, you might end up writing the following view function to implement this.

@app.route('/api/note', methods=['POST'])
def create_note():
    # we don't know that the 'title' parameter exists yet
    title = request.form.get('title', None)
    if title is None:
        abort(BAD_REQUEST)
    # we have to do manual validation on business requirements
    if len(title) > 60:
        abort(BAD_REQUEST)

    # now we have to do it again?!
    note = request.form.get('note', None)
    if note is None:
        abort(BAD_REQUEST)

    ... # more validation

    actually_create_note()
    return 'ok'

The two parameters I validated up there were strings which made this easier, the integer parameter would have additionally needed a type check and don't even think about parsing and validating the datetime there. The above code is long and very prone to bugs. Of course, you could abstract it all to cleaner methods, but that wouldn't solve the underlying problem of having to write all of this manually.

Thankfully, it turns out there is a library that does validation like this straight out of the box. It's called marshmallow and is meant for object serialization. Alongside parsing and dumping, it also comes with some powerful validation functionality built-in.

What is object serialization?

[Serialization](https://en.wikipedia.org/wiki/Serialization) is the process of converting objects and data from the format used internally in your program into a format that can be stored or transmitted. For example, JSON data can be represented and easily accessed as a dictionary in Python, but it needs to be **serialized** to a string to send it anywhere. The reverse operation is called **deserialization** and is what we'll be dealing with in this article.

The core idea in marshmallow is that data structure is represented with a schema. A schema is a class that defines what format the data comes in. It dictates what fields exist, their types and validation on them. You create a schema by sub-classing marshmallow.Schema and creating attributes that will represent the fields in your data.

Using the note-taking endpoint as an example, we'll create a schema that represents the structure of incoming data to the endpoint.

from marshmallow import Schema, fields

class CreateNoteInputSchema(Schema):
    """ /api/note - POST

    Parameters:
     - title (str)
     - note (str)
     - user_id (int)
     - time_created (time)
    """
    # the 'required' argument ensures the field exists
    title = fields.Str(required=True)
    note = fields.Str(required=True)
    user_id = fields.Int(required=True)
    time_created = fields.DateTime(required=True)

This is a pretty simple class, but already it contains a lot of magic. This will check both existence of fields and their types for you. It's important to note that it won't do any business logic validation yet. You can use this in a view function like so:

create_note_schema = CreateNoteInputSchema()

@app.route('/api/note', methods=['POST'])
def create_note():
    errors = create_note_schema.validate(request.form)
    if errors:
        abort(BAD_REQUEST, str(errors))
    # now all required fields exist and are the right type
    # business requirements aren't necessarily satisfied (length, time bounds, etc)
    actually_create_note()
    return 'ok'

We don't have all of the functionality we need, but even still this has cleaned up the code considerably. The other thing we need to do is to add validation methods for the business requirements. You can do this in two ways with marshmallow. Firstly you could create a method in your schema that has the @validates decorator, or for simple cases, you could give the validate keyword argument to the field.

from marshmallow import Schema, fields
# import built-in validators
from marshmallow.validate import Length, Range

class CreateNoteInputSchema(Schema):
    ...
    # no longer than 60 chars
    title = fields.Str(required=True, validate=Length(max=60))
    # no longer than 1000 chars
    note = fields.Str(required=True, validate=Length(max=1000))
    # at least 1
    user_id = fields.Int(required=True, validate=Range(min=1))
    time_created = fields.DateTime(required=True)

You will notice above that marshmallow comes with a bunch of handy validators built-in. You can see a full list of them in the API docs.

We're still missing a validator for checking that the date is not in the future. Luckily we can use the @validates decorator to write our own.

from datetime import datetime
from marshmallow import Schema, fields, validates, ValidationError
...

class CreateNoteInputSchema(Schema):
    ...
    time_created = fields.DateTime(required=True)

    @validates('time_created')
    def is_not_in_future(value):
        """'value' is the datetime parsed from time_created by marshmallow"""
        now = datetime.now()
        if value > now:
            raise ValidationError("Can't create notes in the future!")
        # if the function doesn't raise an error, the check is considered passed

We don't even have to use any extra code in our view function now to use the extra validation, we still just call the validate method.

@app.route('/api/note', methods=['POST'])
def create_note():
    errors = create_note_schema.validate(request.form)
    if errors:
        abort(BAD_REQUEST, str(errors))
    actually_create_note(request.form)
    return 'ok'

Extra information#

marshmallow is very powerful and contains much more than what I have covered here. Thankfully there are many good resources on the internet for you to research further.

marshmallow Quickstart
Flask-Marshmallow - An integration library that adds extra functionality (automatic schemas from models, extra field types, etc)
How to Build RESTful APIs with Python and Flask | Codementor - A wider look at API building as opposed to just the validation.

How to configure DNS for custom domains on GitHub Pages

2018-11-20T00:00:00+00:00

Recently-ish GitHub announced that HTTPS would now work on custom domains with GitHub Pages. This was a great bit of news, because the web is slowly moving towards a HTTPS-only state of being and it's nice not to be left behind.

Unfortunately for me, for this new feature to work you had to be using either CNAME or ALIAS records to point to your GitHub Pages domain. At the time, I was using A records pointed at the old GitHub Pages servers, which didn't have the new features enabled. All these terms can be quite confusing if you're not familiar with DNS, so let's take a look at some basics.

What is DNS?#

DNS stands for Domain Name System and it's the internet's way of mapping URLs (like cameronmacleod.com) to the actual IP addresses (like 185.199.108.153) of the computers that can serve you the website. By means of an analogy, a URL is a bit like a postal address, for example:

1 Castlehill
Edinburgh
EH1 2NG
United Kingdom

Unless you have been to Edinburgh, you would have no idea how to get to that address. You'd need something to take that address and give you directions back. Similarly, if you give a browser a URL then it would need something (in this case, a nameserver) to give it directions (an IP address) to find a website. Just as humans need directions to find a place, browsers need IP addresses to find a website.

DNS is the system by which these URLs are translated into IP addresses. For a basic mental model of how it works, you can think of these steps:

The browser asks a nameserver for the IP address related to a URL
The nameserver looks through some information it holds known as records to find out
The nameserver then tells the browser the IP and the browser goes away and fetches the website

A nameserver is just the DNS name for a computer that keeps records and can do the job of translating URLs to IP addresses.

This is not comprehensive or 100% accurate, but it's a good enough model for our purposes. If you would like to know more, or you're still a bit confused then Cloudflare have a really good intro to the topic.

What are DNS records?#

So far, we know that things called records store the mapping between URLs and IP addresses. Presumably this means that if we want to make our site point at the GitHub Pages servers, then we need to change these. But what are records and how do they work?

At their core, records are just text files written in a certain way, that tell the nameserver how to find the IP address for a certain website. There are multiple different types of DNS records and all of them serve a slightly different purpose. Alongside the information below, all records have a TTL or Time To Live value that contains the number of seconds that the record is still valid for.

A records - These are possibly the simplest type of DNS records and hold the IP address for either a sub-domain (like blog.cameronmacleod.com) or your root domain (like cameronmacleod.com). Similarly, you may see AAAA records which are the same, but they hold an IPv6 address.
CNAME records - CNAME stands for Canonical Name and points one domain name to another. The idea behind these records is to have a single official domain name and to let lots of other domains point to it. They are frequently used to redirect www. subdomains to the root domain. e.g. www.cameronmacleod.com -> cameronmacleod.com. They can only be used on sub-domains, not on root domains.
ALIAS records - These are similar to CNAME records in that they point your domain to another domain but there are a couple of subtle differences. ALIAS records can only be used on the root domain, not sub-domains, and they are not standardised across all DNS providers, so yours may or may not support them.
Other record types - There are many other record types, including MX for pointing to mail servers, NS for nameserver data and even TXT for general notes.

How can I change my DNS records?#

If you know your DNS provider already, then great! If not, you will need to either figure out which one you are using, or set one up if your site is not yet live. For those of you who have yet to set up DNS, often your domain name provider will have a DNS service as part of purchasing your domain name, if not then there are many free DNS providers out there.

If you (like me) can't remember what DNS provider you used, then a really useful tool can be found from DNSstuff. On that page there is a "DNS Lookup" tool that you can type your domain into.

On the page that follows, under the heading "Referral path:" you will see the words "Response from some.dns.server" where some.dns.server is your DNS server. You can then type that URL into Google and the results should tell you your provider.

The actual process of changing your DNS records varies from provider to provider, but most will have an admin panel that you can log into and add/remove records. The records I have configured for my GitHub pages domain and by extension ones that should work for your GitHub Pages site are as follows:

Type	Host name	Points to
CNAME	www	notexactlyawe.github.io
ALIAS		notexactlyawe.github.io

The first is necessary to redirect www.cameronmacleod.com to the GitHub Pages servers and the second redirects the "apex" domain (cameronmacleod.com).

When you come to configure this for yourself, you may find that there are no ALIAS records available under your provider. Unfortunately your best option in this case may be to change DNS provider.

How do I know it worked?#

DNS is notoriously slow to make changes to. Most providers recommend waiting 24 hours (some up to 72!) before you will be able to see your changes. When checking to see whether or not your changes have propagated, you can do one of two things. Use a DNS tool (like the one from DNSstuff above) or use the unix command dig.

dig is a command that can find and list the DNS records for a given domain name. On Linux and Mac machines, you can run the following in terminal:

cameron@isla:~$ dig www.cameronmacleod.com +noall +answer

; <<>> DiG 9.10.3-P4-Ubuntu <<>> www.cameronmacleod.com +noall +answer
;; global options: +cmd
www.cameronmacleod.com. 655 IN  A   192.30.252.154
www.cameronmacleod.com. 655 IN  A   192.30.252.153

The output might look a bit confusing, but it's the last two lines that count. Those are the records that dig has found for your domain. In my case, this output was from before I changed my DNS settings, so it still shows the old A records that I was using. Running it now would give something like this:

cameron@isla:~$ dig www.cameronmacleod.com +noall +answer

; <<>> DiG 9.10.3-P4-Ubuntu <<>> www.cameronmacleod.com +noall +answer
;; global options: +cmd
www.cameronmacleod.com. 3600    IN  CNAME   notexactlyawe.github.io.
notexactlyawe.github.io. 3599   IN  A   185.199.108.153
notexactlyawe.github.io. 3599   IN  A   185.199.109.153
notexactlyawe.github.io. 3599   IN  A   185.199.111.153
notexactlyawe.github.io. 3599   IN  A   185.199.110.153

You might notice that there are a lot of A records, but that is because ALIAS records will resolve automatically to A records in the DNS server, meaning that when you run dig it just shows you the final output.

You will want to check that the IP addresses in the last column of the dig output look similar to the ones above. For a complete and up to date list of GitHub Pages IP addresses, you can check their documentation.

If this worked, then great! Your DNS is properly configured to redirect your custom domain to GitHub Pages, and you should be able to visit your domain in a browser and see your page being served!

HTTPS for custom domains in GitHub Pages#

If you are trying to set up HTTPS on your site, then the changes above are necessary because they will point you at GitHub's content delivery network which has support for HTTPS. See GitHub's announcement for more details. You will also need to perform a couple more steps.

Firstly, if you used to be using A records then you may need to remove and re-add your custom subdomain from your repository's settings. This will generate you a certificate for your domain. To do this, firstly go to your repository settings.

Next, head to the GitHub Pages settings and remove your domain from the "Custom domain" box and click "Save". Put your domain back in that box and click Save again.

)

After doing this you will need to wait about an hour, but after that you should be able to go to https://siteinthecustomdomainbox.com and see a green padlock in the address bar. If the domain you put in the "Custom domain" box had a www. in front of it, then you will need to add that to the URL when visiting your site else it won't work.

For full and up to date instructions on the above see GitHub's documentation.

One thing to note is that HTTPS will only work for either www.yoursite.com or yoursite.com but not both depending on which one you put in the "Custom domain" box. GitHub are aware of the issue, but haven't given any public timeline for fixing it, see this GitHub Community thread.

If something didn't work#

GitHub have a really useful troubleshooting guide for custom domains and HTTPS.

5 Mistakes companies make recruiting software interns

2018-11-14T00:00:00+00:00

August to November is internship application season for computer science undergraduates and over the past few months I've been applying to a good number of internships myself. Through these applications I've learnt a few things about what makes for a good recruitment process and what doesn't and I hope to share a few of them here.

Having a good process is really important in software engineering roles because candidates hold so much power. A good candidate will be progressing with at least 3 or 4 companies that are likely to hire them at any one time. If they find any reason to dismiss your company, then they will. Below I've listed five common mistakes I've seen in recruiting processes and why each one is likely to lose you a good candidate.

1. Timing the process wrong#

The big companies that recruit computer science interns (Google, Amazon, Facebook etc) tend to start their processes around September. This means that all the candidates they wanted have been snapped up by December at the latest. If you are starting your recruiting process any later than September, then you are losing a lot of good candidates to bigger companies that got in earlier.

The earlier you list your positions, the better. If a candidate receives an offer from you whilst they are only in an early stage of the process with other companies, then they are much more likely to take your position. Of course, the same applies to other companies, which makes it even more necessary to get in early.

All this is not to say that there aren't still good candidates left later on, it's just that you will have a harder time finding them. When a candidate receives an offer, they're likely to take it.

2. Bad communication#

There is nothing worse from a candidate's perspective than not hearing anything back from an application. Not being clear on what happens next in a recruitment process comes in a close second. It's crucial when recruiting to be prompt in replying to candidates and exceedingly clear on what the process will entail.

Often companies make this mistake by opening applications for a position, stockpiling them and then come back to review them a few weeks later. Alongside being an unpleasant candidate experience, this will also lose you candidates to other companies that have reviewed and made offers faster.

In terms of making sure the candidate is aware of what the process entails, it's generally a good idea to let them know as early as possible in an email that lists the steps, and then reinforce this at the end of every step. Candidates really appreciate having all the information and will reward you for your effort with a better image of your organisation.

3. Unhelpful interviews#

An interview is your best chance to get to know the candidate and their experience. Something that's often overlooked, however, is that it's also your best chance to sell the candidate your position.

Lists of what is important to software engineers in their job often start with working with smart colleagues. For the candidate to know that your team are excellent at what they do, it's important that the questions they get asked in interview are smart and relevant. Asking candidates to recite some facts from a textbook is unhelpful for you and them, but asking them to solve a problem that's relevant to your business both gives you greater insight into them and demonstrates your team's skill.

4. Not reimbursing travel#

Hiring software engineers is expensive. The amount of recruiting and engineering time that goes into a single candidate is huge. But the reason for this is that hiring the wrong engineer is even more expensive. So why would you vastly narrow your pool of candidates and exclude many good engineers by not paying for a train fare or flight? The price will pale in comparison to what you're already spending on the candidate. Personally I would not attend an interview without travel reimbursement and I know many people that feel the same.

This is also a problem in terms of recruiting diverse candidates. I'm sure that I don't need to go into the benefits of a diverse workforce, but not reimbursing travel expenses reduces the diversity of your talent pool. Only candidates from the local area, or those who can afford to travel would then attend. Assuming that you'll find enough good engineers this way can be a costly mistake.

5. Ignoring early year undergraduates#

This is quite possibly the most common mistake I see companies making today. A candidate not in their penultimate year of study can have an awful experience with all the auto-rejections and non-replies sent their way. This is a real shame because there are a lot of really good candidates that get ignored by companies that prefer to hire penultimate year students.

Understandably, some companies run internships for the express purpose of channelling interns into their graduate program. The argument goes that candidates are more likely to return for a graduate job if they get the offer just after their internship ends and you can only offer graduate jobs to students expecting to graduate the following year. Whilst this is true, I would disagree with this implying excluding earlier year candidates because in my opinion the early bird gets the worm. If you are a company that doesn't ignore non-penultimate year students then those early students will have a more favourable image of you. On top of this many interns return for another internship at the same company, meaning you still get the funnelling effect, just they're even more productive when they join full time due to the extra experience working in your environment.

It's also worth mentioning that a lot of the most talented students I know are not penultimate-year. Whilst university is valuable, some students have been programming for years beforehand and come into university well above their expected level. It would be a shame to ignore these candidates, especially at a time in their career when other companies are less interested.

TL;DR

Start looking early
Review applications as they come in
Be clear with how the process works
Put a lot of thought into your interview questions
Reimburse travel (at least nationally)
Don't ignore first and second years

Internships are a fantastic thing for all parties involved. Interns get great experience and companies get access to some of the smartest engineers at an early stage in their career. Getting the recruiting process right can lead to great things for your organisation, and it's a shame to see simple mistakes being made.

If you found this useful or interesting, why not follow me on Twitter?

How we ran Welcome Week events

2018-10-06T00:00:00+01:00

At the University of Edinburgh we call the week before lectures begin each year Welcome Week. It's designed as a time for new students to meet each other and get settled into university and often a new city. Most societies will run events to try and attract new students to join them and we (EaRS) were no different. After all, a society dies when no-one is interested any more.

I wanted to write down some of our experience running events during Welcome Week because it is different to running events any other time of the year and it may be useful to anyone who is new to running a society in future. If you are not running a society, hopefully it should be an interesting insight into what goes on behind the scenes.

We ran three separate events during Welcome Week 2018. There was a pub quiz, a workshop and an evening of board games. We attempted to make each event as inclusive to various skill levels as possible, because we wanted to attract first year undergraduates and the level of technical knowledge amongst new students varies wildly.

Choosing events to run#

It's important to be as inclusive as possible during your first events as you want to encourage people to join whatever their experience level. For this reason, we chose to run two socials alongside our workshop, which everyone could turn up to and didn't reward technical knowledge to level the playing field. The workshop was aimed squarely at beginners as well. This is especially important for technical societies like ours, since we can often seem intimidating to beginners.

Mistake #1 - Wrong level of workshop#

The workshop that we ran was aimed at complete programming beginners. We decided to do this since we weren't sure what skill level the freshers coming in would be at. Unfortunately we ended up attracting a lot of experienced people to the workshop and had to split it into two sessions at the same time; one for beginners and one for more advanced people. This made it harder to run the workshop as we could give less attention to each group.

Potential solutions#

More heavily emphasise the beginner nature of the event in the advertising.
Advertise the event through other tangentially related societies with potential interests in learning to code. This would attract people that wouldn't have gone out of their way to find a technical society like ours.
Keep the event at a beginner level, but in a less popular topic such as electronics or mechanical engineering.

Advertising events#

Our student union is the best place to advertise fresher's events in Edinburgh as they put a lot of money into printing booklets and directing students towards the places to find these events. Because we submitted late we didn't get to make full use of this, but we still got a few sign-ups from the online version of their events brochure.

Mistake #2 - Late submission to the student union#

This mistake fell squarely on my shoulders. In April before the last academic year ended, our student union sent us an email with details on how to include our Welcome Week events in their advertising. I didn't pay enough attention to the email to see this and so we missed the deadline for submission by a few months.

Potential solutions#

Read all emails from the student union.
Find a better secretary.

Despite this, the channel that gained us the most sign-ups to the events was without a doubt our Activities Fair, which is an event every year where societies gather in an expo format to advertise to new students. Since we scheduled all our events for after the fair, we were able to direct people to them at the fair and a lot of the people present at the events found out there.

Facebook was also a very useful tool for us due to the network effects. If one person is interested in our events, then there's a good chance that their friends are too, so it acts like improved word of mouth advertising. There's no need to spend money on it if you build your page using off-line methods.

We also used our mailing list from last year, but that generally produces quite low engagement rates.

Make sure to delegate

Having someone responsible for marketing and another person responsible for each individual event helps enormously. The year before, there were only 3 of us and the events wouldn't have run if we didn't draft some friends in. This year having 9 members on the committee in specific roles made things a whole lot easier.

We had great fun during Welcome Week, and despite the mistakes we made it was very successful for the society. The turn-out to events was great and we managed to increase our reach with mailing lists and Facebook dramatically!

Questions? Contact me using the links on the left!

Diagnosing performance issues in a Flask app

2018-04-22T00:00:00+01:00

I was recently part of the team that ran CreatED, the UK’s first hardware hackathon of its kind. Organising it was stressful at times and incredibly rewarding at others, especially when it came to the event actually happening. During the run up to the event, we learned a lot about what actually needs to happen for a hackathon to run and one of those things was the setting up of event software.

It’s easy to overlook when you’re thinking about running your first hackathon, but things like registration systems, mentor allocation systems and hardware distribution systems all need to be put in place. Since we were a hardware hackathon, the last one was of particular importance to us. Thankfully HackMIT and MakeMIT, mostly through Noah Moroze had already put together a fantastic system for this, Cog, which had been battle tested at much larger hackathons than ours.

Cog was a great gift to us, as we’d only started to think about a system for this quite late in the game (2 or 3 weeks to go) and it had pretty much all of the functionality we needed. There were, however, a few things that needed to be added and so I set about doing so.

We needed MyMLH, an OAuth provider, integrated since Cog only worked with HackMIT’s Quill registration system which we weren't using.
We needed participants to be able to upload CVs since Eventbrite had no functionality for this and it had been promised to sponsors.
We wanted a list of admin emails to be recognised by Cog so that not everyone had to use the same credentials.

Most of these were quite easy to integrate, thanks to some logical design on the part of the team at MIT. The CV upload functionality took me longer than the others though mostly due to my inexperience with front-end development. Before I was comfortable signing this off for use during the hack (due to it being so critical), I wanted to do some manual testing, and part of that was load testing since I was worried about a huge amount of users at the start of the hackathon.

I wrote up a quick script that started up a number of threads and used them all to fire a request at the home page of the application in a staggered fashion, hoping to simulate actual usage.

import time
import random
import requests
import threading

URL = "http://localhost:5000"

def time_to_get(url):
    print requests.get(url).elapsed.total_seconds()

def load_test(num_users):
    t_pool = []
    for num in range(num_users):
        t = threading.Thread(target=time_to_get, args=(URL,))
        t.start()
        t_pool.append(t)

    print "Waiting"

    for t in t_pool:
        t.join()


load_test(100)

The results I got from this quick test were mildly alarming. Each of these times (in seconds) is how long it took to receive a full response from the server.

cameron@isla:~/src/created/cog$ python load_test.py
Waiting
1.971983
3.797295
5.694615
7.800551
9.669649
...

To put this into context, this isn’t a particularly demanding page. It displays to the user a list of available hardware and does little else.

However, I wasn’t sure whether the problem was with my script or with Cog itself. To investigate this, I started Googling around profiling in Flask (the framework Cog is written in) and this article pointed me to some Werkzeug middleware that did what I wanted. The output is below from one of the endpoint calls - pay attention to the top call.

PATH: '/inventory'
         1761654 function calls (1645404 primitive calls) in 2.829 seconds

   Ordered by: internal time, call count
   List reduced from 1183 to 10 due to restriction <10>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
      759    0.375    0.000    0.377    0.000 {method 'execute' of 'psycopg2.extensions.cursor' objects}
     4536    0.062    0.000    0.280    0.000 /home/cameron/src/created/cog/venv/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py:898(__init__)
     6426    0.056    0.000    0.070    0.000 /home/cameron/src/created/cog/venv/local/lib/python2.7/site-packages/sqlalchemy/sql/elements.py:674(__getattr__)
...

The 2.829 seconds it took to serve the page suggested that the problem was somewhere in the application as opposed to my script and looked at the output. The most suspicious line to my eyes was the one that ended {method 'execute' of 'psycopg2.extensions.cursor' objects}, mostly because it was the most expensive, but also because it meant the database calls were taking a long time to execute. After a while going down the rabbit hole of trying different deployment options with the database in case the problem was with the database itself I went back and saw the ncalls column had 759 in it. This was fishy as all the page was doing was displaying a list of hardware available to the user, which shouldn’t have taken more than a couple of queries at most.

I was stumped as to what could be making this many queries in the application as the code all looked sensible, so I set SQLAlchemy to echo all queries to see whether I could see a pattern. Sparing you the output of 759 queries, a lot of COUNTs were popping up suggesting that the problem may be in the quantities of items since those were the only numbers on the page. Sure enough, looking at the code I found this:

class InventoryEntry(db.Model):
    ...
    @property
    def quantity(self):
        """Returns quantity of items that have not been 'claimed' by a request"""
        requests = RequestItem.query \
                   .filter_by(entry_id=self.id) \
                   .join(hardwarecheckout.models.request.Request) \
                   .filter_by(status=hardwarecheckout.models.request.RequestStatus.APPROVED) \
                   .with_entities(func.sum(RequestItem.quantity)).scalar()
        if not requests: requests = 0
        return Item.query.filter_by(entry_id = self.id, user = None).count() - requests

In the database, there was a table for inventory entries (roughly corresponding to 'classes' of items) and items (think instances of these classes). When calculating how many of a particular inventory entry was available it would check how many of an inventory entry existed in total and then subtract the number of approved requests for that item. This looks fine at first glance, and indeed it gives the correct result. However the issue is that this is calculated on a per-item basis, meaning each item makes 2 queries, when this could be done in a single query on a table basis.

The solution comes through the use of the GROUP BY clause. The following code selects all items that no user has claimed and groups them by the ID in inventory_entry before counting the groups (effectively returning how many of each inventory entry are free). The bottom line is a dictionary comprehension that puts them in a nice format for the Jinja template.

    counts = db.session.query(Item.entry_id, func.count(Item.entry_id))\
            .group_by(Item.entry_id)\
            .filter_by(user_id = None)\
            .all()
    counts = {id_: count for (id_, count) in counts}

The one important thing to note with this code is that counts[“some_id”] will error if some_id has no items free since counts won't have it as a key. The way to fix this is by using counts.get(“some_id”, 0) where 0 is a default value that’s returned when there’s no entry in the dictionary for some_id.

And as thought, the profiler output looks much better. The ncalls to cursor.execute has gone down to 4, which is much more sensible!

PATH: '/inventory'
         30824 function calls (29237 primitive calls) in 0.055 seconds

   Ordered by: internal time, call count
   List reduced from 677 to 10 due to restriction <10>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
      297    0.003    0.000    0.010    0.000 /home/cameron/src/created/cog/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py:30(instances)
        4    0.003    0.001    0.003    0.001 {method 'execute' of 'psycopg2.extensions.cursor' objects}
    162/1    0.003    0.000    0.025    0.025 {method 'join' of 'unicode' objects}
...

The final code that we deployed is now sitting on GitHub so you can feel free to take a look.

Useful links relating to abracadabra

2016-07-18T00:00:00+01:00

Here is a collection of links relating to abracadabra that may be useful for anyone looking to do something similar.

Github Repositories#

abracadabra

dejavu

Academic Papers#

Audio Thumbnailing of Popular Music Using Chroma-Based Representations

The above paper was more about feature detection for search and retrieval as opposed to robust fingerprinting for search after distortion.

Pairwise Boosted Audio Fingerprint

The above paper is about a new approach to finding filters and quantisers for fingerprinting music.

Computer Vision for Music Identification

Dejavu works in a similar way to the paper outlined above.

An Industrial Strength Audio Search Algorithm - The original Shazam paper

Papers referencing Wang:

https://scholar.google.co.uk/scholar?client=ubuntu&channel=fs&oe=utf-8&gfe_rd=cr&um=1&ie=UTF-8&lr&cites=11824032630980601393

Similar Projects#

Probably the closest in scope and implementation is dejavu by Will Drevo.

An algorithm that takes a slightly different approach is Chromaprint.

Musicbrainz

Playing with Shazam fingerprints

Echoprint

Use Cases#

Alignment of videos of same event using audio fingerprinting

De-duplicating music library

How to read WAVE files in Python

2016-04-07T00:00:00+01:00

Any project that uses audio will usually start out using WAVE files for its on-disk presence, and as with many things in Python, there's a standard library module for that. Now don't get me wrong in the rest of this article - wave does the job. The thing is that it can be a bit confusing to get started with and it's not always the best tool for the job. This post will go over my journey in reading WAVEs and the various approaches I found.

When you read the documentation for wave, you quickly find the readframes() function for reading the meat of the file. This just as quickly poses a problem, that of how to parse the data it returns.

'\xd04\xd52\xd63\x824...'

Of course, this wouldn't pose even the hint of a problem if we bothered to read the spec for wave files, but who reads those? As a result, I found myself writing some convoluted string parser that was the textbook example of treating the symptom as opposed to the cause. It also didn't really work. It would technically be a valid solution, but there is little point getting it up and running because there are better ways.

The next level up is discovering the struct module. struct is a module that allows for the reading of raw binary data into native Python types that I discovered - after some pained googling - in [this Stack Overflow question]. struct.unpack() contains all of the magic that we need. It takes a format string and the data that you want to extract. The below is an example from the git history of abracadabra.

def read_whole(filename):
    wav_r = wave.open(filename, 'r')
    ret = []
    while wav_r.tell() < wav_r.getnframes():
        decoded = struct.unpack("<h", wav_r.readframes(1))
        ret.append(decoded)
    return ret

As a quick explanation of the format string, the < indicates little-endian data (defined in the spec) and the h 1 signed 16-bit int. Before you think "Hooray, a code snippet! I can leave now.", there are a few problems with this approach. Firstly, wave data is not guaranteed to contain int16s and so this would fall down on a good number of files. Second, it's horrendously slow.

Tackling the second issue first, you could minimise the number of calls to unpack(). Calling it with fmt as "<hh" it would expect two int16s, "<hhhhh" would expect 5. You could change the above code to use this as so:

...
chunk_size = 16
while wav_r.tell() < wav_r.getnframes():
    fmt = "<" + "h" * chunk_size
    try:
        decoded = struct.unpack(fmt, wav_r.readframes(chunk_size))
    except struct.error:
        # (w.getnframes() - w.tell()) < chunk_size
        tmp_size = w.getnframes() - w.tell()
        tmp_fmt = "<" + "h" * chunk_size
        decoded = struct.unpack(tmp_fmt, wav_r.readframes(tmp_size))
...

Yet again, as this article is a journey as opposed to a straight out answer, this ugly hack does not come recommended. In general, I find that ugly code == doing things wrong. This example is no different, but is easily fixed.

...
fmt = "<{0}h".format(chunk_size)
...

As an aside, if you are not already using this way to construct format strings, please take the time to internalise it. But back to the change, this works due to an unremarkable looking line in the docs for struct.

A format character may be preceded by an integral repeat count. For example, the format string '4h' means exactly the same as 'hhhh'.

Much better, no?

Our next issue was that the data isn't guaranteed to come in signed 16-bit integers. The good news is that we can get what format it is in programmatically with our old friend, the wave module. The below code is from abracadabra.

def __init__(self, filename, read=True, debug=False):
    mode = 'r' if read else 'w'
    sizes = {1: 'B', 2: 'h', 4: 'i'}
    self.wav = wave.open(filename, mode)
    ...
    self.channels = self.wav.getnchannels()
    self.fmt_size = sizes[self.wav.getsampwidth()]
    self.fmt = "<" + self.fmt_size * self.channels

If you have read the WAVE spec you will see that BitsPerSample is 2 bytes, suggesting that getsampwidth() could return an arbitrary value between 0 and 8192. In reality, you are not likely to encounter greater than 32 bit audio in the wild and sizes reflects this. Saying this, it would probably be good practice to catch KeyError when setting fmt_size and raising a more readable error.

So we now have a working solution, and I have used this in production code before now. There is, however, another optimisation you could make assuming that the files you are loading in aren't overly large and it involves the array module. array is advertised in the docs as memory efficent arrays, but for our purposes we care more that its implemented in straight C and is lightning fast. It's also pretty easy to use.

a = array.array(self.fmt_size)
a.fromfile(open(self.filename, 'rb'), os.path.getsize(self.filename)/a.itemsize)

You just pass it a format string (of the same format as struct) and then call fromfile with a file object and the size of it. According to this SO question it is up to 40X faster than struct.unpack YMMV.

This has been my journey in attempting to read WAVE files and hopefully it will help. Most of the code in here has been adapted from my repository abracadabra and if you are looking for an up-to-date version of what I am using there might be a good place to look. This article is also on Github so if you see something wrong, please submit an issue.

The importance of letting go of ideas

2016-03-13T00:00:00+00:00

On May 29 2015, I made my first entry into a fancy notebook I had bought. This was going to be it, I was never again going to struggle in vain to remember that killer idea I had earlier. Everything was getting written down from here on out.

Despite a decent start, by July the entries had started to get less frequent and at the start of August they had dried up completely. As with the vast majority of things that I start, this logbook concept hadn't got very far at all.

Admittedly, some aspects were useful whilst it lasted. Thanks to this little book, I now have a complete record of my trip to Barcelona to a level of detail that photographs and memories could just never fill on their own. It was also great for keeping track of tasks that I had to do. Just to be clear, it didn't increase the speed with which I did them, just kept a record of them.

The obvious question from here is, "Why did I stop writing in the notebook if it was useful?". I still have the notebook, and it still sits most of the time within easy reach, so it's not as if I lost it or anything. The answer to this I feel lies in the concept of value and the exchange of time for value. People, as a rule, only do things if they view the value of doing said thing to be greater than or equal to the value of the resources they use to do the thing. In this case, the only resource to be used was time, but I still didn't view the benefits I would gain from writing down an idea, or a task to be done as worth the time it would take.

At this point, you may be calling me lazy. The time it takes to write a bullet point down is basically negligible, and all this hot air about value is just covering up a fundamental attitude of "can't be bothered". But that's the thing, "can't be bothered" stems from a lack of perceived value. I used to hate piano lessons. I would rarely practise, and would only do the minimum to get by due to what we would usually call not being bothered. In other terms, not seeing the value in doing the practice.

When we don't do something that should be regular for a while, we fall out of the habit. This, I believe, is what happened with my notebook. I didn't see sufficient value in writing things down, despite forcing myself to do so for a time, and after a while just forgot all about the exercise. The question from here is: Even if I didn't see sufficient value in writing all my ideas down, was there sufficient value anyway?

And here's where we get to the crux of the article. I would say no simply because the vast majority of ideas are absolute rubbish. Even the ones that we think are great (just look the failure rate of startups or the save-the-world ideas so often generated down the pub). As manner of example, one shamefully representative entry from July 2 in my notebook was as follows.

Peckish Rice Crackers

Not overly enlightening stuff.

No matter whether you are an extraordinary genius or you struggle with the puzzles on the backs of cereal boxes, you will give too much value to your own ideas. This narcissism is a fundamental and necessary component of human nature. If people weren't to have ideas that they believed in, then we wouldn't have any of the innovations that we have today. More than that, a lack of self-confidence in our ideas would lead to a lack of leaders, and society itself may not have developed - certainly not to the same extent it has.

Beyond the cheapness of ideas, is there really any harm in scratching the itch of putting them somewhere? Not necessarily, but there is a freedom in learning to let them go. An idea that you are doing your best to remember is a bit of your mind that could be assigned to other things. Not only that, but good ideas tend to be remembered through their own virtue as opposed to needing to be explicitly remembered.

Everyone goes through a diary phase or buys a notebook at some point, it's a natural thing to do. But don't beat yourself up for stopping, that much is logical.

Cameron MacLeod

How to sort parent nodes before child nodes? - Topological sort

Why would you need a topological sort?#

What is a topological sort?#

Cycles and topological sorts#

How do you use a topological sort?#

Multi-threading and TopologicalSorter#

Conclusion#

abracadabra: How does Shazam work?

What is Shazam?#

Why is song recognition hard anyway?#

System overview#

Calculating a spectrogram#

The Fourier transform#

Spectrograms#

Fingerprinting#

Why is the fingerprint based on spectrogram peaks?#

Finding peaks#

Hashing#

Matching#

Conclusion#

Enter abracadabra#

Further reading#

What a computer science degree looks like in 2020

Year 1

Semester 1

Semester 2

Full Year

Year 2

Semester 1

Semester 2

Year 3

Semester 1

Semester 2

Year 4

Semester 1

Semester 2

Full Year

Build a snake game on the BBC micro:bit

Pre-requisites#

What is a micro:bit?#

Setting up environment#

Snake#

Skeleton code#

Creating the game state#

Python lists#

The micro:bit co-ordinate system#

Your turn 1: Filling in the state#

Drawing to the screen#

Drawing the food#

Python for loops#

Your turn 2: Drawing the snake#

Running our code#

Moving the snake#

Python if statements#

Your turn 3: Calculating the new head#

Your turn 4: Removing the tail#

Wrapping#

Your turn 5: Wrap the screen#

Getting input#

Accelerometers#

Simulator only: How to test with the accelerometer#

Your turn 6: Updating direction#

Food#

Your turn 7: Eating food#

Random numbers#

Your turn 8: Generating food#

Ending the game#

Loop control#

Your turn 9: Ending the game#

Final notes#

Extensions#

Useful links#

Copyright#

Disappearing documentation (a Kubernetes war story)

Why I chose product management over software development

Background#

Generalism#

Communication#

Rounding my skill-set#

Python `for` loops#

Python `if` statements#