Preparing our institute’s cluster for an OpenStack installation I got into the situation where I had to manually configure several servers in the same way. Of course, since repeating the same terminal commands isn’t all that fun, I went searching for a tool to simplify this. Enter Cluster SSH.
Instructions on how to set up an Ubuntu cluster can be found at https://help.ubuntu.com/community/MpichCluster .
I’ve updated a few outdated commands there myself so it shouldn’t be too hard to follow the instructions. The only thing I personally did differently was that I didn’t create a new user, but instead used my old account on all the machines (the important thing is that the username be the same everywhere).
In this post I’ll explain how to make a Python script to utilize this cluster using the MPI standard for parallel programming.
To prepare your Python interpreter for parallel programming, you first need some sort of an MPI interface. Several exist so it’s up to you to choose. I used mpi4py. This is part of the scipy module and it can be installed through Synaptic or with:
sudo apt-get install python-dev # other potential packages to consider - python-mpi mpichpython python-scipy python-numpy