Project Mpi (Part 2)

I finally decided to write down a step-by-step guide on how to set up a Raspberry Pi for a Beowulf cluster to do parallel programming on, namely MPICH. I've been working on it for a few weeks in my research and have finally gotten to a point where I can say that it's ready to go. The below picture is of our cluster that is safely stored in one of our computer science lab at the university.

The two things I have left to do is to add logging for the update scripts, and include documentation for basically everything I did to set the nodes up. I will be putting these steps into a markdown on my Github repo as well. As this part is only about how to set up each Pi and to get them to communicate with each other, I will be adding Part 3 to this series that will talk about benchmarks/speedup. So, without delaying it any further, let's get started.

1. Install the latest Raspbian OS to an SD Card

You can get the latest version here and if you're unsure how to install it, you can see the instructions here. If you're using Windows, then you can use Win32 Disk Imager.

2. Configure Raspbian Settings

Now that you have an sd card with Raspbian on it, you can put the card into the Pi and boot it up. You should now see a blue screen with a grey window with a bunch of options on it. At this point, you're going to want to follow these steps:

  1. Expand Filesystem (This ensures that the OS uses the whole SD Card)
  2. Overclock - Modest 800MHz (Optional. You can also go higher, but it's not recommended)
  3. Advanced Options - Hostname - Name it pi-00 (Increment the integer for each node you set up)
  4. Advanced Options - Memory Split - Enter 16 (For megabytes, but only enter the number)
  5. Advanced Options - SSH
  6. Finish. Don't reboot

You should now be in the shell. Type sudo vi /etc/inittab (who uses nano anyway?) and comment out this line: 1:2345:respawn:/sbin/getty --noclear 38400 tty1 (by comment, I mean adding # infront of it). Right below the line you just commented, you're going to add this line:
1:2345:respawn:/bin/login -f pi tty1 </dev/tty1 >/dev/tty1 2>&1

You can now type sudo reboot and it should automatically log you in.

3. Add/Update Software

We're back in the shell and we're going to update some software, as well as add some.

  1. Run sudo apt-get update once this is done, then type sudo apt-get upgrade.
  2. Install C++11. Just follow the steps made in this post.
  3. mkdir ~/Downloads && mkdir -p ~/Development/Mpi (You can replace Mpi with whatever your project name is)

4. Compiling MPICH

(As of when this post was made, 3.1.3 is the latest version of MPICH. Go here to check for other versions)

  1. cd ~/Downloads
  2. wget http://www.mpich.org/static/downloads/3.1.3/mpich-3.1.3.tar.gz
  3. tar xfz mpich-3.1.3.tar.gz
  4. cd ~/Development/Mpi && mkdir tmp && cd tmp
  5. /home/pi/Downloads/mpich-3.1.3/configure --prefix=/home/pi/Development/Mpi/lib/mpich --disable-fortran CXXFLAGS="-Wall -g -std=c++11" 2>&1 | tee c.txt
  6. make 2>&1 | tee m.txt (NOTE: This is goint to take a very long time, about ~2 hours)
  7. make install 2>&1 | tee mi.txt (NOTE: Similarly, this is going to take awhile)
  8. cd .. && rm -rf /path/to/project/tmp

If you've come this far, maybe you're willing to come a little further. Now we need to add the files you compiled into the lib folder inside your project, into your path so we can use them of course!

Enter sudo vi ~/.bashrc and go to the very bottom of the file and add this line: PATH=$PATH:/home/pi/Development/Mpi/lib/mpich/bin then sudo reboot. Now you should be able to use the MPICH library to compile your code! Try it out by typing mpiexec -n 1 hostname. If You get an error, you've probably typed in the PATH variable wrong in to the .bashrc file.

5. Copy Your Configured OS

Now you can safely turn the Raspberry Pi off and put the SD Card back in your computer. Instead of having to do Step 1 - 4 for every single node, we're just going to get the OS from the SD Card, which is already pre-configured and just install it as is on to the other nodes! Awesome, isn't it?

If you're on Mac, you probably used the dd tool in the command line. Now this time, you're just going to switch the of and if paths. Makes sense, right? Again, if you're using Windows, just use the Win32 Disk Imager to copy it to your desktop.

After copying the image file from the SD Card onto your computer, you can just install this .img file to all of your other SD Cards, and voilà! Now you have the same exact configuration on all of your nodes!

6. Node Communication

At this point you should now have the came OS configuration on all of your nodes. Go ahead and boot them all up and connect the ethernet on all of them. To check the ip addresses on each node, you can just tpye ifconfig | grep inet to get the local address. This might get a littly tricky here because, if you want to set up static IP addresses, then you'll have to do that configuration now.

(Optional: configuration for static IP addresses)

sudo vi /etc/network/interfaces

Edit the file as such (replacing the XXX with your desired address of course):

auto lo eth0  
iface lo inet loopback  
iface eth0 inet static  
        address XXX.XXX.XXX.XXX
        netmask XXX.XXX.XXX.XXX
        gateway XXX.XXX.XXX.XXX
        dns-nameservers XXX.XXX.XXX.XXX

Remember a while ago when we changed the hostname of the Pi to pi-00? Well, it's time to change the name for the other nodes. Incrementally of course. Go ahead and run sudo raspi-config and do this:
Advanced Options - Hostname - Name it pi-01 and then increment it for each node you rename.

The road has almost come to an end, but we just have one step left, the damn SSH keys. Starting at pi-00:

  1. ssh-keygen
  2. cd ~/.ssh
  3. cp id_rsa.pub pi-00

This is what you'll have to do on all of the other nodes:

  1. ssh-keygen
  2. cd ~/.ssh
  3. cp id_rsa.pub pi-N (Replace N with the number of the Pi node)
  4. scp pi@XXX.XXX.XXX.XXX:/home/pi/.ssh/pi-00 pi-00 (Replace XXX with the address of pi-00 node)
  5. cat pi-00 >> authorized_keys

Now that all of this is done, we'll have to add all of these SSH keys to pi-00. On pi-00 do this:

  1. cd ~/.ssh
  2. scp pi@XXX.XXX.XXX.XXX:/home/pi/.ssh/pi-N pi-N (Replace XXX with the address of the nth node)
  3. cat pi-N >> authorized_keys

You'll have to do steps 2-3 as many times as you have nodes.

To test that everything is working, you can create a machine file: vi machinefile and add the IP addresses of all your nodes in there, each in a new line. Here comes the moment of truth:

mpiexec -f machinefile -n N hostname (Replace N with the number of nodes you have)

If this listed the name of each node, then:

You can give yourself a pat on the back, because you are done!

This seemed to have taken forever, however if you've made it this far, I can tell you it was definitely worth it. This may seem a bit complicated, however, once you get into it, it's not bad at all. I've tried to make this guide as clear as possible, so if there's anything confusing then you can drop me a line either by email, or twitter. I will be adding this guide to the repository and will update it in both places if I come across an error.

In Part 3, I will be talking about the speedup from 1 processor vs 32, as well as what to do with the cluster moving forward.