I finally decided to write down a step-by-step guide on how to set up a Raspberry Pi for a Beowulf cluster to do parallel programming on, namely MPICH. I've been working on it for a few weeks in my research and have finally gotten to a point where I can say that it's ready to go. The below picture is of our cluster that is safely stored in one of our computer science lab at the university.
The two things I have left to do is to add logging for the update scripts, and include documentation for basically everything I did to set the nodes up. I will be putting these steps into a markdown on my Github repo as well. As this part is only about how to set up each Pi and to get them to communicate with each other, I will be adding Part 3 to this series that will talk about benchmarks/speedup. So, without delaying it any further, let's get started.
1. Install the latest Raspbian OS to an SD Card
2. Configure Raspbian Settings
Now that you have an sd card with Raspbian on it, you can put the card into the Pi and boot it up. You should now see a blue screen with a grey window with a bunch of options on it. At this point, you're going to want to follow these steps:
Expand Filesystem(This ensures that the OS uses the whole SD Card)
Overclock - Modest 800MHz(Optional. You can also go higher, but it's not recommended)
Advanced Options - Hostname- Name it
pi-00(Increment the integer for each node you set up)
Advanced Options - Memory Split- Enter 16 (For megabytes, but only enter the number)
Advanced Options - SSH
- Finish. Don't reboot
You should now be in the shell. Type
sudo vi /etc/inittab (who uses nano anyway?) and comment out this line:
1:2345:respawn:/sbin/getty --noclear 38400 tty1 (by comment, I mean adding
# infront of it). Right below the line you just commented, you're going to add this line:
1:2345:respawn:/bin/login -f pi tty1 </dev/tty1 >/dev/tty1 2>&1
You can now type
sudo reboot and it should automatically log you in.
3. Add/Update Software
We're back in the shell and we're going to update some software, as well as add some.
sudo apt-get updateonce this is done, then type
sudo apt-get upgrade.
- Install C++11. Just follow the steps made in this post.
mkdir ~/Downloads && mkdir -p ~/Development/Mpi(You can replace Mpi with whatever your project name is)
4. Compiling MPICH
(As of when this post was made, 3.1.3 is the latest version of MPICH. Go here to check for other versions)
tar xfz mpich-3.1.3.tar.gz
cd ~/Development/Mpi && mkdir tmp && cd tmp
/home/pi/Downloads/mpich-3.1.3/configure --prefix=/home/pi/Development/Mpi/lib/mpich --disable-fortran CXXFLAGS="-Wall -g -std=c++11" 2>&1 | tee c.txt
make 2>&1 | tee m.txt(NOTE: This is goint to take a very long time, about ~2 hours)
make install 2>&1 | tee mi.txt(NOTE: Similarly, this is going to take awhile)
cd .. && rm -rf /path/to/project/tmp
If you've come this far, maybe you're willing to come a little further. Now we need to add the files you compiled into the
lib folder inside your project, into your path so we can use them of course!
sudo vi ~/.bashrc and go to the very bottom of the file and add this line:
sudo reboot. Now you should be able to use the MPICH library to compile your code! Try it out by typing
mpiexec -n 1 hostname. If You get an error, you've probably typed in the
PATH variable wrong in to the
5. Copy Your Configured OS
Now you can safely turn the Raspberry Pi off and put the SD Card back in your computer. Instead of having to do Step 1 - 4 for every single node, we're just going to get the OS from the SD Card, which is already pre-configured and just install it as is on to the other nodes! Awesome, isn't it?
If you're on Mac, you probably used the dd tool in the command line. Now this time, you're just going to switch the
if paths. Makes sense, right? Again, if you're using Windows, just use the Win32 Disk Imager to copy it to your desktop.
After copying the image file from the SD Card onto your computer, you can just install this .img file to all of your other SD Cards, and voilà! Now you have the same exact configuration on all of your nodes!
6. Node Communication
At this point you should now have the came OS configuration on all of your nodes. Go ahead and boot them all up and connect the ethernet on all of them. To check the ip addresses on each node, you can just tpye
ifconfig | grep inet to get the local address. This might get a littly tricky here because, if you want to set up static IP addresses, then you'll have to do that configuration now.
(Optional: configuration for static IP addresses)
sudo vi /etc/network/interfaces
Edit the file as such (replacing the XXX with your desired address of course):
auto lo eth0 iface lo inet loopback iface eth0 inet static address XXX.XXX.XXX.XXX netmask XXX.XXX.XXX.XXX gateway XXX.XXX.XXX.XXX dns-nameservers XXX.XXX.XXX.XXX
Remember a while ago when we changed the hostname of the Pi to
pi-00? Well, it's time to change the name for the other nodes. Incrementally of course. Go ahead and run
sudo raspi-config and do this:
Advanced Options - Hostname - Name it
pi-01 and then increment it for each node you rename.
The road has almost come to an end, but we just have one step left, the damn SSH keys. Starting at
cp id_rsa.pub pi-00
This is what you'll have to do on all of the other nodes:
cp id_rsa.pub pi-N(Replace N with the number of the Pi node)
scp pi@XXX.XXX.XXX.XXX:/home/pi/.ssh/pi-00 pi-00(Replace XXX with the address of
cat pi-00 >> authorized_keys
Now that all of this is done, we'll have to add all of these SSH keys to
pi-00 do this:
scp pi@XXX.XXX.XXX.XXX:/home/pi/.ssh/pi-N pi-N(Replace XXX with the address of the nth node)
cat pi-N >> authorized_keys
You'll have to do steps 2-3 as many times as you have nodes.
To test that everything is working, you can create a machine file:
vi machinefile and add the IP addresses of all your nodes in there, each in a new line. Here comes the moment of truth:
mpiexec -f machinefile -n N hostname (Replace N with the number of nodes you have)
If this listed the name of each node, then:
You can give yourself a pat on the back, because you are done!
This seemed to have taken forever, however if you've made it this far, I can tell you it was definitely worth it. This may seem a bit complicated, however, once you get into it, it's not bad at all. I've tried to make this guide as clear as possible, so if there's anything confusing then you can drop me a line either by email, or twitter. I will be adding this guide to the repository and will update it in both places if I come across an error.
In Part 3, I will be talking about the speedup from 1 processor vs 32, as well as what to do with the cluster moving forward.