Myrinet: Compiling MPICH-GM with support for IBM XLF and GCC 3.3

 

This page has been written for MPICH-GM, support for Myrinet low latency interconnects on Mac OS X v.10.3. Specifically this page is for compiling Myricom MPICH-GM with support for IBM XLC and GCC 3.3 compilers. Once again, I had a lot of help from various colleagues and the folks at Myricom provided great documentation of their own. And just to be sure, for everything to run correctly, you will have to install MPICH XLC and GCC on every compute node.

 

The assumption for this page is that all of the Myrinet fabric has already been installed on the Xserves. The guides and installation instructions can be downloaded from Myricoms website. I also found Myricoms technical support very helpful, you can reach them by emailing: help@myri.com.

 

Document Key:

Plain black text are the descriptions of what we are doing

Green text are input to the terminal

To begin, make sure that passwordless ssh is enabled on ALL nodes in which you will be installing MPICH-GM.

 

Download and untar the latest Mac OS X MPICH-GM distribution from Myricom's website (You will need a password from Myricom for this). As of the push of this page, the MPICH-GM version is 1.2.5..2 and we have IBM's XLF v8.1  and GCC 3.3 compilers. Go to the top-level directory of the MPICH-GM source code.

 

Myricom provides excellent installation instructions as well as an installation script. This script is fine but you will need to run it as root to enable MPICH to install properly. For this page, we assume a more manual installation. Enter the following steps at the command prompt in the terminal window to get MPICH correctly installed and running on OSX (assuming bash shell, which is default on Panther):

 

To begin, change directories to the MPICH-GM directory. Next we will set the environment or compiler flags for compiling MPICH with XLF and GCC.

export RSHCOMMAND=ssh

export FC=/opt/ibmcmp/xlf/8.1/bin/xlf_r

export F90=/opt/ibmcmp/xlf/8.1/bin/xlf90_r

export CC=/usr/bin/gcc

 

We will install MPICH-GM in usr/local/mpich. In order to do this we will add the --prefix=/usr/local/mpich flag to the ./configure script. After setting the environment variables, the sequence of steps should look like this:

 

./configure --with-device=ch_gm -prefix=$PREFIX --enable-sharedlib -opt="-O" --without-romio -lib="-L/opt/gm/lib/ -lgm " >& config-mine.log

make

sudo make install

 

If you need more information on configuring the "configure" script, you can get help on the script by typing "./configure --help" at the command prompt.

 

Running the mpi demos on a single machine

Compiling the demo programs. Below is an example:

Compiling the Pi demo. Change directories to where your Pi examples are stored, in this case, to the mpich directory.

cd /usr/local/mpich/examples

/usr/local/mpich/bin/mpicc -o cpi cpi.c lm

/usr/local/mpich/bin/mpif77 -o pi3 pi3.f -lm

 

Running MPICH jobs over the Myrinet fabric cluster

When running MPICH across a cluster of machines, you will need to install MPICH on all of the compute nodes as well as all of the compilers. IMPORTANT: every compute node will need to have access to the application; this is usually done with a common shared directory across all machines.

 

The machinefile: For MPICH to run jobs across the cluster, it will also need to have a list of available machines. An example machinefile mpich_machines containing four Myrinet computer nodes is shown below:

 

# begin sample machinefile

node1.cluster.private

node2.cluster.private

node3.cluster.private

node4.cluster.private

# end sample machinefile

 

To launch a job with these machines, you will need to point to this machine file with the "-machinefile" option. Running the Pi demo on the above machines would look like this (assuming four dual processor G5 Xserves):

 

mpirun -np <number of processes> -machinefile <machinefile_name> <app>

cd /usr/local/mpich/bin

./mpirun np 8 machinefile mpich_machines /shared/examples/cpi