Compiling LAM-MPI with support for IBM XLC and XLF

I created this page to provide information for the growing number of people that are interested in using Mac OS X for high performance computing. This is designed for folks that want to compile and run parallel MPI applications with MPI. This is just a quick primer on getting MPI running on Mac OS X, for a more comprehensive set of documentation on LAM-MPI please refer to the LAM-MPI installation or user guide.

I had a lot of help from various colleagues and the friendly folks at LAM-MPI. This first page is centered on the use of LAM-MPI with the IBM XLC and XL Fortran compilers. If you don't need FORTRAN support and want to use GCC instead of IBM's XLC, you can get by with just using the LAM-MPI Mac OS X package installer. If these first pages are useful, I'll put up follow-on pages with directions for building other MPIs and other compilers.

And just to be sure, for everything to run correctly, you will have to install LAM-MPI and the IBM compilers on every compute node. In the meantime, have a LAM-MPI day!

Document Key:

Plain black text are the descriptions of what we are doing

Green text are input to the terminal

Download and untar the latest lam distribution. As of the push of this page, the LAM-MPI version is v7.1.1 and we have IBM's XLC v6.0 and XLF v8.1 compilers. (Please note that if you need support for Myrinet or Infiniband, you will need to download the latest beta version of LAM-MPI) Go to the top-level directory of the lam source code. Enter the following steps at the command prompt in the terminal window to get LAM-MPI correctly installed and running on OSX (assuming bash shell, which is default on Panther):

Setting the environment or compiler flags for compiling LAM-MPI with the XLC and XLF.

cd lam-7.0.x

export CC=/opt/ibmcmp/vacpp/6.0/bin/xlc

export CXX=/opt/ibmcmp/vacpp/6.0/bin/xlc++

export FC=/opt/ibmcmp/xlf/8.1/bin/xlf

export F77=/opt/ibmcmp/xlf/8.1/bin/xlf

export CFLAGS="-O3"

export CXXFLAGS="-O3"

export FFLAGS="-O3"

By default, LAM-MPI installs itself in /usr/local. However, if you would like to maintain a clean path to LAM-MPI or install different versions of LAM-MPI, you might want to install LAM-MPI in a different directory location. For example, you might want to install LAM-MPI in /usr/local/lam/ directory. In order to do that simply add the --prefix=/usr/local/lam flag to the ./configure script. After setting the environment variables, the sequence of steps should look like this:

./configure --with-rsh=/usr/bin/ssh --prefix=/usr/local/lam --with-memory-manager=none

The "--with-memory-manager=none" is used only if you are using LAM with the IBM XL Compilers. Additionally, if you are installing the latest 7.1.x beta version of LAM, you also do not need the "--with-memory-manager=none". I have been successfully running the LAM-MPI beta release with GigE, Myrinet, and Infiniband support; when configuring with support for the other fabrics just specify "--with-gm=[path to gm]none" or "--with-ib=[path to ib]none"

If you need to configure the "configure" script, get help on the script by typing "./configure --help" at the command prompt. If you need more control over the LAM installation process, read the manual :-).

Running make and installing LAM

make

sudo make install

Setting the Path for LAM and LAM Man Pages and paths for IBM XLC and XLF Compilers

In order for all users to be able to run LAM-MPI and dynamically link to the IBM libraries, we will need to edit the global bashrc file located in in /etc. Additionally, we will make any necessary edits to /etc/profile to make sure we source the bashrc file that just modified. Below are the instructions.

Editing /etc/bashrc:

Editing the /etc/bashrc file, as opposed to creating a .bash_profile in your own user account will allow any user account on your Mac to have the correct paths.

You will want to append the bashrc file to look like the file below:

# System-wide .bashrc file for interactive bash(1) shells.

PS1='\h:\w \u\$ '

# Make bash check it's window size after a process completes

shopt -s checkwinsize

if [ -r if [ -r /usr/local/lam/bin ]; then

PATH="$PATH:/usr/local/lam/bin"

if [ -r /opt/ibmcmp/xlf/8.1/bin ]; then

PATH="$PATH:/opt/ibmcmp/xlf/8.1/bin"

if [ -r /opt/ibmcmp/vacpp/6.0/bin ]; then

PATH="/opt/ibmcmp/vacpp/6.0/bin:$PATH"

export PATH

Editing /etc/profile

In order for the edits in the bashrc to take effect, we have to make sure to source the bashrc by editing /etc/profile. The file should like like this:

# System-wide .profile for sh(1)

PATH="/bin:/sbin:/usr/bin:/usr/sbin"

export PATH

[ -r /etc/bashrc ] && source /etc/bashrc

Running the mpi demos on a single machine

Compiling the demo programs. I had some problems running the Makefile and didn't want to figure it out. Instead, I went into the directories of each of the demo programs and compiled using the appropriate commands. You can find these out by reading the README file in each of the example files. Below is an example:

Compiling the Pi demo. Change directories to where your Pi examples are stored, in this case, to the LAM-MPI directory.

cd ~/myaccount/lam-7.0.6/examples/pi

mpicc -o cpi cpi.c -lm

mpic++ -o cxxpi cxxpi.cc -lm

mpif77 -o fpi fpi.f -lm

Runnig the Pi demo. Change directories to where your Pi examples are stored, in this case, to the LAM-MPI directory.

cd /usr/local/lam/bin

Start the LAM daemon:

./lamboot

Launching the MPI job:

./mpirun C /Users/myaccount/lam-7.0.6/examples/pi/cpi

./mpirun C /Users/myaccount/lam-7.0.6/examples/pi/fpi

./mpirun C /Users/myaccount/lam-7.0.6/examples/pi/cxxpi

Running LAM-MPI jobs over Ethernet across a cluster

If you will be running LAM-MPI across a cluster of machines, you will need to install MPI on all of the compute nodes as well as all of the compilers. IMPORTANT: every compute node will need to have access to the application; this is usually done with a common shared directory across all machines.

The machinefile: For LAM-MPI to run jobs across the cluster, it will also need to have a list of available machines. An example machine file containing four computers is shown below:

Shell$ more lam_nodes

# begin sample lam machines

node1.cluster.private cpu=2

node2.cluster.private cpu=2

node3.cluster.private cpu=2

node4.cluster.private cpu=2

# end sample lam machines

To launch a job with these machines, we will run lamboot and point to this machine file with the -machinefile option. Running the Pi demo on the above machines would look like this (assuming four dual processor G5 Xserves):

Starting the LAM daemon:

cd /usr/local/lam/bin/

./lamboot -v -ssi boot rsh lamnode_file

Launching the MPI job:

Important, in order for a successful launch of the jobs, the applications will need to be compiled and installed in the exact same location on every node (Usually, this done with a common shared directory across all the nodes).

./mpirun C /Users/myaccount/lam-7.0.6/examples/pi/cpi

./mpirun C /Users/myaccount/lam-7.0.6/examples/pi/fpi

./mpirun C /Users/myaccount/lam-7.0.6/examples/pi/cxxpi