BLAS and Linpack installation on Rocks cluster

You should run the programs by a non-root user. So create a new user in the system.

Add a new user e.g. linpacker

# useradd linpacker

# passwd linpacker

Sync the newly created user,

# rocks sync users

1. Installation of BLAS:

1.1 Login from user ‘linpacker’.

1.2. Download GotoBLAS from TACC website http://www.tacc.utexas.edu/tacc-projects/gotoblas2

1.3. Untar it into /export/home/linpacker/

# tar -xvf GotoBLAS2-1.13.tar.gz

1.4. Change directory to the GoToBLAS2 directory

# cd /export/home/linpacker/GotoBLAS2

1.5. Build GoToBLAS2

# /export/home/linpacker/GotoBLAS2/quickbuild.64bit

If you face problem as above, follow step 1.7.

1.6. Run the MAKE command

# make

1.7. The compilation might fail. To resolve the issue, do the following:

# gmake clean

# gmake TARGET=NEHALEM

It will compile and will produce the required files.

After running gmake command, you don’t need to follow step 1.6.

2. HPL (Linpack Portion):

2.1. Change to home directory

# cd /export/home/linpacker

2.2. Download HPL from NetLib website http://www.netlib.org/benchmark/hpl/hpl-2.1.tar.gz

2.3. Untar it into /export/home/linpacker/  

# tar -xvf hpl-2.1.tar.gz

2.4. Create a Symbolic Link for the HPL directory

# ln -s /export/home/linpacker/hpl-2.1 /export/home/linpacker/hpl

2.5. Copy the Make.Linux_PII_FBLAS from the HPL setup directory to the HPL root

# cp /export/home/linpacker/hpl/setup/Make.Linux_PII_FBLAS /export/home/linpacker/hpl

2.6. Edit the Make.Linux_PII_FBLAS file with the following changes:

# vim /export/home/linpacker/hpl/Make.Linux_PII_FBLAS

o    TOPdir       = /export/home/linpacker/hpl

o    MPdir        = /opt/openmpi/

o    MPlib        = $(MPdir)/lib/libmpi.a

o    LAdir        = /export/home/linpacker/GotoBLAS2

o    LAlib        = $(LAdir)/libgoto2.a -lm -L/usr/lib/gcc/i386-redhat-linux/4.1.2

o    CC           = /opt/openmpi/bin/mpicc

o    LINKER       = /opt/openmpi/bin/mpif77

Save and Quit the Make.Linux_PII_FBLAS file.

2.7. Change to the HPL directory.

# cd hpl

2.8. Run the MAKE command specifying the architecture

# make arch=Linux_PII_FBLAS

3. Running benchmarking:

The benchmarking executable is ‘xhpl’.

Login from the linpacker user.

3.1. Make a backup of the HPL.dat file.
# cp /export/home/linpacker/hpl/bin/Linux_PII_FBLAS/HPL.dat /export/home/linpacker/hpl/bin/Linux_PII_FBLAS/HPL.dat.original

3.2. Create a file in your home directory named machines, and put two entries in it, such as:

# vim /export/home/linpacker/machines

compute-0-0

compute-0-1

3.3 To launch the job from frontend:

# ssh-agent $SHELL

# ssh-add

Change directory where the executable is present:

# cd /export/home/linpacker/hpl/Linux_PII_FBLAS/

And run the command:

# /opt/openmpi/bin/mpirun -np 6 -

machinefile/export/home/linpacker/machines/export/home/linpacker/hpl/bin/Linux_PII_FBLAS/xhpl

Note:

Just to check whether the ‘xhpl’ is being executed or not, simply run the command:

Before running the command, go to the path of the ‘xhpl’ file i.e.

# cd /export/home/linpacker/hpl/Linux_PII_FBLAS/

# mpirun –np 5 ./xhpl

To run the command on compute nodes, follow this:

1. The compute nodes should be installed after the completion of all the processes or installations on the master node so that all the changes could be installed on the compute nodes.

2. The executable ‘xhpl’ will be presented at the same path on the both master node and the compute node.

3. Simply run the command from master node from the non-root user:

# cd /export/home/linpacker/hpl/Linux_PII_FBLAS/

# mpirun –np 5 –machinefile /export/home/linpacker/machines  xhpl

OR

# mpirun –np 5 –H compute-0-1  xhpl

Note:

1. The HPL.dat file:

------------------------------------------------------------------------------------------------------------------

HPLinpack benchmark input file
Innovative Computing Laboratory, University of Tennessee
HPL.out      output file name (if any)
6            device out (6=stdout,7=stderr,file)
1            # of problems sizes (N)
1000 Ns
1            # of NBs
64 NBs
1            # of process grids (P x Q)
2 Ps
2 Qs
16.0         threshold
3            # of panel fact
0 1 2            PFACTs (0=left, 1=Crout, 2=Right)
1            # of recursive stopping criterium
8            NBMINs (>= 1)
1            # of panels in recursion
2            NDIVs
1            # of recursive panel fact.
2            RFACTs (0=left, 1=Crout, 2=Right)
1            # of broadcast
1            BCASTs (0=1rg,1=1rM,2=2rg,3=2rM,4=Lng,5=LnM)
1            # of lookahead depth
1            DEPTHs (>=0)
2            SWAP (0=bin-exch,1=long,2=mix)
80           swapping threshold
0            L1 in (0=transposed,1=no-transposed) form
0            U  in (0=transposed,1=no-transposed) form
1            Equilibration (0=no,1=yes)
8            memory alignment in double (> 0)------------------------------------------------------------------------------------------------------------------