Monday, November 30, 2015

BLAS and Linpack installation on Rocks cluster




You should run the programs by a non-root user. So create a new user in the system.

Add a new user e.g. linpacker
# useradd linpacker
# passwd linpacker

Sync the newly created user,
# rocks sync users

1. Installation of BLAS:

1.1 Login from user ‘linpacker’.

1.2. Download GotoBLAS from TACC website http://www.tacc.utexas.edu/tacc-projects/gotoblas2

1.3. Untar it into /export/home/linpacker/
# tar -xvf GotoBLAS2-1.13.tar.gz

1.4. Change directory to the GoToBLAS2 directory
# cd /export/home/linpacker/GotoBLAS2

1.5. Build GoToBLAS2
# /export/home/linpacker/GotoBLAS2/quickbuild.64bit



If you face problem as above, follow step 1.7.

1.6. Run the MAKE command
# make

1.7. The compilation might fail. To resolve the issue, do the following:
# gmake clean
# gmake TARGET=NEHALEM



It will compile and will produce the required files.

After running gmake command, you don’t need to follow step 1.6.

2. HPL (Linpack Portion):

2.1. Change to home directory
# cd /export/home/linpacker

2.2. Download HPL from NetLib website http://www.netlib.org/benchmark/hpl/hpl-2.1.tar.gz

2.3. Untar it into /export/home/linpacker/  
# tar -xvf hpl-2.1.tar.gz

2.4. Create a Symbolic Link for the HPL directory
# ln -s /export/home/linpacker/hpl-2.1 /export/home/linpacker/hpl

2.5. Copy the Make.Linux_PII_FBLAS from the HPL setup directory to the HPL root
# cp /export/home/linpacker/hpl/setup/Make.Linux_PII_FBLAS /export/home/linpacker/hpl

2.6. Edit the Make.Linux_PII_FBLAS file with the following changes:
# vim /export/home/linpacker/hpl/Make.Linux_PII_FBLAS
o    TOPdir       = /export/home/linpacker/hpl
o    MPdir        = /opt/openmpi/
o    MPlib        = $(MPdir)/lib/libmpi.a
o    LAdir        = /export/home/linpacker/GotoBLAS2
o    LAlib        = $(LAdir)/libgoto2.a -lm -L/usr/lib/gcc/i386-redhat-linux/4.1.2
o    CC           = /opt/openmpi/bin/mpicc
o    LINKER       = /opt/openmpi/bin/mpif77

Save and Quit the Make.Linux_PII_FBLAS file.

2.7. Change to the HPL directory.
# cd hpl

2.8. Run the MAKE command specifying the architecture
# make arch=Linux_PII_FBLAS


3. Running benchmarking:
The benchmarking executable is ‘xhpl’.
Login from the linpacker user.

3.1. Make a backup of the HPL.dat file.
# cp /export/home/linpacker/hpl/bin/Linux_PII_FBLAS/HPL.dat /export/home/linpacker/hpl/bin/Linux_PII_FBLAS/HPL.dat.original

3.2. Create a file in your home directory named machines, and put two entries in it, such as:
# vim /export/home/linpacker/machines
compute-0-0
compute-0-1

3.3 To launch the job from frontend:
# ssh-agent $SHELL
# ssh-add
 
Change directory where the executable is present:
# cd /export/home/linpacker/hpl/Linux_PII_FBLAS/
 
And run the command:
# /opt/openmpi/bin/mpirun -np 6 -
machinefile/export/home/linpacker/machines/export/home/linpacker/hpl/bin/Linux_PII_FBLAS/xhpl


Note:
Just to check whether the ‘xhpl’ is being executed or not, simply run the command:
Before running the command, go to the path of the ‘xhpl’ file i.e.
# cd /export/home/linpacker/hpl/Linux_PII_FBLAS/
# mpirun –np 5 ./xhpl

To run the command on compute nodes, follow this:

1. The compute nodes should be installed after the completion of all the processes or installations on the master node so that all the changes could be installed on the compute nodes.

2. The executable ‘xhpl’ will be presented at the same path on the both master node and the compute node.

3. Simply run the command from master node from the non-root user:
# cd /export/home/linpacker/hpl/Linux_PII_FBLAS/
# mpirun –np 5 –machinefile /export/home/linpacker/machines  xhpl
OR
# mpirun –np 5 –H compute-0-1  xhpl


Note:
1. The HPL.dat file:
------------------------------------------------------------------------------------------------------------------
HPLinpack benchmark input file
Innovative Computing Laboratory, University of Tennessee
HPL.out      output file name (if any)
6            device out (6=stdout,7=stderr,file)
1            # of problems sizes (N)
1000 Ns
1            # of NBs
64 NBs
1            # of process grids (P x Q)
2 Ps
2 Qs
16.0         threshold
3            # of panel fact
0 1 2            PFACTs (0=left, 1=Crout, 2=Right)
1            # of recursive stopping criterium
8            NBMINs (>= 1)
1            # of panels in recursion
2            NDIVs
1            # of recursive panel fact.
2            RFACTs (0=left, 1=Crout, 2=Right)
1            # of broadcast
1            BCASTs (0=1rg,1=1rM,2=2rg,3=2rM,4=Lng,5=LnM)
1            # of lookahead depth
1            DEPTHs (>=0)
2            SWAP (0=bin-exch,1=long,2=mix)
80           swapping threshold
0            L1 in (0=transposed,1=no-transposed) form
0            U  in (0=transposed,1=no-transposed) form
1            Equilibration (0=no,1=yes)
8            memory alignment in double (> 0)
------------------------------------------------------------------------------------------------------------------

1 comment:

  1. First off I would like to say awesome blog! I had a quick question that
    I'd like to ask if you do not mind. I was curious to know how you center yourself and clear your thoughts prior to writing.
    I've had difficulty clearing my thoughts in getting my ideas
    out there. I do enjoy writing but it just seems like the first 10 to
    15 minutes tend to be wasted simply just trying to figure out how to begin. Any ideas or hints?
    Many thanks!

    ReplyDelete