Amber MPI execution on Gfarm

From PRAGMAgridWIKI

Jump to: navigation, search

Contents

[edit]

Goal

A user places an amber-mpi executable and an input data file on the Gfarm filesystem once, and then starts the amber-mpi on several clusters. The output file will be stored on the Gfarm filesystem, too. A user does not worry about file transfer between clusters before/after the execution.

[edit]

Met problem

The amber-mpi could not be executed on the rocks-52 cluster because user proxy cert was not delegated by mpirun.

[edit]

Solution

The following three items must be satisfied.

  1. In order to start amber-mpi executables on the Gfarm filesystem, a user must submit an amber-mpi job through Globus GRAM. Globus's job_type must be "mpi."
  2. The amber-mpi executable must be started by gfarmfs-exec.sh, which automatically mounts/unmounts the Gfarm filesystem before/after the amber-mpi run.
  3. mpirun command must propagate the environment variable, X509_USER_PROXY, to the all launched processes on the compute nodes. So far, it seems that mpirun command of MPICH does not have the function. We are sure GridMPI has the function.
[edit]

Work on Resource-WG in PRAGMA 12

We decided to recompile the amber-mpi with GridMPI on the AIST X cluster and demonstrate a job submission from the SDSC rocks-52 node to the X cluster.

First, we logined to xsvc.asc.hpcc.jp for compilation of amber-mpi.

Entered the amber source directory and edited config.h to add header file path and library path for GridMPI. 

 $ cd amber8/src
 $ vi config.h

Added these paths in config.file. 

 For header file: -I/usr/local/gridmpi/include
 For library: -L/usr/local/gridmpi/lib -lmpif -lmpi

Built the amber mpi. 

 $ make clean
 $ make parallel

Copied the executable, sander, to the Gfarm filesystem. 

 $ grid-proxy-init
 $ gfreg amber8/exe/sander gfarm:~/sander

Then we logined to the rocks-52.sdsc.edu. The usual run command, 

 $ mpirun -np 4 ./sander -O -i gbin -o output1

was described in the RSL file (sander_x.rsl) as follows. Note that gfarmfs-exec.sh must be specified as an executable and main program (sander) must be specified in arguments. job_type must be mpi. 

&(executable="/home/habibah/gfarmfs-exec.sh")
 (arguments="-wdir benchmark_amber/parallel ./sander -O -i gbin -o output1")
 (job_type=mpi)
 (count=4)

Submitted the request from rocks-52.sdsc.edu to the X cluster. xsvc.asc.hpcc.jp is a job submission node of the X cluster. 

 $ grid-proxy-init
 $ globusrun -r xsvc.asc.hpcc.jp -f sander_x.rsl

Checked the result file on the Gfarm filesystem. 

 $ gfarmfs -lsu ~/gfs
 $ cd ~/gfs/habibah/habibah/benchmark_amber/parallel/
 $ less outptu1
Retrieved from "http://goc.pragma-grid.net/wiki/index.php/Amber_MPI_execution_on_Gfarm"
Personal tools