From PRAGMA wiki
Jump to: navigation, search

Rocks 5.3, X86_64 virtual cluster globus application base-line system setup

(In this example, the cluster's head node is connected to public internet, and its 8 compute nodes are connected to the head node on a private net.)

Install rocks 5.3 in virtual cluster

  • Follow Rocks Xen roll documentation to install and start a virtual cluster. Make sure to install at least the following rolls:
    • base
    • kernel
    • os
    • area51
    • hpc
    • sge or pbs
    • web-server

Prepare for Globus installation

  • Prepare /etc/grid-security directory
    • create /etc/grid-security directory
    • create /etc/grid-security/grid-mapfile (empty is ok)
    • get host certificates for
    • copy the host certificate files to /etc/grid-security directory
    • make sure the file permissions are set correctly
-rw-r--r-- 1 root   root       0 Mar 30 12:46 grid-mapfile
-rw-r--r-- 1 root   root    1285 Mar 11 19:43 hostcert.pem
-r-------- 1 root   root     887 Mar 11 19:43 hostkey.pem
  • create globus user
# groupadd -g 500 globus
# useradd -c "Globus" -g globus -d /export/home/globus -m -u 500 globus
# rocks sync users
  • Enable SGE reporting
    • run "qconf -mconf global" and set "reporting=true"
    • create /opt/gridengine/default/common/reporting file with the following ownership and permission
-rw-r--r-- 1 sge sge 0 Mar 30 12:56 /opt/gridengine/default/common/reporting
  • Install pacman
# cd /opt
# wget
# tar xvzf pacman-latest.tar.gz 
# cd pacman-3.29/
# source

Install/setup VDT Globus

  • Install VDT packages
# mkdir /opt/vdt
# cd /opt/vdt
# pacman -pretend-platform linux-rhel-5
# pacman -allow trust-all-caches
# pacman -v all -get
# pacman -v all -get
# pacman -v all -get
# pacman -v all -get
  • add the following line in /opt/vdt/vdt/etc/vdt-update-certs.conf
cacerts_url =
  • run
# source /opt/vdt/
# vdt/sbin/vdt-setup-ca-certificates --certs-dir /etc/grid-security
  • Check the results in /etc/grid-security
  • Check the /opt/vdt/globus/TRUSTED_CA is a symbolic link to /etc/grid-security/certificates
  • Start all VDT services
# vdt-control --list
# vdt-control --enable fetch-crl
# vdt-control --enable vdt-rotate-logs
# vdt-control --enable vdt-update-certs
# vdt-control --enable globus-gatekeeper
# vdt-control --enable gsiftp
# vdt-control --on --force

Install/setup VDT Globus-WS

  • Install VDT Globus-WS
# cd /opt/vdt
# source ../pacman-3.29/
# source
# vdt-control --off
# pacman -v all -get
  • Add the following lines in /etc/sudoers
# VDT globus-ws
globus ALL=(ALL) NOPASSWD: /opt/vdt/globus/libexec/globus-gridmap-and-execute -g /etc/grid-security/grid-mapfile /opt/vdt/globus/libexec/ *
globus ALL=(ALL) NOPASSWD: /opt/vdt/globus/libexec/globus-gridmap-and-execute -g /etc/grid-security/grid-mapfile /opt/vdt/globus/libexec/globus-gram-local-proxy-tool *
  • Comment-out the "Defaults requiretty" line in /etc/sudoers
  • If you like to change the port# to standard default (8443)
    • Edit /etc/init.d/globus-ws, change port# 9443 to 8443
    • Edit /opt/vdt/post-install/globus-ws, change port# 9443 to 8443
  • Add VDT setup script to systemwide user environment setup
# cp /opt/vdt/ /etc/profile.d/
  • Start all VDT services
# vdt-control --list
# vdt-control --enable mysql5
# vdt-control --enable globus-ws
# vdt-control --list
# vdt-control --on

Install/setup Globus-SGE interface

  • Install VDT Globus-SGE-Setup
# pacman -get
# pacman -get
  • If SGE_QMASTER_PORT is not defined in /opt/vdt/globus/lib/perl/Globus/GRAM/JobManager/, back it up, then edit it according to the patch provided by AIST. But, note that the line numbers may not correspond to your version of file.
  • If the sge jobmanager script still have problem, try more modifications in
  • If encounter the error "$GLOBUS_LOCATION/libexec/globus-scheduler-provider-sge: not found" in $GLOBUS-LOCATION/var/container-real.log, create the file with the 2 lines:
echo "<Scheduler xmlns=\"\">"; 
echo "</Scheduler>"; 

and make the file executable using the command 'chmod + x globus-scheduler-provider-sge'.

Setup PRAGMA users accounts

# mkdir /root/setuser
# cd /root/setuser
# wget --no-check-certificate
# tar xvzf updaccounts.tar.gz
# mv updaccounts.txt
# source
# rocks sync users


  • Make sure that your DN is in the target system's gridmapfile
  • Logon to another system as an user, where
    • globus client software is installed
    • your globus certificate is setup (~/.globus)
  • Testing globus authentication, job submissions. See Test_Essential_Services