PRAGMA VOMRS resource site setup at SDSC

From PRAGMA wiki
Jump to: navigation, search

On a GUMS server

Install GUMS via VDT

This installation is on a rocks 4.2.1 with globus roll.

  • Make sure that globus (2 or 4) is working
  • Install pacman (for example, in /opt directory)
# wget http://physics.bu.edu/pacman/sample_cache/tarballs/pacman-latest.tar.gz
# cd /opt
# tar xvzf /root/pacman-latest.tar.gz
# ln -s pacman-3.26/ pacman
  • Install GUMS use VDT
# cd pacman
# source ./setup.sh
# cd ..
# mkdir vdt
# cd vdt
# export PATH=$PATH:/opt/pacman/bin
# pacman -get http://vdt.cs.wisc.edu/vdt_1101_cache:GUMS

If you answered "yes" for auto-update CA certificates, make sure to include PRAGMA non-IGTF CA certificates. See an example.

Also, make sure the /opt/vdt/globus/TRUSTED-CA exist. If not, create it as a symbolic link to your certificate directory. Then run "vdt-control --off" then vdt-control --on" to verify the installation.

Configure GUMS

  • Modify /opt/vdt/vdt-app-data/gums/config/gums.config file. (See SDSC's gums.config file as an example.)
  • Make sure that the http cert and key files are owned by daemon. If not, for example, do
# chown daemon.daemon /etc/grid-security/http/http*

Testing GUMS

  • Add yourself as GUMS admin
# cd $VDT_LOCATION/tomcat/v55/webapps/gums/WEB-INF/scripts/
# ./gums-add-mysql-admin "[your-DN-string]"
  • Go to https://[your-GUMS-server-FQDN]:8443/gums and check all the links on the page.

Install Auth Tool

If you use LDAP or NIS with PAM for local user authentication, you may want to follow the example at http://projects.arcs.org.au/trac/systems/wiki/HowTo/InstallAuthTool.

The installation below is done by following the example at BeSTGrid.

Install authtool package

# wget http://www.vpac.org/~sam/authtool.tar.gz
# cd /opt/vdt/apache/htdocs
# tar xvzf /root/authtool.tar.gz
# mv authtool hpc
# chown -R daemon hpc
# chgrp -R daemon hpc

Build and install authtool external module

# wget http://www.unixpapa.com/software/mod_authnz_external-3.1.0.tar.gz
# tar xvzf mod_authnz_external-3.1.0.tar.gz
# source /opt/vdt/setup.sh
# which apxs
/opt/vdt/apache/bin/apxs
# apxs -c mod_authnz_external.c
# apxs -i -a mod_authnz_external.la

Setup Expect script

  • Rocks systems should have Expect installed already. To check, do
# rpm -qa | grep expect
  • In case Expect has not been installed, find and install Rocks Expect rpm package: (Install Expect using YUM is NOT recommended on Rocks systems, because many rpms, especially those distributed with base and os rolls, have inter-dependencies.)
# locate expect | grep /home/install
# rpm -i <path-to-expet-rpm>.rpm
AddExternalAuth sshauth-hpc /opt/vdt/apache/bin/sshauth-hpc
SetExternalAuthMethod sshauth-hpc pipe
<Directory "/opt/vdt/apache/htdocs/hpc/auth">
        AllowOverride AuthConfig
</Directory>
  • Customize /opt/vdt/apache/htdocs/hpc/auth/.htaccess
    • Change AuthName to your own site name
    • Change AuthExternal to use sshauth-hpc

For example:

# cat /opt/vdt/apache/htdocs/hpc/auth/.htaccess
AuthType Basic
AuthName "Grid Authorization Tool - PRAGMA Grid SDSC site"
AuthBasicProvider external
AuthExternal sshauth-hpc
require valid-user
SSLVerifyClient require
  • Customize /opt/vdt/apache/htdocs/hpc/auth/index.php, change "vpac" to your own site name.
  • Copy the key in /etc/ssh/ssh_host_rsa_key.pub from each gateway system into /etc/ssh/ssh_known_hosts file on the GUMS server.

Mapfile-GUMS synchronization

For more detail explaination, please see http://www.bestgrid.org/index.php/Setup_AuthTool_for_HPC_at_University_of_Canterbury.

  • Download gumsmanualmap.py
# cd /opt/vdt/apache/bin
# wget wget http://projects.gridaus.org.au/trac/systems/attachment/wiki/HowTo/InstallAuthTool/gumsmanualmap.py.txt?format=raw
# mv gumsmanualmap.py.txt\?format\=raw gumsmanualmap.py
# cp gumsmanualmap.py gumsmanualmap.py.orig
  • Modify /opt/vdt/apache/bin/gumsmanualmap.py with the following changes
# diff gumsmanualmap.py*
53c53
< MAPFILE = "/opt/vdt/apache/htdocs/mapfile/mapfile"
---
> MAPFILE = "/opt/vdt/apache/htdocs/hpc/mapfile/mapfile"
56c56,58
< DBHOST = "localhost"
---
> DBHOST = "rocks56.sdsc.edu"
> DBPORT = 49151
> ### ORIG: DBPORT = 3306
60,61c62,63
< MAPPEDUSERS = 'mappedUsers'
< MANUALGROUP = 'manualGroup'
---
> MAPPEDUSERS = 'mappedUsersHPC'
> MANUALGROUP = 'manualGroupHPC'
131c133
<             # Check that local account name exists
---
>             # HPC: DISABLE: VM 2008-12-18 Check that local account name exists
133,137c135,139
<             status, output = commands.getstatusoutput("getent passwd %s" % localAccount)
<             if status != 0:
<                 log.warning('-- %3d:Local account for %s does not exist' \
<                              % (lineno, mapEntry))
<                 invalidAccount = 1
---
> #            status, output = commands.getstatusoutput("getent passwd %s" % localAccount)
> #            if status != 0:
> #                log.warning('-- %3d:Local account for %s does not exist' \
> #                             % (lineno, mapEntry))
> #                invalidAccount = 1
179c181
<         db = MySQLdb.Connect(host=DBHOST, port=3306, user=DBUSER, passwd=getGumsConfigPasswd(), db=DBNAME)
---
>         db = MySQLdb.Connect(host=DBHOST, port=DBPORT, user=DBUSER, passwd=getGumsConfigPasswd(), db=DBNAME)
  • Create /etc/cron.d/gumsmanualmap-hpc.cron with this entry
*/8 * * * * daemon /opt/vdt/apache/bin/gumsmanualmap-hpc.py >> /opt/vdt/apache/logs/gumsmanualmap-hpc.log 2>&1
  • Create log file
# touch /opt/vdt/apache/logs/gumsmanualmap-hpc.log
# chown daemon.daemon /opt/vdt/apache/logs/gumsmanualmap-hpc.log
  • Test the script by manually run it
# sudo -u daemon /opt/vdt/apache/bin/gumsmanualmap-hpc.py
  • Restart crond
# service crond restart

On Gateway Systems

If your situation allow, the best way would be to upgrade your rocks cluster to version 5.x (there is no grid roll for Rocks 5.x), then install Globus and PRIMA all from VDT. (We hope to provide an example soon.)

If for whatever reason you can't upgrade to Rocks 5.x yet, here are 3 implementation examples for Rocks 4.x systems:

  1. On a rocks 4.2.1 system with grid roll installed, reinstall GT4 pre-WS and WS and PRIMA all through VDT. See Method 1.
  2. On a rocks 4.1 system, install VDT PRIMA for GT4 pre-WS by rocks globus roll. See Method 2.
  3. On a rocks 4.2.1 system, install GT4 WS and PRIMA-GT4 from VDT and keep the previous GT4 pre-WS by rocks globus roll. This does not work with rocks 4.1 system, but may work on newer versions of rocks systems. See Method 3.

Method 1: Install Globus/WS/PRIMA all from VDT (recommended)

  • Step 1. Disable current globus/grid-ftp/gris if any is running
# service globus stop
# service globus-ws stop
# move /etc/init.d/globus <path-to-backup-dir>
# move /etc/init.d/globus-ws <path-to-backup-dir>

Comment-out the lines for gsigatekeeper, gsiftp, gris in /etc/services files.

# move /etc/xinet.d/globus-gatekeeper <path-to-backup-dir>
# move /etc/xinet.d/grid-ftp <path-to-backup-dir>
# /etc/init.d/xinetd restart
  • Step 2. Remove but save any system profile for globus
# mv /etc/profile.d/globus.sh <path-to-backup-dir>
# mv /etc/profile.d/globus.csh <path-to-backup-dir>
  • Step 3. Logoff and re-login

To get a clean environment setup.

  • Step 4. Create user accounts for the PRAGMA VO groups

For an example in corresponding to the GUMS configuration file, create the following UNIX user accounts.

# adduser g-avian
# adduser g-pragma
# adduser g-eairs
# adduser g-nimrod
# adduser g-prime
  • Step 5. Unlock the new user accounts

To allow users gsissh to the group accounts, unlock the above user accounts by replace "!!" to "*" in the password field of /etc/shadow.

  • Step 6. Install pacman (for example in /opt)
# wget http://physics.bu.edu/pacman/sample_cache/tarballs/pacman-latest.tar.gz
# cd /opt
# tar xvzf /root/pacman-latest.tar.gz
# ln -s pacman-3.26/ pacman
  • Step 7. Install GLOBUS
# cd pacman
# source ./setup.sh
# cd ..
# mkdir vdt
# cd vdt
# pacman -get http://vdt.cs.wisc.edu/vdt_1101_cache:Globus

This example is based on answering "yes" to all questions during the Globus installation, except "No" for install certificates.

  • Step 8. Post-install setup
# cd /opt/vdt
# . setup.sh

See post-install/README for what need to be done. Here are some example actions:

  • Step 9. If you answered "yes" for auto-update CA certificates and would like to use IGTF+PRAGMA combo distribution, add the line below in /opt/vdt/vdt/etc/vdt-update-certs.conf
cacerts_url = http://rocks56.sdsc.edu/certs/igtf-pragma-ca-certs-version
  • Step 10. run
# . /opt/vdt/vdt-questions.sh
# /opt/vdt/sbin/vdt-setup-ca-certificates --root
  • Step 11. Make sure the /opt/vdt/globus/TRUSTED-CA is a symbolic link which points to the desired certificates directory. If not, create it.
  • Step 12. Install PRIMA-GT4
# cd pacman
# source ./setup.sh
# cd ..
# mkdir vdt
# cd vdt
# pacman -get http://vdt.cs.wisc.edu/vdt_1101_cache:PRIMA-GT4

This example is based on answering "yes" to all questions by the PRIMA-GT4 installation.

  • Step 13. Add the following lines in /etc/sudoers
Runas_Alias GLOBUSUSERS = user1, user2 
# VDT globus-ws
globus ALL=(GLOBUSUSERS) NOPASSWD: /opt/vdt/globus/libexec/globus-gridmap-and-execute -g /etc/grid-security/grid-mapfile /opt/vdt/globus/libexec/globus-job-manager-script.pl *
globus ALL=(GLOBUSUSERS) NOPASSWD: /opt/vdt/globus/libexec/globus-gridmap-and-execute -g /etc/grid-security/grid-mapfile /opt/vdt/globus/libexec/globus-gram-local-proxy-tool *

# VDT PRIMA-GT4
globus ALL=(GLOBUSUSERS) NOPASSWD: /opt/vdt/globus/libexec/globus-job-manager-script.pl *
globus ALL=(GLOBUSUSERS) NOPASSWD: /opt/vdt/globus/libexec/globus-gram-local-proxy-tool *
  • Step 14. Configure PRIMA-GT4

In file /opt/vdt/post-install/prima-authz.conf, change the hostname in "imsContact" line to the FQDN of your GUMS server.

# cp /opt/vdt/post-install/gsi-authz.conf /etc/grid-security
# cp /opt/vdt/post-install/prima-authz.conf /etc/grid-security
# /opt/vdt/vdt/setup/configure_prima_gt4 --enable --gums-server <your-gums-server-FQDN>
  • Step 15. Setup jobmanager for batch system (This example is for SGE)
# cd /opt/vdt
# pacman -get http://vdt.cs.wisc.edu/vdt_1101_cache:Globus-SGE-Setup
pacman -get http://vdt.cs.wisc.edu/vdt_1101_cache:Globus-WS-SGE-Setup
  • Step 16. If SGE_QMASTER_PORT is not defined in /opt/vdt/globus/bin/perl/Globus/GRAM/JobManager/sge.pm, back it up, then edit it according to the patch provided by AIST. But, note that the line numbers may not correspond to your version of sge.pm file.
  • Step 17. Make sure all VDT components are enabled
# vdt-control --list

If not, enable it with "vdt-control --enable <component-name>".

  • Step 18. Activate and verify all VDT components
# vdt-control --on
  • Step 19. Add VDT environment setup script to system profiles
# cp /opt/vdt/setup.sh /etc/profile.d/vdt.sh
  • Step 20. Test Globus and PRIMA

(If you encounter problems, you may want to test Globus and PRIMA seperately. To disable PRIMA temporarily, create a sub-directory under /etc/grid-security and move the gsi-authz.conf and prima-authz.conf to the sub-directory. To reenable it later, just move these 2 files back to /etc/grid-security.)
Login an user account with Globus certificates properly setup. Test pre-WS:

$ grid-proxy-init
$ globusrun -a -r <the-gateway-system-FQDN>
$ globus-job-run <the-gateway-system-FQDN>/jobmanager-sge /bin/date

Test WS: (Note - VDT sets container port# to 9443)

$ globusrun-ws -submit -factory <the-gateway-system-FQDN>:9443 -Ft SGE -J -S -s -c /bin/date
  • Step 21. If you like to change the port# to standard default (8443)
    • Edit /etc/init.d/globus-ws, change port# 9443 to 8443
    • Edit /opt/vdt/post-install/globus-ws, change port# 9443 to 8443
    • Restart /etc/init.d/globus-ws
    • Test job submission without specifying port#
$ globusrun-ws -submit -factory <the-gateway-system-FQDN> -Ft SGE -J -S -s -c /bin/date

Method 2: Install PRIMA on a Rocks system with GT4 pre-WS only (complete)

  • Create UNIX accounts for the groups. For example,
# adduser g-avian
# adduser g-pragma
# adduser g-eairs
# adduser g-nimrod
# adduser g-prime
  • Unlock the new user accounts

To allow users gsissh to the group accounts, unlock the above user accounts by replace "!!" to "*" in the password field of /etc/shadow.

  • Install pacman (for example in /opt)
# wget http://physics.bu.edu/pacman/sample_cache/tarballs/pacman-latest.tar.gz
# cd /opt
# tar xvzf /root/pacman-latest.tar.gz
# ln -s pacman-3.26/ pacman
  • Install GLOBUS from VDT

But answer NO on enable GRID and GLOBUS - we only want to use the globus libraries

# cd pacman
# source ./setup.sh
# cd ..
# mkdir vdt
# cd vdt
# pacman -get http://vdt.cs.wisc.edu/vdt_1101_cache:Globus-Base-RM-Server

If you answered "yes" for auto-update CA certificates, make sure to include PRAGMA non-IGTF CA certificates. See an example.
Also, make sure the /opt/vdt/globus/TRUSTED-CA exist. If not, create it as a symbolic link to your certificate directory. Then run "vdt-control --off" then vdt-control --on" to verify the installation.

  • Install PRIMA
# cd /opt/vdt
# . setup.sh
# pacman -get http://vdt.cs.wisc.edu/vdt_1101_cache:PRIMA
  • Enable PRIMA

In file /opt/vdt/post-install/prima-authz.conf, change the hostname in "imsContact" line to the hostname of your GUMS server. Then

# cp /opt/vdt/post-install/gsi-authz.conf /etc/grid-security
# cp /opt/vdt/post-install/prima-authz.conf /etc/grid-security
  • Modify globus-gatekeeper and grid-ftp environment setup
    • Backup then edit /etc/xinetd.d/globus-gatekeeper and /etc/xinetd.d/grid-ftp
      • add /opt/vdt/prima/lib:/opt/vdt/globus/lib to LD_LIBRARY_PATH.
    • Restart xinetd.d
# service xinetd restart
  • Create an user enviroment setup script
    • Copy /opt/vdt/setup.sh to /opt/vdt/prima.sh
    • Modify /opt/vdt/prima.sh
      • comment-out all settings (sections) for the components not used in VDT, in this case, is globus.
      • add /opt/vdt/globus/lib to LD_LIBRARY_PATH
    • Here is an example of prima.sh.
    • Copy prima.sh to /etc/profile.d
  • Restart globus-ws
# service globus restart
  • Testing PRIMA

Login to an user account with certificates properly installed. Then run

$ grid-proxy-init
$ globusrun -a -r <your-host-FQDN>
$ globus-job-run <your-host-FQDN>/jobmanager /bin/hostname

Method 3: Install PRIMA-GT4 on a Rocks system with GT4 WS configured (complete)

WARNING: You may want to test PRIMA on a non-production system first, or during a scheduled maintenance time on a production system. To disable PRIMA, simply rename /etc/grid-security/gsi-authz.conf and restart Globus.

  • Create UNIX accounts for the groups
# adduser g-avian
# adduser g-pragma
# adduser g-eairs
# adduser g-nimrod
# adduser g-prime
  • Unlock the new user accounts

To allow users gsissh to the group accounts, unlock the above user accounts by replace "!!" to "*" in the password field of /etc/shadow.

  • Install PRIMA

Install pacman if it's not installed yet (See 1.1 for pacman install)

# cd /opt/pacman
# source ./setup.sh
# cd ..
# mkdir vdt
# cd vdt
# pacman -get http://vdt.cs.wisc.edu/vdt_1101_cache:PRIMA-GT4

You will be asked various questions, including whether or not to install globus-ws, gridftp and mysql. In this example, all three exist and working. If so, answer no.
If you answered "yes" for auto-update CA certificates, make sure to include PRAGMA non-IGTF CA certificates. See an example.
Also, make sure the /opt/vdt/globus/TRUSTED-CA exist. If not, create it as a symbolic link to your certificate directory. Then run "vdt-control --off" then vdt-control --on" to verify the installation.

  • Enable PRIMA

In file /opt/vdt/post-install/prima-authz.conf, change the hostname in "imsContact" line to the hostname of your GUMS server. Then

# source ./setup.sh
# /opt/vdt/vdt/setup/configure_prima_gt4 --enable --gums-server rocks56.sdsc.edu
  PRIMA for GT4 web services has been enabled.
  Modifications to the /etc/sudoers file are still required.
  You will need to restart the /etc/init.d/globus-ws container to effect the changes.
  • Create an user enviroment setup script
    • Copy /opt/vdt/setup.sh to /opt/vdt/prima-setup.sh
    • Modify /opt/vdt/prima-setup.sh
      • comment-out all settings (sections) for the components not used in VDT, in this case, they are globus, gridftp, mysql, ant, java
      • Modify the 2 lines in the section near the end
LD_LIBRARY_PATH="/opt/vdt/prima/lib:$LD_LIBRARY_PATH"
...
export LIBPATH=/opt/vdt/prima/lib:$LIBPATH

to these (add the path /opt/vdt/globus/lib)

LD_LIBRARY_PATH="/opt/vdt/globus/lib:/opt/vdt/prima/lib:$LD_LIBRARY_PATH"
...
export LIBPATH=/opt/vdt/globus/lib:/opt/vdt/prima/lib:$LIBPATH
    • Here is an example prima-setup.sh.
    • Copy prima-setup.sh to /etc/profile.d
  • Restart globus-ws
# service globus-ws stop
# service globus-ws start
  • Testing PRIMA

Login to an user account with certificates properly installed. Then run

$ grid-proxy-init
$ globusrun-ws -submit -factory <your-host-FQDN> -Ft SGE -J -S -s -c /bin/hostname

On Client Systems

These are the systems where users may login and launch applications to local or remote grids.

Install VOMS client

  • Install pacman if it's not installed yet (See 1.1 for pacman install). If /opt/vdt does not exist yet, create it. Then
# cd /opt/pacman
# source ./setup.sh
# cd ../vdt
# pacman -get http://vdt.cs.wisc.edu/vdt_1101_cache:VOMS-Client
  • Create /etc/grid-security/vomsdir/vomses
# cat /etc/grid-security/vomsdir/vomses 
"PRAGMA" "vomrs-pragma.sdsc.edu" "15001" "/DC=NET/DC=PRAGMA-GRID/OU=SDSC/CN=vomrs-pragma.sdsc.edu" "PRAGMA"
  • Create /etc/profile.d/voms-client.sh
# cat /etc/profile.d/voms-client.sh 
export VOMS_USERCONF=/etc/grid-security/vomsdir/vomses
export VOMS_LOCATION=/opt/vdt/glite
export PATH=/opt/vdt/glite/bin:$PATH
LD_LIBRARY_PATH="/opt/vdt/glite/lib:/opt/vdt/prima/lib:$LD_LIBRARY_PATH"
export LIBPATH=/opt/vdt/glite/lib:/opt/vdt/prima/lib:$LIBPATH

Test VOMS client

Login to an user account with globus certificates properly installed, for example - if you are a member of /PRAGMA/USERS group, to create a proxy as this group member, run

$ voms-proxy-init -voms PRAGMA -order /PRAGMA/USERS

For more testing examples, see http://www.bestgrid.org/index.php/Configuring_BeSTGRID_systems_to_accept_PRAGMA_users#Using_the_system

Install GSISSH

See http://grid.ncsa.uiuc.edu/ssh/install.html