Gfarm roll user guide

From PRAGMA wiki
Jump to: navigation, search

You can install gfarm roll at the initial build of a Rocks 5.4 x86_64 system, or to add it on a running Rocks 5.4 x86_64 system.

Read this before decide to install or not (md5sum=3c311bc6bd85eae703a440aa61a0aa9f) is a Rocks Gfarm roll. This roll is based on Gfarm 2.4.2 release and for Rocks 5.4 (Maverick) x86_64 systems.
In order to install and setup GSI-enabled Gfarm system, this Gfarm roll also does the following:

  • On a single node system or a cluster frontend
    • install VDT Globus and Globus-Base-SDK
    • setup certificate auto-update from PRAGMA certificates distribution point (including IGTF certificates distribution)
    • setup CRL auto-update
    • enable certificate files synchronization for compute nodes
  • On a cluster compute node
    • install VDT Globus and Globus-Base-SDK
    • setup certificate files synchronization with its frontend

The above info is to enable you to determine if there is any conflict with your existing system setups. So you can make a sound decision on whether of not this roll is suitable to your system.

Add gfarm roll to your frontend or single node system

For example:

# rocks add roll gfarm-1.0-2.x86_64.disk1.iso
# rocks enable roll gfarm
# cd /export/rocks/install
# rocks create distro
# rocks run roll gfarm | bash
  • it's best to reboot the system at this point. If you are installing the node as a gfarm meta server of a gfarm file system server, then you must reboot. But if you are installing gfarm client ONLY on a frontend and would like to avoid rebooting the system, do the following manually:
# source /opt/vdt/
# echo "export PATH=/opt/gfarm/bin:/opt/gfarm2fs/bin:$PATH" > /etc/profile.d/
# cp /opt/vdt/ /etc/profile.d/
# export PATH="/opt/gfarm/bin:$PATH"
# mkdir -p /etc/grid-security
# vdt-control --on
# vdt-update-certs --force
# /bin/tar -chf /etc/grid-security/certs.tar /etc/grid-security/certificates
# echo "FILES += /etc/grid-security/certs.tar /opt/gfarm/etc/gfarm2.conf" >> /var/411/
# rocks sync users
# /opt/rocks/bin/rocks add firewall global=global action=ACCEPT chain=INPUT protocol=udp service=600 network=all rulename=A100-GFARM-UDP-600
# /opt/rocks/bin/rocks add firewall global=global action=ACCEPT chain=INPUT protocol=tcp service=600 network=all rulename=A100-GFARM-TCP-600
# /opt/rocks/bin/rocks add firewall global=global action=ACCEPT chain=INPUT protocol=tcp service=601 network=all rulename=A100-GFARM-TCP-601
# chmod o+rx /bin/fusermount
# rm /etc/rc.d/rocksconfig.d/post-*-gfarm-*

Gfarm setup on the single node system or a frontend

  • Obtain/install host certificate (note the file permission settings)
# ls -l /etc/grid-security/host*
-rw-r--r-- 1 root root 1241 Aug 15 13:51 /etc/grid-security/hostcert.pem
-r-------- 1 root root  891 Aug 15 13:51 /etc/grid-security/hostkey.pem
  • If the system is a gfarm metaserver
# chkconfig gfarm-pgsql on
# chkconfig gfmd on
# /etc/init.d/gfarm-pgsql start
# /etc/init.d/gfmd start
  • If the system is not a gfarm metaserver, get gfarm2.conf from gfarm metaserver, install in /opt/gfarm/etc directory (note the file permission)
# ls -l /opt/gfarm/etc/gfarm2.conf
-rw-r--r-- 1 root root 200 Aug 15 16:59 /opt/gfarm/etc/gfarm2.conf
  • If the system is a gfarm file system node
    • Obtain gfsd certificate. Make sure the CN is in the form of gfsd/your.gfarm.file.server.fqdn. For example: "/O=grid/O=pragma/OU=SDSC/CN=gfsd/" @host@
    • Send the following info to your Gfarm metaserver administrator to register your file system node
      • system architecture (for example: x86_64)
      • number of cores in your system (for example: 4)
      • hostname (for example:
      • DN string for gfsd (from gfsd certificate)
    • Install gfsd certificate files (note the file name, ownership and permissions) and set gfsd start-up
# ls -ld /etc/grid-security/gfsd/
drwxr-xr-x 2 root root 4096 Aug 15 14:02 /etc/grid-security/gfsd/
# ls -l /etc/grid-security/gfsd/
total 8
-rw-r--r-- 1 _gfarmfs _gfarmfs 1249 Aug 16 15:41 gfsdcert.pem
-r-------- 1 _gfarmfs _gfarmfs  887 Aug 16 15:41 gfsdkey.pem
# chkconfig --add gfsd
# /etc/init.d/gfsd start
  • Setup /etc/grid-security/grid-mapfile to include DNs for gfarm metaserver, gfarm users. For example
"/DC=NET/DC=PRAGMA-GRID/OU=SDSC/CN=Cindy Zheng" zhengc
"/C=JP/O=AIST/OU=GRID/CN=Osamu Tatebe" tatebe
"/O=grid/O=pragma/OU=SDSC/CN=gfsd/" @host@
"/DC=NET/DC=PRAGMA-GRID/OU=SDSC/CN=Nadya Williams" nadya
"/C=JP/O=AIST/OU=GRID/CN=Yoshio Tanaka" yoshio
"/O=grid/O=pragma/OU=LZU/CN=Zhyang" zhyang
"/C=JP/O=AIST/OU=GRID/CN=Akihiko Ota" ota
"/DC=NET/DC=PRAGMA-GRID/OU=SDSC/CN=Philip Papadopoulos" phil
"/DC=TW/DC=ORG/DC=NCHC/O=NCHC/OU=STD/CN=hsing/USERID=hsing" serena
"/O=JP/OU=Osaka University/OU=Cybermedia Center/ User" osakauser
"/O=grid/O=pragma/OU=Indiana University/CN=Yuan Luo" yuanluo
"/O=grid/O=pragma/OU=Indiana University/CN=gfsd/" @host@
"/O=grid/O=pragma/OU=LZU/CN=gfsd/" @host@
"/DC=CN/DC=Grid/DC=SDG/CN=Kejun Dong (" kevin
"/DC=CN/DC=Grid/DC=SDG/CN=gfsd/" @host@
"/O=JP/OU=Osaka University/OU=Cybermedia Center/CN=gfsd/" @host@

Gfarm client installation on compute nodes

  • After gfarm roll is installed on the frontend and the gfarm setup on the frontend in above section is completed, reinstall compute nodes
  • After the compute nodes are reinstalled and boot up, on the frontend run
# rocks sync users
# rocks run host "cd /; tar xf /etc/grid-security/certs.tar"


If you are a Gfarm metaserver administrator

  • Create users and groups
  • Register gfarm system nodes
  • Setup gfarm file system structures
  • Create users gfarm home directories

If you are NOT a Gfarm metaserver administrator

  • Ask your Gfarm metaserver administrator to setup and grant you access to Gfarm file system


(Run all tests in your user account on the newly setup system)

log_level debug
log_auth_verbose enable

then restart gfsd, and see /var/log/messages.