SDSC Gfarm file server

From PRAGMA wiki
Jump to: navigation, search

Gfarm has released newer versions since then. Please use the most recent version of Gfarm.

Pre-installation setups

  • Install postgresql
# rpm -ivh /export/rocks/install/rocks-dist/x86_64/RedHat/RPMS/postgresql84-*8.4.7-1.el5_6.1.x86_64.rpm
  • Get host certificate for gfarm-fs.rocksclusters.org and install the host certificate files in /etc/grid-security directory
  • Install pacman
# wget http://physics.bu.edu/pacman/sample_cache/tarballs/pacman-latest.tar.gz
# cd /opt
# tar xvzf /root/pacman-latest.tar.gz
# ln -s pacman-3.29/ pacman
  • Install GLOBUS

Gfarm only needs Globus libraries. We install VDT Globus and Globus-Base-SDK packages to easily setup certificates and CRL auto-updates.

# cd pacman
# source ./setup.sh
# cd ..
# mkdir vdt
# cd vdt
# pacman -get http://vdt.cs.wisc.edu/vdt_200_cache:Globus
# pacman -get http://vdt.cs.wisc.edu/vdt_200_cache:Globus-Base-SDK
# . setup.sh
# vdt-ca-manage setupca --location root --url vdt

Edit /opt/vdt/vdt/etc/vdt-update-certs.conf and add the following line

cacerts_url=http://ca.pragma-grid.net/dist/igtf-pragma-ca-certs-version

Test the certificate update

# vdt-update-certs --force
  • Make sure the /opt/vdt/globus/TRUSTED-CA is a symbolic link which points to the desired certificates directory. If not, create it.
  • Enable some VDT services
# vdt-control --enable fetch-crl
# vdt-control --enable vdt-rotate-logs
# vdt-control --enable vdt-update-certs
# vdt-control --list
Service                 | Type   | Desired State
------------------------+--------+--------------
fetch-crl              | cron    | enable
vdt-rotate-logs        | cron    | enable
vdt-update-certs       | cron    | enable
globus-gatekeeper      | inetd   | do not enable
gsiftp                 | inetd   | do not enable
# vdt-control --on

Install Gfarm Software

  • Download and install source rpm
# wget http://sourceforge.net/projects/gfarm/files/gfarm_v2/2.4.2/gfarm-2.4.2.tar.gz
# tar xzvf gfarm-2.4.2.tar.gz
# cd gfarm-2.4.2
# ./configure --prefix=/opt/gfarm2.4.2 --with-openldap=/usr --with-postgresql=/usr --with-openssl=/usr --with-globus=/opt/vdt/globus --with-globus-flavor=gcc64 --enable-xmlattr
...
...
checking openssl... configure: error: OpenSSL cannot be linked with "-I/opt/vdt/globus/include/gcc64 
-R/opt/vdt/globus/lib -L/opt/vdt/globus/lib -lssl_gcc64 -lcrypto_gcc64" options, aborted

If you receive this error shown above, find or install 64-bit libssl.so and libcrypto.so and link them into the globus lib directory.

# cd /opt/vdt/globus/lib
# ln -s /usr/lib64/libssl.so libssl_gcc64.so
# ln -s /usr/lib64/libcrypto.so libcrypto_gcc64.so
# cd ~/gfarm-2.4.2
# ./configure --prefix=/opt/gfarm2.4.2 --with-openldap=/usr --with-postgresql=/usr --with-openssl=/usr --with-globus=/opt/vdt/globus --with-globus-flavor=gcc64 --enable-xmlattr
using lib/gfs_hook/sysdep/linux directory for system dependent objects.
using gftool/config-gfarm/sysdep/linux directory for system dependent part of config-gfarm.
checking use of private source directory... not use.
configure: creating ./config.status
config.status: creating Makefile
config.status: creating makes/config.mk
config.status: WARNING:  'makes/config.mk.in' seems to ignore the --datarootdir
setting
config.status: creating makes/install-doc
config.status: creating gftool/config-gfarm/config-gfarm
config.status: WARNING:  'gftool/config-gfarm/config-gfarm.in' seems to ignore the --datarootdir setting
config.status: creating gftool/config-gfarm/config-gfsd
config.status: WARNING:  'gftool/config-gfarm/config-gfsd.in' seems to ignore the --datarootdir setting
config.status: creating gftool/config-gfarm/config-gfarm-update
config.status: WARNING:  'gftool/config-gfarm/config-gfarm-update.in' seems to ignore the --datarootdir setting
config.status: creating gftool/gfdump/gfdump.postgresql
config.status: WARNING:  'gftool/gfdump/gfdump.postgresql.in' seems to ignore the --datarootdir setting
config.status: creating package/solaris/gfmd
config.status: creating package/solaris/gfsd
config.status: creating include/gfarm/gfarm_config.h
# make
# make install
  • Add user environment setups
# cp /opt/vdt/setup.sh /etc/profile.d/globus.sh
# vi /etc/profile.d/gfarm.sh
# cat /etc/profile.d/gfarm.sh
export PATH="/opt/gfarm2.4.2/bin:$PATH"
  • logout, then login, and make sure that Globus environment variables and Gfarm path are setup correctly in user environment.

Configure Gfarm file system node

  • Create an account for gfsd
# useradd -c "Gfarm gfsd" _gfarmfs
  • Open TCP and UDP port 600 and 601

Edit /etc/sysconfig/iptables and add these 3 lines

-A INPUT -p tcp --dport 600 -j ACCEPT
-A INPUT -p udp --dport 600 -j ACCEPT
-A INPUT -p tcp --dport 601 -j ACCEPT

Then reload iptables

# service iptables restart
  • Obtain a service certificate for gfsd (CN=gfsd/your.gfarm.file.server.fqdn)
  • Install gfsd certificate files as
# ls -l /etc/grid-security/gfsd
total 8
-rw-r--r-- 1 _gfarmfs _gfarmfs 1371 Jun  6 16:18 gfsdcert.pem
-r-------- 1 _gfarmfs _gfarmfs  887 Jun  6 16:18 gfsdkey.pem
  • Add gfsd DN in grid-mapfile as
"/DC=NET/DC=PRAGMA-GRID/OU=SDSC/CN=gfsd/your.gfarm.file.server.fqdn" @host@ your.gfarm.file.server.fqdn
  • Get gfarm2.conf from Gfarm meta-server administrator and install it in /opt/gfarm2.4.2/etc directory
  1. ls -l /opt/gfarm2.4.2/etc

total 8 -rw-r--r-- 1 root root 200 Jul 28 11:26 gfarm2.conf

  • Setup Gfarm spool directory

In this case, we have an external storage mounted on /gfarmdata for gfarm use. Alternatively, it can be a local disk. Simply specify the directory path where you want the Gfarm files reside.

# config-gfsd /gfarmdata
created /etc/init.d/gfsd
config-gfsd success

Please ask admin_user to register your host by the following command:

/opt/gfarm2.4.2/bin/gfhost -c -a x86_64-rocks5.4-linux -p 600 -n 4 your.gfarm.file.server.fqdn

After that, start gfsd by the following command as a root:

/etc/init.d/gfsd start
  • Send the above output to Gfarm meta-server administrator to register your file system node
  • After your Gfarm meta-server administrator registered your file system node, start gfsd
# /etc/init.d/gfsd start
# chkconfig --add gfsd

Testing

Make sure that Gfarm meta-server admin has registered you as a Gfarm user. This notes below are using gsi authentication to access gfarm. Then login to your user account

$ grid-proxy-init
$ gfls -la
drwxrwxr-x 4 gfarmadm gfarmadm          4 Jun 12 10:57 .
drwxrwxr-x 4 gfarmadm gfarmadm          4 Jun 12 10:57 ..
drwxr-xr-x 6 zhengc   gfarmadm          0 Jun 12 14:24 home
drwxr-xr-x 3 zhengc   gfarmadm          0 Jun 12 10:59 wwwo 
$ gfhost -M
x86_64-rocks5.4-linux 4 gfarm-fs.rocksclusters.org 600 0
$ gfhost -l
0.00/0.00/0.00 x x86_64-rocks5.4-linux 4 gfarm-fs.rocksclusters.org 600 0(198.202.88.96)
$ gfdf
    1K-blocks          Used         Avail Use% Host
   8589934592      12577792    8577356800   0% gfarm-fs.rocksclusters.org
----------------------------------------------
   8589934592      12577792    8577356800   0%

You can try more Gfarm commands.