SDSC Gfarm meta server

From PRAGMA wiki
Jump to: navigation, search

gfarm-meta1

  • Hardware:
    • Dell 1425, x86_64
    • 4 x 3.2GHz CPUs
    • 16GB memory
    • 2TB local disks (configured RAD1)
    • 8TB NFS-mounted backend storage
  • Network:
    • 1 public network interface (1 Gbps) for Gfarm
    • 2 private network interfaces (10 Gbps)
      • 10.3.x.x connect to backend storage
      • 10.5.x.x connect to all local gfarm clients
  • System: Rocks 5.4
  • Gfarm software: 2.4.2

Planning and decisions

  • Setup this server as a gfarm meta-server and a gfarm file system node
  • If Globus GSI library is installed use RPM, then install Gfarm use RPM. If Globus GSI library is NOT installed use RPM, then install Gfarm from source.
  • Install Globus use VDT. Install Gfarm from source.

Pre-installation setups

  • Install postgresql
[root@gfarm-meta1 ~]# rpm -ivh /export/rocks/install/rocks-dist/x86_64/RedHat/RPMS/postgresql84-*8.4.7-1.el5_6.1.x86_64.rpm
Preparing...                ########################################### [100%]
   1:postgresql84-libs      ########################################### [  8%]
   2:postgresql84           ########################################### [ 17%]
   3:postgresql84-server    ########################################### [ 25%]
   4:postgresql84-contrib   ########################################### [ 33%]
   5:postgresql84-devel     ########################################### [ 42%]
   6:postgresql84-docs      ########################################### [ 50%]
   7:postgresql84-plperl    ########################################### [ 58%]
   8:postgresql84-plpython  ########################################### [ 67%]
   9:postgresql84-pltcl     ########################################### [ 75%]
  10:postgresql84-python    ########################################### [ 83%]
  11:postgresql84-tcl       ########################################### [ 92%]
  12:postgresql84-test      ########################################### [100%]
  • Get host certificate for gfarm-meta1.rocksclusters.org and install the host certificate files in /etc/grid-security directory
# ls -l /etc/grid-security/
total 8
-rw------- 1 root root 1241 Aug  8 11:52 hostcert.pem
-r-------- 1 root root  891 Aug  8 11:52 hostkey.pem
  • Install pacman
# wget http://physics.bu.edu/pacman/sample_cache/tarballs/pacman-latest.tar.gz
# cd /opt
# tar xvzf /root/pacman-latest.tar.gz
# ln -s pacman-3.29/ pacman
  • Install GLOBUS
# cd pacman
# source ./setup.sh
# cd ..
# mkdir vdt
# cd vdt
# pacman -get http://vdt.cs.wisc.edu/vdt_200_cache:Globus
# pacman -get http://vdt.cs.wisc.edu/vdt_200_cache:Globus-Base-SDK
# . setup.sh
# vdt-ca-manage setupca --location root --url vdt

Edit /opt/vdt/vdt/etc/vdt-update-certs.conf and add the following line

cacerts_url=http://ca.pragma-grid.net/dist/igtf-pragma-ca-certs-version

Test the certificate update

# vdt-update-certs --force
  • Make sure the /opt/vdt/globus/TRUSTED-CA is a symbolic link which points to the desired certificates directory. If not, create it.
  • Enable some VDT services
# vdt-control --enable fetch-crl
# vdt-control --enable vdt-rotate-logs
# vdt-control --enable vdt-update-certs
# vdt-control --list
Service                 | Type   | Desired State
------------------------+--------+--------------
fetch-crl              | cron    | enable
vdt-rotate-logs        | cron    | enable
vdt-update-certs       | cron    | enable
globus-gatekeeper      | inetd   | do not enable
gsiftp                 | inetd   | do not enable
# vdt-control --on

Install Gfarm Software

  • Download and install source rpm
# wget http://sourceforge.net/projects/gfarm/files/gfarm_v2/2.4.2/gfarm-2.4.2.tar.gz
# tar xzvf gfarm-2.4.2.tar.gz
# cd gfarm-2.4.2
# ./configure --prefix=/opt/gfarm2.4.2 --with-openldap=/usr --with-postgresql=/usr --with-openssl=/usr --with-globus=/opt/vdt/globus --with-globus-flavor=gcc64 --enable-xmlattr
...
...
checking openssl... configure: error: OpenSSL cannot be linked with "-I/opt/vdt/globus/include/gcc64 
-R/opt/vdt/globus/lib -L/opt/vdt/globus/lib -lssl_gcc64 -lcrypto_gcc64" options, aborted

If you receive this error shown above, find or install 64-bit libssl.so and libcrypto.so and link them into the globus lib directory.

# cd /opt/vdt/globus/lib
# ln -s /usr/lib64/libssl.so libssl_gcc64.so
# ln -s /usr/lib64/libcrypto.so libcrypto_gcc64.so
# cd ~/gfarm-2.4.2
# ./configure --prefix=/opt/gfarm2.4.2 --with-openldap=/usr --with-postgresql=/usr --with-openssl=/usr --with-globus=/opt/vdt/globus --with-globus-flavor=gcc64 --enable-xmlattr
using lib/gfs_hook/sysdep/linux directory for system dependent objects.
using gftool/config-gfarm/sysdep/linux directory for system dependent part of co
nfig-gfarm.
checking use of private source directory... not use.
configure: creating ./config.status
config.status: creating Makefile
config.status: creating makes/config.mk
config.status: WARNING:  'makes/config.mk.in' seems to ignore the --datarootdir
setting
config.status: creating makes/install-doc
config.status: creating gftool/config-gfarm/config-gfarm
config.status: WARNING:  'gftool/config-gfarm/config-gfarm.in' seems to ignore t
he --datarootdir setting
config.status: creating gftool/config-gfarm/config-gfsd
config.status: WARNING:  'gftool/config-gfarm/config-gfsd.in' seems to ignore th
e --datarootdir setting
config.status: creating gftool/config-gfarm/config-gfarm-update
config.status: WARNING:  'gftool/config-gfarm/config-gfarm-update.in' seems to i
gnore the --datarootdir setting
config.status: creating gftool/gfdump/gfdump.postgresql
config.status: WARNING:  'gftool/gfdump/gfdump.postgresql.in' seems to ignore th
e --datarootdir setting
config.status: creating package/solaris/gfmd
config.status: creating package/solaris/gfsd
config.status: creating include/gfarm/gfarm_config.h
# make
# make install
  • Add user environment setups
# cp /opt/vdt/setup.sh /etc/profile.d/globus.sh
# vi /etc/profile.d/gfarm.sh
# cat /etc/profile.d/gfarm.sh
export PATH="/opt/gfarm2.4.2/bin:$PATH"
  • logout, then login, and make sure that Globus environment variables and Gfarm path are setup correctly in user environment.

Setup

Configure Gfarm meta-server

  • Modify config-gfarm

There is a bug in the Updates-CentOS-5.6: 2011-07-10 x86_64 that "SU -" does not work as expected. So we need to change it to "SU" in both /opt/gfarm2.4.2/bin/config-gfarm and /opt/gfarm2.4.2/share/gfarm/config/linux/redhat/gfarm-pgsql.in files.

# cd /opt/gfarm/bin 
# mv config-gfarm config-gfarm.orig
# sed 's/SU -/SU/' config-gfarm.orig > config-gfarm
# diff config-gfarm config-gfarm.orig
590c590
<               run_with_backend_privilege="$SU $BACKEND_PRIVILEGE -c /bin/sh"
---
>               run_with_backend_privilege="$SU - $BACKEND_PRIVILEGE -c /bin/sh"
# cd /opt/gfarm/share/gfarm/config/linux/redhat/ 
# mv gfarm-pgsql.in gfarm-pgsql.in.orig
# sed 's/SU -/SU/' gfarm-pgsql.in.orig > gfarm-pgsql.in
# diff gfarm-pgsql.in gfarm-pgsql.in.orig
24c24
< *)    RUN="$SU $PRIVILEGE";;
---
> *)    RUN="$SU - $PRIVILEGE";;
  • Configure gfmd
# config-gfarm -t -a gsi
# config-gfarm -a gsi -X
created /opt/gfarm2.4.2/etc
created /var/gfarm-pgsql
created /opt/gfarm2.4.2/etc/gfarm2.conf
created /opt/gfarm2.4.2/etc/gfmd.conf
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale C.
The default text search configuration will be set to "english".

fixing permissions on existing directory /var/gfarm-pgsql ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 32MB
creating configuration files ... ok
creating template1 database in /var/gfarm-pgsql/base/1 ... ok
initializing pg_authid ... ok
initializing dependencies ... ok
creating system views ... ok
loading system objects' descriptions ... ok
creating conversions ... ok
creating dictionaries ... ok
setting privileges on built-in objects ... ok
creating information schema ... ok
vacuuming database template1 ... ok
copying template1 to template0 ... ok
copying template1 to postgres ... ok

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the -A option the
next time you run initdb.

Success. You can now start the database server using:

    /usr/bin/postgres -D /var/gfarm-pgsql
or
    /usr/bin/pg_ctl -D /var/gfarm-pgsql -l logfile start

created /etc/init.d/gfarm-pgsql
created /etc/init.d/gfmd
wait 5 seconds for postmaster to start up
Stopping gfarm-pgsql: server stopped
                                                           [  OK  ]
added gfarm-pgsql service
Starting gfarm-pgsql:                                      [  OK  ]
added gfmd service
Starting gfmd:                                             [  OK  ]
config-gfarm success

In this case, I had logged in to my user account, then su'ed to root to run config-gfarm, so my user account will be picked up by config-gfarm as the admin_user for gfarm meta server. Alternatively, you can explicitly specify admin_user with -A (and -D) option. See Gfarm 2.4.2 Setup.

  • Enable gsi_auth and sharedsecret authentication

Edit /opt/gfarm2.4.2/etc/gfarm2.conf and /opt/gfarm2.4.2/etc/gfmd.conf, replace the line

auth enable gsi *

with these 2 lines

auth enable sharedsecret *
auth enable gsi_auth *
  • Open TCP and UDP port 600 and 601

Edit /etc/sysconfig/iptables and add these 3 lines

-A INPUT -p tcp --dport 600 -j ACCEPT
-A INPUT -p udp --dport 600 -j ACCEPT
-A INPUT -p tcp --dport 601 -j ACCEPT

Then reload iptables

# service iptables restart
  • Finish setup Gfarm meta-server
# useradd -c "Gfarm gfsd" _gfarmfs
# su _gfarmfs
$ gfkey -f -p 63072000
$ su -
# /etc/init.d/gfmd restart
# chkconfig --add gfmd
# chkconfig --add gfarm-pgsql

Configure Gfarm file system node

  • Obtain a service certificate for gfsd (CN=gfsd/gfarm-meta1.rocksclusters.org)
  • Install gfsd certificate files as
# ls -l /etc/grid-security/gfsd
total 8
-rw-r--r-- 1 _gfarmfs _gfarmfs 1371 Jun  6 16:18 gfsdcert.pem
-r-------- 1 _gfarmfs _gfarmfs  887 Jun  6 16:18 gfsdkey.pem
  • Add gfsd DN in grid-mapfile as
"/DC=NET/DC=PRAGMA-GRID/OU=SDSC/CN=gfsd/gfarm-meta1.rocksclusters.org" @host@ gfarm-meta1.rocksclusters.org
  • Setup Gfarm spool directory
[root@gfarm-meta1 gfarm2.4.2]# config-gfsd /gfarmdata
created /etc/init.d/gfsd
config-gfsd success

Please ask admin_user to register your host by the following command:

/opt/gfarm2.4.2/bin/gfhost -c -a x86_64-rocks5.4-linux -p 600 -n 4 gfarm-meta1.rocksclusters.org

After that, start gfsd by the following command as a root:

/etc/init.d/gfsd start
  • Register gfsd with meta-server

Do this in the admin_user account, not root account

$ /opt/gfarm2.4.2/bin/gfhost -c -a x86_64-rocks5.4-linux -p 600 -n 4 gfarm-meta1.rocksclusters.org
  • Start gfsd as root
# /etc/init.d/gfsd start
# chkconfig --add gfsd

Testing

Make sure that Gfarm meta-server admin has registered you as a Gfarm user. This notes below are using gsi authentication to access gfarm. Then login to your user account

$ grid-proxy-init
$ gfls -la
drwxrwxr-x 4 gfarmadm gfarmadm          4 Jun 12 10:57 .
drwxrwxr-x 4 gfarmadm gfarmadm          4 Jun 12 10:57 ..
drwxr-xr-x 6 zhengc   gfarmadm          0 Jun 12 14:24 home
drwxr-xr-x 3 zhengc   gfarmadm          0 Jun 12 10:59 wwwo 
$ gfhost -M
x86_64-rocks5.4-linux 4 gfarm-meta1.rocksclusters.org 600 0
$ gfhost -l
0.00/0.00/0.00 x x86_64-rocks5.4-linux 4 gfarm-meta1.rocksclusters.org 600 0(198.202.88.96)
$ gfdf
    1K-blocks          Used         Avail Use% Host
   8589934592      12577792    8577356800   0% gfarm-meta1.rocksclusters.org
----------------------------------------------
   8589934592      12577792    8577356800   0%

You can try more Gfarm commands.

Other notes for possible issues

Hostname issue

If the primary hostname in DNS is different from the hostname you build your system with

  • Get host cert and gfsd cert using the primary hostname in DNS
  • Register the gfsd using the primary hostname in DNS