Gfarm file server

From PRAGMA wiki
Jump to: navigation, search

Install Prerequisites

  1. install Globus yum repo configuration file
    # rpm -Uvh
  2. install Globus utilities
    # yum install globus-gssapi-gsi-devel
    # yum install globus-gsi
    # yum install globus-data-management-sdk
  3. check for postgresql-devel; if not present install
    # yum --enablerepo base list postgresql-devel
    # yum --enablerepo base install postgresql-devel
    NOTE: Yum repo may have newer RPM versions of the dependent RPMS thatn those already installed on your host.
    In this case theses RPMS will be updated adn you will see a message similar to:
    Updating for dependencies:
    postgresql x86_64 8.4.13-1.el6_3
    postgresql-libs x86_64 8.4.13-1.el6_3
    postgresql-server x86_64 8.4.13-1.el6_3
  4. check for openssl-devel; if not present install
    # yum list openssl-devel
    # yum install openssl-devel
  5. check for openldap-devel; if not present install
    # yum list openldap-devel
    # yum install openldap-devel
  6. check for pkgconfig; if not present install
    # yum list pkgconfig
    # yum install pkgconfig
  7. check for fuse-devel; if not present install
    # yum --enablerepo base list fuse-devel
    # yum --enablerepo base install fuse-devel
  8. install fetch-crl (needed for IGTF certificates updates)
    # yum --enablerepo epel install fetch-crl
  9. If you have a previous installation of gfarm remove it. Specifically, should be no /opt/gfarm and no /etc/init.d/gfsd prior to this installation.

Install IGTF and PRAGMA grid Certificates

  1. download and verify IGTF certificates (use the latest version available, in this example it is 1.54)
    # wget
    # wget
    # gpg --verify igtf-policy-installation-bundle-1.54.tar.gz.asc
    NOTE: If you see a message similar to one below
    gpg: Signature made Mon 17 Jun 2013 05:02:03 AM PDT using DSA key ID 3CDBBC71
    gpg: Can't check signature: No public key
    You will need to install specified gpg ID key and verify the downloaded file signature as in:
    # gpg --recv-keys 3CDBBC71
    # gpg --verify igtf-policy-installation-bundle-1.54.tar.gz.asc
  2. Install IGTF certificates
    # tar xzvf igtf-policy-installation-bundle-1.54.tar.gz
    # cd igtf-policy-installation-bundle-1.54
    # ./configure --with-profile=classic --with-profile=slcs --with-profile=mics
    # make install
    # ls /etc/grid-security/certificates/
    NOTE: You should see output that lists contents of /etc/grid-security/certificates/ directory.
  3. Install PRAGMA grid certificates
    # wget --no-check-certificate
    # tar xzvf pragma-certs.tar.gz -C /etc/grid-security/certificates/
    NOTE: Pragma certificates will be added to /etc/grid-security/certificates/
  4. Request the latest version of grid-mapfile from Gfarm meta-server admin.
  5. Setup CRL updates cron job.
    # crontab -e
    and add a line similar to the line below that will run CRL updates every 6 hrs
    24 2,8,14,20 * * * /opt/perl/bin/perl /usr/sbin/fetch-crl
  6. Test your CRL update
    # /opt/perl/bin/perl /usr/sbin/fetch-crl

Obtain your Gfarm Service Certificate

Gfarm services use gsi authentication thus it is necessary to obtain a service certificate for gfsd. The service certificate is a host certificate. To obtain your service certificate please follow these steps:

  1. Please see PRAGMA Certificate Authority and apply for a host certificate. Once your application is received and processed you will get email notification from the CA that confirms your request.
  2. After you receive a CA confirmation, follow instructions in CA Guide to create a service certificate request and obtain a certificate. Please make sure when creating a certificate request (when executing command grid-hostreq) that CN has a string gfsd/ in it. For example, for a host the CN will be: gfsd/

Install Gfarm Software

  1. Download and install gfarm disribution
    # wget
    # tar xzvf gfarm-
    # cd gfarm-
    # ./configure --prefix=/opt/gfarm- --with-openldap=/usr --with-postgresql=/usr \
    --with-openssl=/usr --with-globus=/usr --with-globus-flavor=gcc64 --enable-xmlattr
    # make
    # make install
    # ln -s /opt/gfarm- /opt/gfarm
  2. Create a new environment profile file
    # vi /etc/profile.d/
    and add the following line to it:
    export PATH="/opt/gfarm/bin:$PATH"
  3. Source the file to make changes available now
    # . /etc/profile.d/

Configure Gfarm file system node

  1. Create an account for gfsd
    # useradd -c "Gfarm gfsd" _gfarmfs
  2. Open Firewall for gfarm ports (TCP and UDP port 600 and TCP port 601)
    # rocks list host firewall
    # rocks add firewall global=global action=ACCEPT chain=INPUT protocol=udp service=600 network=all rulename=A100-GFARM-UDP-600
    # rocks add firewall global=global action=ACCEPT chain=INPUT protocol=tcp service=600 network=all rulename=A100-GFARM-TCP-600
    # rocks add firewall global=global action=ACCEPT chain=INPUT protocol=tcp service=601 network=all rulename=A100-GFARM-TCP-601
    # rocks sync host firewall
    # service iptables restart
  3. Install gfsd certificate files in /etc/grid-security/gfsd. Permissions and file names must be:
    # ls -l /etc/grid-security/gfsd
    -rw-r--r-- 1 _gfarmfs _gfarmfs 1371 Jun 6 16:18 gfsdcert.pem
    -r-------- 1 _gfarmfs _gfarmfs 887 Jun 6 16:18 gfsdkey.pem
  4. Modify /etc/grid-securiy/grid-mapfile and add a line in the format: "DN" @host@ FQDN where
    "DN" - distinguished name used in your gfsd service certificate. NOTE: quotes are needed.
    @host@ - fixed string
    FQDN - your host fully-qualified domain name.
    To find a subject DN use the following command:
    # grid-cert-info -subject -f /etc/grid-security/gfsd/gfsdcert.pem
    For example, for a host a line in grid-mapfile will be :
    "/DC=NET/DC=PRAGMA-GRID/OU=SDSC/CN=gfsd/" @host@
    IMPORTANT: email the line to Gfarm meta-server admin.
  5. Get current gfarm2.conf and install it in /opt/gfarm/etc directory
    ls -l /opt/gfarm/etc/gfarm2.conf
    -rw-r--r-- 1 root root 188 Jan 23 13:53 /opt/gfarm/etc/gfarm2.conf
  6. Setup Gfarm spool directory
    A spool directory stores physical files in Gfarm file system. Note that the spool directory should be a non-shared area among filesystem nodes. It can be a local filesystem or a a filesystem mounted from another host. To setup a spool directory, run config-gfsd command followed up by the spool directory. For example, to create a spool in /state/partition1/gfarmdata, first run the command with "-t flag to verify setting (no action will be taken during this command):
    # config-gfsd -t /state/partition1/gfarmdata
    Then run the command to create the spool directory
    # config-gfsd /state/partition1/gfarmdata
    created /etc/init.d/gfsd
    config-gfsd success
    Please ask admin_user to register your host by the following command:
    /opt/gfarm- -c -a x86_64-centos6.2-linux -p 600 -n 2
    After that, start gfsd by the following command as a root:
    /etc/init.d/gfsd start
    IMPORTANT: Send the above output to Gfarm meta-server administrator to register your file system node
  7. After your Gfarm meta-server administrator registered your file system node, start gfsd
    # /etc/init.d/gfsd start
    To automatically start gfsd at boot run
    # chkconfig --add gfsd
    When gfsd starts verify that there are be 2 gfsd processes running. Your output should be similar to :
    ps -ef | grep gfsd
    _gfarmfs 21526 1 0 14:22 ? 00:00:00 /opt/gfarm/sbin/gfsd -P /var/run/ -v -f /opt/gfarm/etc/gfarm2.conf -h FQDN -r /DATA
    _gfarmfs 21528 21526 0 14:22 ? 00:00:00 /opt/gfarm/sbin/gfsd -P /var/run/ -v -f /opt/gfarm/etc/gfarm2.conf -h FQDN-r /DATA
    where FQDN is fqdn of your host and DATA is your gfarm spool directory.
    If there is only one gfsd process running it means there is an error. Please see troubleshooting section below.

Troubleshooting gfsd

Most problems are related to GSI authentication.

  1. Make sure you have the latest version of grid-mapfile
  2. Verify that your host, server, ca and user certificates are installed and have correct permissions. Check the IGTF certificates and service certificate sections of this guide.
  3. Check if there are authentication problems
    • Enable verbose authentication logging in /your-gfarm-install-path/etc/gfarm2.conf file, uncomment a line
    log_auth_verbose enable
    • Restart gfsd:
    /etc/init.d/gfsd stop
    /etc/init.d/gfsd start
    • Check /var/log/messages file for gfsd errors
    If you see errors similar ot the following:
    Aug 2 16:54:42 rocks-89 gfsd[3438]: <notice> [1000700] Can't initiate session because of:
    Aug 2 16:54:42 rocks-89 gfsd[3438]: <info> [1000607] #011 : GSS Major Status: Authentication Failed
    Aug 2 16:54:42 rocks-89 gfsd[3438]: <info> [1000607] #011 : GSS Minor Status Error Chain:#012globus_gsi_gssapi: SSLv3 handshake problems#012globus_gsi_gssapi: Unable to verify remote side's credentials#012globus_gsi_gssapi: SSLv3 handshake problems: Couldn't do ssl handshake#012OpenSSL Error: s3_pkt.c:1193: in library: SSL routines, function SSL3_READ_BYTES: tlsv1 alert unknown ca SSL alert number 48
    Aug 2 16:54:42 rocks-89 gfsd[3438]: <warning> [1000550] connecting to gfmd at failed, sleep 1 sec: operation not permitted
    it means there is a problem with GSI authentication.
  4. Check PRAGMA GRID CA certificate hash
    The certificate of a ca that was used to sign your host and user certificate must have a correct
    hash in /etc/grid-security/certificates/. Usually this is done automatically by installing all the certificates
    because the correct hash of PRAGMA GRID CA 5456d9ca is already in place.
    Newer versions of openssl create a hash different from the original used in the certificate.
    Check the hash on your host with the following command:
    openssl x509 -subject_hash -in /etc/grid-security/gfsd/cacert.pem
    If the output differs from 5456d9ca create a specified hash.
    For example:
    openssl x509 -subject_hash -in /etc/grid-security/gfsd/cacert.pem
    1a912308 <-- this is a new hash that is different form the existing certificate hash
    cd /etc/grid-security/certificates
    ln -s 5456d9ca.0 1a912308.0
    ln -s 5456d9ca.signing_policy 1a912308.signing_policy
    Restart gfsd and check /var/log/messages for errors.


Make sure that Gfarm meta-server admin has registered you as a Gfarm user. The steps below are using gsi authentication to access gfarm. The steps are executed on your host from your user account

  1. Create a certificate proxy
    $ grid-proxy-init
  2. List contents of gfarm directory
    $ gfls -la
    drwxrwxr-x 4 gfarmadm gfarmadm 4 Jun 12 10:57 .
    drwxrwxr-x 4 gfarmadm gfarmadm 4 Jun 12 10:57 ..
    drwxr-xr-x 6 zhengc gfarmadm 0 Jun 12 14:24 home
    drwxr-xr-x 3 zhengc gfarmadm 0 Jun 12 10:59 vm-images
  3. Show hosts metadata
    $ gfhost -M
    x86_64-rocks5.4-linux 4 600 0
  4. Show hosts metadata in long format
    $ gfhost -l
    0.00/0.00/0.00 x x86_64-rocks5.4-linux 4 600 0(
  5. Display gfarm filesystem free disk space
    $ gfdf
    1K-blocks Used Avail Use% Host
    8589934592 12577792 8577356800 0%
    8589934592 12577792 8577356800 0%

You can try more Gfarm commands.