PRAGMA Cloud Rocks/Xen site setup

From PRAGMA wiki
Revision as of 18:03, 21 October 2015 by Wikiadmin (Talk | contribs)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Setup Rocks Xen hosting server

Is this Gfarm roll compatible with my system?

http://goc.pragma-grid.net/softdepot/gfarm-1.0-3.x86_64.disk1.iso (md5sum=ff5292550a39e7a77b41eb94f254817c) is a Rocks Gfarm roll. This roll is based on Gfarm 2.4.2 release and for Rocks 5.4 (Maverick) x86_64 systems.
In order to install and setup GSI-enabled Gfarm system, this Gfarm roll also does the following:

  • On a single node system or a cluster frontend
    • install VDT Globus and Globus-Base-SDK
    • setup certificate auto-update from PRAGMA certificates distribution point (including IGTF certificates distribution)
    • setup CRL auto-update
    • enable certificate files synchronization for compute nodes
  • On a cluster compute node
    • install VDT Globus and Globus-Base-SDK
    • setup certificate files synchronization with its frontend

The above info is to enable you to determine if there is any conflict with your existing system setups. So you can make a sound decision on whether of not this roll is suitable to your system.

Add gfarm roll to your frontend or single node system

If you did not install Gfarm roll at your initial install of your Rocks system

For example:

# rocks add roll gfarm-1.0-3.x86_64.disk1.iso
# rocks enable roll gfarm
# cd /export/rocks/install
# rocks create distro
# rocks run roll gfarm | bash
# reboot

Gfarm file system server setup on a single node system or a frontend

  • Ask Gfarm metaserver admin (Cindy) to setup an account for you on Gfarm metaserver
  • Obtain a host certificate and a gfsd certificate for your gfarm file system server. Make sure the gfsd CN is in the form of gfsd/your.gfarm.file.server.fqdn. For example: /O=grid/O=pragma/OU=SDSC/CN=gfsd/gfarm-fs.ucsd.edu
  • Send the following info to your Gfarm metaserver administrator to register your file system server
    • system architecture (for example: x86_64)
    • number of cores in your system (for example: 4)
    • hostname (for example: gfarm-fs.ucsd.edu)
    • DN string for gfsd (from gfsd certificate)
  • Install host certificate (note the file permission settings)
# ls -l /etc/grid-security/host*
-rw-r--r-- 1 root root 1241 Aug 15 13:51 /etc/grid-security/hostcert.pem
-r-------- 1 root root  891 Aug 15 13:51 /etc/grid-security/hostkey.pem
  • Install gfsd certificate files (note the file name, ownership and permissions)
# ls -ld /etc/grid-security/gfsd/
drwxr-xr-x 2 root root 4096 Aug 15 14:02 /etc/grid-security/gfsd/
# ls -l /etc/grid-security/gfsd/
total 8
-rw-r--r-- 1 _gfarmfs _gfarmfs 1249 Aug 16 15:41 gfsdcert.pem
-r-------- 1 _gfarmfs _gfarmfs  887 Aug 16 15:41 gfsdkey.pem
  • Copy 2 files from Gfarm metaserver to your Gfarm file server. Same paths and same file permissions
    • /opt/gfarm/etc/gfarm2.conf
    • /etc/grid-security/grid-mapfile
  • Adjust your system and your oganization's firewall settings according to the firewall requirements
  • Set gfsd start-up and start gfsd
# chkconfig --add gfsd
# chkconfig gfsd on
# /etc/init.d/gfsd start
  • Create Unix user accounts for yourself and zhengc, ssmallen.
    • You can use the pragma_setuser script and user tarballs at https://goc.pragma-grid.net/secure/updaccounts
    • It's easiest if you can keep user account names consistent with the user account names in the grid-mapfile you copied from the Gfarm metaserver.
    • If the account name for a user is different from the user's account name on the metaserver
      • Modify your grid-mapfile
      • setup /etc/gfarm-usermap in the format of
user1-account-name-on-metaserver local-user1-account-name
user2-account-name-on-metaserver local-user2-account-name
...
  • Check if iperf is installed (/opt/iperf)
    • If not, install iperf
rpm -ivh /export/rocks/install/rocks-dist/x86_64/RedHat/RPMS/iperf-2.0.5-1.x86_64.rpm
  • Make sure the iperf port (default 5001) is open to rocks-96.sdsc.edu
  • Setup iperf startup
    • If /etc/init.d/iperf does not exist, create one
    • Make sure the file permission is 755
    • Add iperf to chkconfig (# chkconfig --add iperf)

Gfarm testing

  • Run all tests in your user account on the newly setup system
  • Get your user certificate files and install it in ~/.globus
  • Try to copy a small file from local disk to your gfarm file system. For example
$ gfreg -v -h your.gfarm.server.fqdn /etc/motd /home/your-user-name/test1

If any problems, the output errors will give you some clue on the cause. If no complaint, verify the file test1 is created in gfarm file system

$ gfls -latr /home/your-user-name

Setup VM auto-deployment on your VM hosting server frontend

Install/setup Gfarm client

If you already have gfarm file system server setup on the frontend, your gfarm client is already setup. Go to next section.

# rocks add roll gfarm-1.0-3.x86_64.disk1.iso
# rocks enable roll gfarm
# cd /export/rocks/install
# rocks create distro
# rocks run roll gfarm | bash
  • it's best to reboot the system at this point. But if you are installing gfarm client ONLY on a frontend and would like to avoid rebooting the frontend, do the following manually:
# source /opt/vdt/setup.sh
# echo "export PATH=/opt/gfarm/bin:/opt/gfarm2fs/bin:$PATH" > /etc/profile.d/gfarm.sh
# cp /opt/vdt/setup.sh /etc/profile.d/globus.sh
# export PATH="/opt/gfarm/bin:$PATH"
# mkdir -p /etc/grid-security
# vdt-control --on
# vdt-update-certs --force
# /bin/tar -chf /etc/grid-security/certs.tar /etc/grid-security/certificates
# echo "FILES += /etc/grid-security/certs.tar /opt/gfarm/etc/gfarm2.conf" >> /var/411/Files.mk
# rocks sync users
# /opt/rocks/bin/rocks add firewall global=global action=ACCEPT chain=INPUT protocol=udp service=600 network=all rulename=A100-GFARM-UDP-600
# /opt/rocks/bin/rocks add firewall global=global action=ACCEPT chain=INPUT protocol=tcp service=600 network=all rulename=A100-GFARM-TCP-600
# /opt/rocks/bin/rocks add firewall global=global action=ACCEPT chain=INPUT protocol=tcp service=601 network=all rulename=A100-GFARM-TCP-601
# chmod o+rx /bin/fusermount
# rm /etc/rc.d/rocksconfig.d/post-*-gfarm-*

VM deployment scripts installation and setup

  • If you installed gfarm roll, check if vm-deploy scripts was included and installed at /opt/vm-scripts/ on the frontend.
  • If vm-deploy scripts are NOT installed
    • Download the most current vm-scripts tarball from gfarm:/vm-images/SDSC/ or here
    • Choose an installation directory (for example, /opt/vm-scripts) and un-tar the files there
    • Edit vm-deploy to set "scriptdir" to the installation directory path
  • Edit AvailableIP, AvailableNodes, LocalSettings and resolv.conf files in the installation directory
  • Copy the vm-script installation directory to all nodes specified in the AvailableNodes file
  • On all the nodes specified in the AvailableNodes file, create a group (for example, vmdisks) and give it rwx access to VM disk image directory path (default is /state/partition1/xen/disks) on frontend and all vm-containers. For example,
$ ls -ld /state/partition1/xen
drwxr-x--- 4 root vmdisks 4096 Jan  7 2011 /state/partition1/xen
$ ls -ld /state/partition1/xen/disks
drwxrwx--- 2 root vmdisks 4096 Oct  5 00:05 /state/partition1/xen/disks
  • Add users to the group
  • Add a line in /etc/sudoers (visudo) to enable the group sudo sub-scripts. For example,
 
%vmdisks ALL=NOPASSWD:/opt/vm-scripts/vm-new, /opt/vm-scripts/vm-makeover, /opt/vm-scripts/vm-start, /opt/vm-scripts/vm-cleanup, /opt/vm-scripts/vm-free, /opt/rocks/bin/rocks
  • add "/etc/sudoers" to the file list in /var/411/Files.mk, then run "rocks sync users"

Iperf installation and setup

  • Check if iperf is installed (/opt/iperf)
    • If not, install iperf
rpm -ivh /export/rocks/install/rocks-dist/x86_64/RedHat/RPMS/iperf-2.0.5-1.x86_64.rpm
  • Make sure the iperf port (default 5001) is open to rocks-96.sdsc.edu
  • Setup iperf startup
    • If /etc/init.d/iperf does not exist, create one
    • Make sure the file permission is 755
    • Add iperf to chkconfig (# chkconfig --add iperf)

Testing

  • There is a feature in rocks 5.4-5.4.3 that output of some Rocks commands are different when you have only one VM vs. more than one VMs. This cause problem for the VM deployment scripts. This will be changed in future versions of Rocks. To get around this problem now, if there is no VM created on your hosting system yet, please create a dummy VM first. For example
# rocks add host vm <your-frontend-fqdn> membership="hosted vm" name=dummy-vm

Testing vm-deploy scripts in your user account. For example

  • To deploy vm image nyouga on the frontend
$ grid-proxy-init
$ gfexport vm-images/vmdb.txt
$ vm-deploy nyouga

A VM named nyouga-<your-user-name> should be created.

  • To remove the vm image
$ vm-remove nyouga-<your-user-name>

Updates

  • Fix the cron bug in Gfarm roll
    • Check your root crontab on the FRONTEND (# crontab -l)
    • Remove the entry "00 02 * * * cd /; /bin/tar xf /etc/grid-security/certs.tar"
    • Add the entry "00 01 * * * /bin/tar -chf /etc/grid-security/certs.tar /etc/grid-security/certificates"
  • Add vm-remove scripts
    • Check /opt/vm-scripts directory
    • If you don't have vm-remove and vm-free files, please download the most recent vm-scripts tar ball from http://goc.pragma-grid.net/softdepot
    • Extract vm-remove and vm-free files to /opt/vm-scripts directory, file permissions should be 755.
    • Add vm-free in /etc/sudoers as
%vmdisks ALL=NOPASSWD:/opt/vm-scripts/vm-new, /opt/vm-scripts/vm-makeover, /opt/vm-scripts/vm-start, /opt/vm-scripts/vm-cleanup, /opt/vm-scripts/vm-free, /opt/rocks/bin/rocks

User testing

Whence all above tested and working, inform each user the following info:

  • UNIX account username
  • Your hosting server frontend FQDN
  • IP address(es) or IP# range for creating new VM(s)

Then help users resolve any issues.

Gfarm roll installation on vm-containers and compute nodes

  • After gfarm roll is installed on the frontend and the gfarm setup on the frontend in above section is completed, reinstall vm-containers and compute nodes.
    • To avoid reinstall (This method has not been fully tested)
      • Copy the following files from the frontend
        • /opt/gfarm
        • /opt/gfarm2fs
        • /opt/vm-scripts
        • /opt/vdt
        • /etc/profile.d/gfarm.sh
        • /etc/profile.d/globus.sh
        • /etc/sudoers
      • Add the entry "00 02 * * * cd /; /bin/tar xf /etc/grid-security/certs.tar" in root crontab.
  • After the compute nodes are reinstalled and boot up, on the frontend, run
# rocks sync users
# rocks run host "cd /; tar xf /etc/grid-security/certs.tar"
  • Test gfarm access from each vm-containers and compute nodes

Other issues and tips

  • To relocate your gfarm file system directory on your gfarm file system server
# service gfsd stop

Edit OPTIONS line in /etc/init.d/gfsd

# cp -a /old/gfarm/dir-path /new/gfarm/dir-path
# service gfsd start