Rocks-52 Cluster Users Guide

1. About this document
   The San Diego Supercomputer Center (SDSC) is contributing a cluster system to the PRAGMA Testbed. This document is a users' guide for Pacific Rim Applications and Grid Middleware Assembly resources and includes
   1. requirements for users
   2. procedures for getting an account
   3. hardware specifications and available Grid software
   4. resource utilization policy
   5. support staff available and mailing lists provided to users
   According to future administrative decisions, SDSC may modify this document and distribute the revised version through its web site.

2. Requirements for users
   SDSC requests all users to fulfill the following conditions.
   1. All users must belong to one of member organizations of either NPACI, SDSC or PRAGMA.
   2. All users participating in NPACI/SDSC must comply with the operating principles of NPACI/SDSC.
   3. All users participating in PRAGMA must comply with the operating principles of PRAGMA.
   4. All users must have concrete purposes for using SDSC resources.
   5. All users must contribute to SDSC/PRAGMA through the use of SDSC/PRAGMA resources.
   6. All users must have a user certificate issued by a PRAGMA trusted CA.
   PRAGMA trusts the following CAs: AIST, AusGrid, BMG, DOEGrids, ESnet, KISTI, NCSA, NPACI, ThaiGrid, USM.
   Procedures for obtaining a user certificate are available on the PRAGMA-GOC home page.

3. How to obtain an account
   In order to obtain an account for using SDSC resources, send the following information to Cindy Zheng.
   1. Full Name
   2. Affiliation
   3. Address
   4. Country
   5. Phone and facsimile number
   6. Email address
   7. Purpose for using SDSC resources
   8. Contribution to NPACI/SDSC and/or PRAGMA
   9. Desired account name (1st: 2nd : 3rd choices:)
   10. Shell (bash, csh, tcsh, etc.)
   11. ssh public key * **
   12. User certificate issued by a PRAGMA Trusted CA. ** ***
   The application will be reviewed by the administrators of SDSC resources within a few days of receipt of the application. The administrators will verify whether the applicant fulfills the requirements described in Section 2. If the application is approved, SDSC will create an account and send the necessary information to the applicant. Otherwise, the applicant will be notified that the request is rejected. Basically, a newly created account will be valid until the end of May. You are expected to move your files from SDSC resources to your resources by the expiration of your account. If you want to keep your account and files, you are required to renew your account every May. SDSC will remind all users to renew accounts and inform them of the procedures required when the expiration date gets close.

   * You can login to SDSC resources via an ssh command using RSA authentication. SDSC will NOT inform you of your initial UNIX password.
   ** An ssh public key and your user certificate should be sent as attachments to your application.
   *** A mapping of your subject name to your local account will be added to the grid-mapfile on the SDSC resources.

4. Resource information
   SDSC is providing a Linux cluster, called rocks-52.
   Rocks-52 is a 15-nodes Linux Cluster whose interconnection is Giga-bit Ethernet. Each node has 4 Xeon 3.06GHz and 4GB memory. Users' home directories are shared among all nodes by NFS.
   On rocks-52, the Globus gatekeeper is running on User Service Nodes and computing nodes can be utilized via the Globus jobmanager (jobmanager-sge).
   • Rocks-52
     Rocks-52 is available to users and projects approved by PRAGMA resources group.
     

       User Service Node:  rocks-52.sdsc.edu
       Computing Nodes:    compute-0-0
                           :
                           compute-0-14
                           15 nodes / 60 cpus
     

   • [Primitive Software]
     The operating system used on rocks-52 is Rocks 3.3.0 (based on RedHat 8.0), and general GNU tools are also installed.
     • GCC version 3.2.3
     • Intel(R) C++/Fortran Compiler version 8.0
       These are NON-COMMERCIAL version. You need a valid license to use these products. If you don't have a valid license, see below sites.
       
       • C++ Compiler: http://www.intel.com/software/products/compilers/clin/noncom.htm
       • Fortran Compiler: http://www.intel.com/software/products/compilers/flin/noncom.htm
       If you have a valid license, set INTEL_LICENSE_FILE environment variable to point license file or directory. Version 8.0 is installed in /opt/intel_cc_80 and /opt/intel_fc_80.

   • [Grid Software]
     • Globus Toolkit 2.4.3
     • GLOBUS_LOCATION = /opt/nmi (GT 2.4.3)

     • MDS is running on port 2135, on User Service Nodes
     • jobmanager-sge is available on User Service Nodes

     • Globus Packaging Toolkit 3.0.1
     • GPT_LOCATION = /opt/gpt

     • Jobmanager
       Sun Grid Engine 5.3
       • SGE_ROOT = /opt/gridengine
       • master node: User Service Node

     • Ninf-G version 2 (developmental version)
       NG_DIR = /home/tanisuke/local/ngv2

       A Globus-based GridRPC middleware suites, called Ninf-G, which was developed by GTRC/PRAGMA, is also available on the rocks-52 cluster.

     • MPICH version 1.2.5..12
       • ch_p4 device
         • /opt/mpich
         • /opt/mpich/gnu (same as above)
         • /opt/mpich/intel
         • /opt/mpich/myrinet
       • globus2 device
         • /opt/mpich2
         • /opt/mpich2/gnu (same as above)
         • /opt/mpich2/intel

       MPICH globus2 device ver. (a.k.a. MPICH-G2) uses GT 2.4.3 for now. Please do following procedure BEFORE you use,

       • B-shell)
         $ GLOBUS_LOCATION=/opt/nmi; export GLOBUS_LOCATION
         $ . ${GLOBUS_LOCATION}/etc/globus-user-env.sh

       • C-shell)
         % setenv GLOBUS_LOCATION /opt/nmi
         % source ${GLOBUS_LOCATION}/etc/globus-user-env.csh

5. Resource utilization policy
   1. Resource Reservation
      Rocks-52 is intended to be used for development and testing of Grid middleware and Grid applications in PRAGMA.

      You must apply exclusive use at least three business days before you intend to use rocks-52. Please send your application to crayon-admin. The application should indicate

      • Your name
      • Your account
      • Number of CPUs/nodes which you would like to occupy
      • the start time / end time (time zone must be indicated)
        (you are allowed to occupy, at most, 24 hours for benchmarking)
      • the purpose of the experiments

      Your application will be reviewed by the administrators and you will be notified of the decision within a few business days of your application.
      SDSC will announce the schedule of the reservation via the users mailing list (crayon-users@sdsc.edu).
      SDSC will provide the dedicated resources during the specified (or permitted) period. Other users will not be able to use (login) the dedicated resources during the period.

   2. Installation of additional software
      SDSC clusters can be used for R&D of Grid middleware and applications. You can install, test and evaluate your software on SDSC clusters. Since SDSC clusters are shared by many users, you are requested to test and debug your software thoroughly before installing it on SDSC clusters so that it will not damage the systems. If you need supervisor privilege for installing your software, please contact the support staff.

6. Support staff and mailing lists

   SDSC clusters are maintained by support staff. Your requests will be processed in a best-effort manner by the staff. SDSC provides the following mailing list for users:

   • crayon-users
     This mailing list is used mainly for announcements from administrators and support staff. All users are automatically included in the list.

   • crayon-admin
     System administrators of SDSC Rocks-52 Cluster are on this mailing list. This mailing list is used for correspondence from all users. Any requests, questions, and comments should be sent to this mailing list.