Rocks VC experiment 1

From PRAGMA wiki
Jump to: navigation, search

Create Rocks VC images and xml

Should be done by the VC owner.

  • Created a generic Rocks VC
  • Create a xml file for the VC
    • The input xml file for the VC contains information necessary for resource providers to deploy the VC. It's provided by VC owner and used by resources providers as the input file for deployment. To create this file
    • make sure the VC is running
    • run "virsh dumpxml" on the frontend and one compute node and save the output into 2 seperate xml files
    • copy the example xml file template - vc-in.xml - to file (in this example) calit2-119-222.xml
    • edit calit2-119-222.xml with the info you got from "virsh dumpxml"
  • Deposit VC images and xml

Deposit calit2-119-222.img.gz and calit2-119-222.xml to a distribution poit or Gfarm.

  • Inform the resources providers the VC images/xml files location and specify the number of compute nodes for deployment.

Deploy the VC

Should be done by the resources provider.
This specific example of VC - calit2-119-222 - is created on a Rocks 6.0 hosting server and is then deployed to a Rocks 6.1 hosting server. Assume the request is to deploy the VC with 3 compute nodes.

Deploy the VC frontend

  • Create a new VC
# rocks add cluster 137.110.119.221 3 fe-name=calit2-119-221
  • Set private ip to be consistent with the VC frontend image being imported (in this case is 10.1.1.1)
# rocks list host interface calit2-119-221
SUBNET  IFACE MAC               IP              NETMASK       MODULE NAME           VLAN OPTIONS CHANNEL
private eth0  7a:77:6e:40:00:02 10.1.255.250    255.255.0.0   ------ calit2-119-221 2    ------- -------
public  eth1  7a:77:6e:40:00:03 137.110.119.221 255.255.255.0 ------ calit2-119-221 0    ------- -------
# rocks set host interface ip calit2-119-221 eth0 10.1.1.1
# rocks list host interface calit2-119-221            SUBNET  IFACE MAC               IP              NETMASK       MODULE NAME           VLAN OPTIONS CHANNEL
private eth0  7a:77:6e:40:00:02 10.1.1.1        255.255.0.0   ------ calit2-119-221 2    ------- -------
public  eth1  7a:77:6e:40:00:03 137.110.119.221 255.255.255.0 ------ calit2-119-221 0    ------- -------
  • Set private ip of compute nodes
# rocks set host interface ip hosted-vm-0-0-1 eth0 10.1.255.254
# rocks set host interface ip hosted-vm-0-1-1 eth0 10.1.255.253
# rocks set host interface ip hosted-vm-0-2-1 eth0 10.1.255.252
# rocks list host interface
HOST              SUBNET  IFACE MAC               IP              NETMASK       MODULE NAME             VLAN OPTIONS CHANNEL
...
hosted-vm-0-0-1:  ------- eth0  7a:77:6e:40:00:09 10.1.255.254    ------------- ------ hosted-vm-0-0-1  3    ------- -------
hosted-vm-0-1-1:  ------- eth0  7a:77:6e:40:00:0a 10.1.255.253    ------------- ------ hosted-vm-0-1-1  3    ------- -------
hosted-vm-0-2-1:  ------- eth0  7a:77:6e:40:00:0b 10.1.255.252    ------------- ------ hosted-vm-0-2-1  3    ------- -------
  • Find the disk path of calit2-119-221
# rocks list host vm calit2-119-221 showdisks=yes
SLICE MEM   CPUS MAC               HOST           VIRT-TYPE DISK                                                        DISKSIZE
0     1024  1    7a:77:6e:40:00:02 calit2-119-121 hvm       file:/state/partition2/kvm/disks/calit2-119-221.vda,vda,virtio 36
----- ----- ---- 7a:77:6e:40:00:03 -------------- --------- ----------------------------------------------------------- --------
  • Unzip the frontend image file to the path. Note that in calit2-119-222.xml file, the original disk image file was type of hda.
# gunzip -c calit2-119-222.img.gz > state/partition2/kvm/disks/calit2-119-221.hda
# rocks set host vm calit2-119-221 disk=file:/state/partition2/kvm/disks/calit2-119-221.hda,hda,ide
  • Modify the frontend image
# kpartx -av /state/partition2/kvm/disks/calit2-119-221.hda
add map loop0p1 (253:0): 0 12288000 linear /dev/loop0 2048
add map loop0p2 (253:1): 0 4096000 linear /dev/loop0 12290048
add map loop0p3 (253:2): 0 2048000 linear /dev/loop0 16386048
add map loop0p4 (253:3): 0 2 linear /dev/loop0 18434048
add map loop0p5 (253:4): 0 20625408 linear /dev/loop0 18436096
# mkdir /tmp/mnt
# mount /dev/mapper/loop0p1 /tmp/mnt

Modify the following files to reflect the new network settings at the deployment site:
- /tmp/mnt/etc/hosts
- /tmp/mnt/etc/sysconfig/static-routes
- /tmp/mnt/etc/sysconfig/network
- /tmp/mnt/etc/sysconfig/network-scripts/ifcfg-eth0
- /tmp/mnt/etc/sysconfig/network-scripts/ifcfg-eth1
- /tmp/mnt/etc/resolv.conf
Create /tmp/mnt/root/vc-out.xml file use this template - vc-out.xml

# umount /tmp/mnt
# kpartx -dv calit2-119-221
failed to stat() calit2-119-221
# kpartx -dv /state/partition2/kvm/disks/calit2-119-221.hda
del devmap : loop0p5
del devmap : loop0p4
del devmap : loop0p3
del devmap : loop0p2
del devmap : loop0p1
loop deleted : /dev/loop0
  • boot up the frontend
# rocks set host boot action=os calit2-119-221
# rocks start host vm calit2-119-221

Deploy Compute Nodes

  • Check the 3 compute nodes disk paths and modify if needed
# rocks list host vm showdisks=yes
VM-HOST          SLICE MEM   CPUS MAC               HOST             VIRT-TYPE DISK                                                         DISKSIZE
calit2-119-221:  0     1024  1    7a:77:6e:40:00:02 calit2-119-121   hvm       file:/state/partition2/kvm/disks/calit2-119-221.hda,hda,ide     36
calit2-119-221:  ----- ----- ---- 7a:77:6e:40:00:03 ---------------- --------- -------------------------------------------------------------- --------
hosted-vm-0-0-0: 0     1024  1    7a:77:6e:40:00:04 vm-container-0-0 hvm       file:/state/partition2/kvm/disks/hosted-vm-0-0-0.vda,vda,virtio 36
hosted-vm-0-1-0: 0     1024  1    7a:77:6e:40:00:05 vm-container-0-1 hvm       file:/state/partition2/kvm/disks/hosted-vm-0-1-0.vda,vda,virtio 36
hosted-vm-0-2-0: 0     1024  1    7a:77:6e:40:00:06 vm-container-0-2 hvm       file:/state/partition2/kvm/disks/hosted-vm-0-2-0.vda,vda,virtio 36
# rocks set host vm hosted-vm-0-0-0 disk=file:/state/partition2/kvm/disks/hosted-vm-0-0-0.hda,hda,ide
# rocks set host vm hosted-vm-0-1-0 disk=file:/state/partition2/kvm/disks/hosted-vm-0-1-0.hda,hda,ide
# rocks set host vm hosted-vm-0-2-0 disk=file:/state/partition2/kvm/disks/hosted-vm-0-2-0.hda,hda,ide
  • Copy the compute node VM image file to the 3 above vm-containers and paths
  • Modify each of the 3 compute node image files

Ssh to the appropriate vm-container, then do (compute node hosted-vm-0-0-0 as an example):

# kpartx -av /state/partition2/kvm/disks/hosted-vm-0-0-0.hda
add map loop0p1 (253:0): 0 37013504 linear /dev/loop0 2048
add map loop0p2 (253:1): 0 2045952 linear /dev/loop0 37015552
# mkdir /tmp/mnt
# mount /dev/mapper/loop0p1 /tmp/mnt

Modify the following files to reflect the network settings for the compute node at the deployment site:
/tmp/mnt/etc/hosts
/tmp/mnt/etc/sysconfig/static-routes
/tmp/mnt/etc/sysconfig/network
/tmp/mnt/etc/sysconfig/network-scripts/ifcfg-eth0

# umount /tmp/mnt
# kpartx -dv /state/partition2/kvm/disks/hosted-vm-0-0-0.hda
del devmap : loop0p2
del devmap : loop0p1
loop deleted : /dev/loop0
  • Boot up the compute nodes (Best to wait after reconfiguration of the frontend)

Back to the frontend of the VM hosting server, do

# rocks set host boot action=os hosted-vm-0-0-0
# rocks set host boot action=os hosted-vm-0-1-0
# rocks set host boot action=os hosted-vm-0-2-0
# rocks start host vm hosted-vm-0-0-0
# rocks start host vm hosted-vm-0-1-0
# rocks start host vm hosted-vm-0-2-0

Reconfigure the deployed new VC

Should be done by the VC owner.
This example is Rocks-specific. The changes done above are only in some system files, not complete, not in Rocks database. The changes are just enough to boot up and get basic network access temporarily. When run rocks sync ..., the changes made in files will be overwritten. So we must complete the changes in Rocks database and in other configuration files.

Reconfigure the frontend

  • Change hostname in Rocks database
# /opt/rocks/bin/mysql -u apache
...
mysql> update catindex set Name='calit2-119-221' where Name='calit2-119-222' and Category=(select ID from categories where Name='host');
Query OK, 1 row affected (0.01 sec)
Rows matched: 1  Changed: 1  Warnings: 0
mysql> exit
Bye
# rocks set host name frontend calit2-119-221
# rocks set attr hostname calit2-119-221
  • Change network settings in Rocks database

Run "rocks list host interface" and "rocks list attr" and correct every wrong settings. You may need more changes then this example below shows.

# rocks set host interface name frontend eth0 calit2-119-221
# rocks set host interface mac frontend eth0 7A:77:6E:40:00:02
# rocks set host interface mac frontend eth1 7A:77:6E:40:00:03
# rocks set host interface name frontend eth1 calit2-119-221
# rocks set host interface ip frontend eth1 137.110.119.221
# rocks set attr Kickstart_PublicHostname calit2-119-221.ucsd.edu
# rocks set attr Kickstart_PublicAddress 137.110.119.221
# rocks set attr Kickstart_PrivateHostname calit2-119-221
# rocks set attr Info_ClusterName calit2-119-221
# rocks set attr tripwire_mail root@calit2-119-221.ucsd.edu
  • Update the hostname and IPs in the following files

/etc/auto.share
/etc/yum.repos.d/rocks-local.repo
/etc/ganglia/gmetad.conf
/etc/ganglia/gmond.conf
/etc/postfix/sender-canonical
/etc/httpd/conf.d/rocks.conf
/etc/security/ca/ca.cfg

  • Add compute nodes in Rocks frontend database
# rocks list host
HOST            MEMBERSHIP CPUS RACK RANK RUNACTION INSTALLACTION
calit2-119-221: Frontend   1    0    0    os        install
compute-0-1:    Compute    1    0    1    os        install
# rocks add host compute-0-0 cpus=1 membership=compute rack=0 rank=0
# rocks add host compute-0-2 cpus=1 membership=compute rack=0 rank=2
# rocks list host
HOST            MEMBERSHIP CPUS RACK RANK RUNACTION INSTALLACTION
calit2-119-221: Frontend   1    0    0    os        install
compute-0-0:    Compute    1    0    0    os        install
compute-0-1:    Compute    1    0    1    os        install
compute-0-2:    Compute    1    0    2    os        install
  • Add network interface for compute nodes
# rocks list host interface
HOST            SUBNET  IFACE MAC               IP              NETMASK       MODULE NAME           VLAN OPTIONS CHANNEL
calit2-119-221: private eth0  7A:77:6E:40:00:02 10.1.1.1        255.255.0.0   ------ calit2-119-221 ---- ------- -------
calit2-119-221: public  eth1  7A:77:6E:40:00:03 137.110.119.221 255.255.255.0 ------ calit2-119-221 ---- ------- -------
compute-0-1:    private eth0  7a:77:6e:40:00:0a 10.1.255.253    255.255.0.0   ------ compute-0-1    ---- ------- -------
# rocks add host interface compute-0-0 iface=eth0 subnet=private ip=10.1.255.254 netmask=255.255.255.0
# rocks add host interface compute-0-2 iface=eth0 subnet=private ip=10.1.255.252 netmask=255.255.0.0
# rocks list host interface
HOST            SUBNET  IFACE MAC               IP              NETMASK       MODULE NAME           VLAN OPTIONS CHANNEL
calit2-119-221: private eth0  7A:77:6E:40:00:02 10.1.1.1        255.255.0.0   ------ calit2-119-221 ---- ------- -------
calit2-119-221: public  eth1  7A:77:6E:40:00:03 137.110.119.221 255.255.255.0 ------ calit2-119-221 ---- ------- -------
compute-0-0:    private eth0  ----------------- 10.1.255.254    255.255.0.0   ------ compute-0-0    ---- ------- -------
compute-0-1:    private eth0  7a:77:6e:40:00:0a 10.1.255.253    255.255.0.0   ------ compute-0-1    ---- ------- -------
compute-0-2:    private eth0  ----------------- 10.1.255.252    255.255.0.0   ------ compute-0-2    ---- ------- -------
# rocks set host interface mac compute-0-0 eth0 7a:77:6e:40:00:04
# rocks set host interface mac compute-0-1 eth0 7a:77:6e:40:00:05
# rocks set host interface mac compute-0-2 eth0 7a:77:6e:40:00:06
[root@calit2-119-221 ~]# rocks list host interface                              HOST            SUBNET  IFACE MAC               IP              NETMASK       MODULE NAME           VLAN OPTIONS CHANNEL
calit2-119-221: private eth0  7A:77:6E:40:00:02 10.1.1.1        255.255.0.0   ------ calit2-119-221 ---- ------- -------
calit2-119-221: public  eth1  7A:77:6E:40:00:03 137.110.119.221 255.255.255.0 ------ calit2-119-221 ---- ------- -------
compute-0-0:    private eth0  7a:77:6e:40:00:04 10.1.255.254    255.255.0.0   ------ compute-0-0    ---- ------- -------
compute-0-1:    private eth0  7a:77:6e:40:00:05 10.1.255.253    255.255.0.0   ------ compute-0-1    ---- ------- -------
compute-0-2:    private eth0  7a:77:6e:40:00:06 10.1.255.252    255.255.0.0   ------ compute-0-2    ---- ------- -------
  • Set compute nodes boot actions
rocks set host boot action=os compute-0-0
rocks set host boot action=os compute-0-1
rocks set host boot action=os compute-0-2
  • Synch all changes
# rocks sync config
# rocks report host interface localhost |rocks report script | bash
  • Reboot the frontend

Reconfigure Compute Node

  • Update the hostname and IPs in these files on each compute node

/etc/ganglia/gmond.conf
/etc/yum.repos.d/rocks-local.repo

  • Reboot all compute nodes

Test the new VC

  • Test ssh connection to the compute nodes
  • Add a new user on the frontend, then run "rocks sync users", check /etc/passwd files on the compute nodes
  • Su to the new user on the frontend, ssh to compute node and run "df" to check if your home directory is properly mounted