Condor-ViNe test

From PRAGMA wiki
Jump to: navigation, search

Network environment

  • See UCSD implementation. Specifically, vm-container-0-13 and vm-container-0-14 are in ViNe subnet.
  • Added a network route (physical and network configuration) to 198.202.88.0 on vm-container-0-14
  • Added VLAN 4001 for the new route
# rocks add host interface vm-container-0-14 vlan4001 subnet=private vlan=4001

Create virtual cluster

  • A Rocks virtual cluster with condor roll is created on fiji
# rocks add cluster 192.202.88.214 1 container-hosts=vm-container-0-13 fe-container=vm-container-0-14 fe-name=rocks-214 vlan=354
# rocks set host interface vlan rocks-214 eth1 4001
  • Set DHCP on compute node
# rocks set host interface options compute-0-0 eth0 dhcp
# rocks sync host network compute-0-0

Make compute node disk image

On compute-0-0

  • Prevent compute node to reinstall on boot
# chkconfig --del rocks-grub
# poweroff

On vm-container-0-13, gzip compute node disk image to /share/vine-disk

# gzip -c /state/partition1/kvm/disks/hosted-vm-0-13-1.vda > /share/vine-disk/vm-images/SDSC/vine_condor.img.gz

Deploy the compute node image to UFL ViNe subnet

  • Download the VM image

On UFL VMM, /share/vine-disk is NSF mounted.

# gunzip -c /share/vine-disk/vm-images/SDSC/vine_condor.img.gz > /var/lib/libvirt/images/pg/vine_condor.img
  • Modify the image

Some configuration files in the VM image need to be edited. To access the files, the filesystem in the image needs to be mounted on the VMM.

# kpartx -a /var/lib/libvirt/images/pg/vine_condor.img
# mount /dev/mapper/loop0P1 /mnt

1. Edit /mnt/etc/sysconfig/network-scripts/ifcfg-eth0 and comment out the MAC address of eth0

2. Edit /mnt/etc/sysconfig/network and change the HOSTNAME

3. Edit /mnt/etc/hosts and add hosts as needed

4. Edit /mnt/etc/udev/rules.d/70-persistent-net.rules and remove all entries

5. Create a file /mnt/etc/sysconfig/network-scripts/route-eth0 with a static route needed to join ViNe overlay.

# echo "172.16.0.0/12 via 172.31.10.34 dev eth0" >> /mnt/etc/sysconfig/network-scripts/route-eth0

On the VMM, unmount the image.

# umount /mnt
# kpartx -d /var/lib/libvirt/images/pg/vine_condor.img

Start the VM.

# /usr/libexec/qemu-kvm -m 1024 -hda /var/lib/libvirt/images/pg/vine_condor.img -net nic,macaddr=54:52:00:1e:c5:98 -net tap,script=/root/pragma/qemu-ifup

To get condor to work between rocks-214.sdsc.edu and compute-ufl-0-1

Changes made on UFL compute node

These changes can be included in the compute node disk image

  • Modify /opt/condor/etc/confor_config.local
# cd /opt/condor/etc
# diff condor_config.local condor_config.local.backup
9c9
< CONDOR_HOST = 172.31.11.100
---
> CONDOR_HOST = rocks-214.sdsc.edu
18c18
< HOSTALLOW_WRITE = 172.31.11.100, rocks-214.sdsc.edu, *.local, *.local
---
> HOSTALLOW_WRITE = rocks-214.sdsc.edu, *.local, *.local
25a26
> NETWORK_INTERFACE = 172.31.11.254
  • Restart condor
# service rocks-condor stop
# service rocks-condor start
  • Add the line "172.31.11.100 rocks-214.local" to /etc/hosts

Changes made on rocks-214.sdsc.edu (condor master)

  • Change Condor attribute
# rocks set attr Condor_HostAllow value="198.202.88.214,172.16.0.0/12"
# rocks sync condor
  • Add a static route to the whole ViNe subnet
# rocks add route 172.16.0.0 172.31.11.1 netmask=255.240.0.0
# rocks sync host network compute-0-0 rocks-214
  • Change /etc/exports and restart NFS
# cat /etc/exports
/export 172.31.11.100(rw,async,no_root_squash) 172.31.11.0/255.255.255.0(rw,async) 172.16.0.0/255.240.0.0(rw,async)
# service nfs restart