KVM - Rocks VM (Xen)

From PRAGMA wiki
Jump to: navigation, search

Introduction

  • Migrate simple VM between different sites with either Rocks VM or KVM platforms
  • Contact:
    • Rocks/Xen:
      • SDSC: Phil Papadopoulos, Cindy Zheng
      • AIST: Yoshio Tanaka, Akihiko Ota
    • Ubuntu/KVM:
      • NCHC: Weicheng Huang, Serena Pan

Experiments

Rocks VM -> another Rocks VM hosting server

  • Create a compute node in a VM container at SDSC VM hosting server
[root@fiji ~]# rocks add host vm vm-contrainer-0-15 membership="Hosted VM"
[root@fiji ~]# rocks set host interface ip hosted-vm-0-15-1 eth0 10.1.2.254
[root@fiji ~]# rocks set host interface subnet hosted-vm-0-15-1 eth0 private
[root@fiji ~]# rocks list host interface hosted-vm-0-15-1
SUBNET  IFACE MAC               IP       NETMASK     MODULE NAME             VLAN OPTIONS CHANNEL
private eth0  b6:58:ca:00:00:5e 10.1.2.254 255.255.0.0 ------ hosted-vm-0-15-1 ---- ------- -------
[root@fiji ~]# rocks sync config

  • Place AIST VM image for the compute node
[root@fiji ~]# rocks list host vm hosted-vm-0-15-1 showdisks=y
SLICE MEM   CPUS MAC               HOST              DISK
                                 DISKSIZE
1     1024  1    b6:58:ca:00:00:5e vm-container-0-15 file:/state/partition1/xen/disks/hosted-vm-0-15-1.hda,hda,w 36
[root@fiji ~]# ssh vm-container-0-15
[root@vm-container-0-15 disks]# cd /state/partition1/xen/disks
[root@vm-container-0-15 disks]# wget http://www.apgrid.org/frontend-0-0-9.pywps_kvm.hda.gz
[root@vm-container-0-15 disks]# mv frontend-0-0-9.pywps_kvm.hda.gz hosted-vm-0-15-1.hda.gz
[root@vm-container-0-15 disks]# gunzip hosted-vm-0-15-1.hda.gz
[root@vm-container-0-15 disks]# ls -l hosted-vm-0-15-1.hda
-rw-r--r-- 1 root root 36000000000 Feb  8 18:48 hosted-vm-0-15-1.hda
  • Modify the guest VM image
[root@vm-container-0-15 disks]# lomount -partition 1 -diskimage /state/partition1/xen/disks/hosted-vm-0-15-1.hda /mnt
[root@vm-container-0-15 disks]# vi /mnt/root/.ssh/authorized_keys
<< Add your ssh public key there >>
[root@vm-container-0-15 disks]# vi /mnt/etc/sysconfig/network-scripts/ifcfg-eth0
<< Change HWADDR and IPADDR to be consistent with host interface settings >>
[root@vm-container-0-15 disks]# cat /mnt/etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
HWADDR=b6:58:ca:00:00:5e
IPADDR=10.1.2.254
NETMASK=255.255.0.0
BOOTPROTO=static
ONBOOT=yes
MTU=1500
[root@vm-container-0-15 disks]# umount /mnt
  • Boot VM
[root@fiji ~]# rocks set host boot action=os hosted-vm-0-15-1
[root@fiji ~]# rocks list host boot hosted-vm-0-15-1
ACTION
os
[root@fiji ~]# rocks start host vm hosted-vm-0-15-1
[root@fiji ~]# rocks list host vm hosted-vm-0-15-1 status=yes
SLICE MEM   CPUS MAC               HOST              STATUS
1     1024  1    b6:58:ca:00:00:5e vm-container-0-15 active
  • Login
[root@fiji ~]# ssh hosted-vm-0-15-1
Last login: Thu Feb 10 07:01:44 2011 from wpsvm.local
Rocks 5.3 (Rolled Tacos)
Profile built 09:25 03-Feb-2011

Kickstarted 18:55 03-Feb-2011
  • Test run BLOSS application
[root@wpsvm ~]# useradd -c "Cindy Zheng" zhengc
[root@wpsvm ~]# su - zhengc
[zhengc@wpsvm ~]$ tar xvzpf /state/partition1/home/ota/bloss_test.tar.gz
bloss_test/
bloss_test/dsy500_a.mat
bloss_test/dsy_rr_lapack
bloss_test/dsy500_b.mat
bloss_test/dsy2000_a.mat
bloss_test/libgfortran.so.1
bloss_test/dsy100_a.mat
bloss_test/dsy1000_b.mat
bloss_test/dsy1000_a.mat
bloss_test/dsy100_b.mat
bloss_test/bench.sh
bloss_test/dsy2000_b.mat
[zhengc@wpsvm ~]$ cd bloss_test/
[zhengc@wpsvm bloss_test]$ source /usr/lib64/openmpi/1.4-gcc/etc/mpivars.sh
[zhengc@wpsvm bloss_test]$ ./bench.sh
librdmacm: couldn't read ABI version.
librdmacm: assuming: 4
--------------------------------------------------------------------------
[[11844,1],0]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:

Module: OpenFabrics (openib)
  Host: wpsvm.apgrid.org

Another transport will be used instead, although this may result in
lower performance.
--------------------------------------------------------------------------
librdmacm: couldn't read ABI version.
librdmacm: assuming: 4
--------------------------------------------------------------------------
[[11841,1],0]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:

Module: OpenFabrics (openib)
  Host: wpsvm.apgrid.org

Another transport will be used instead, although this may result in
lower performance.
--------------------------------------------------------------------------
librdmacm: couldn't read ABI version.
librdmacm: assuming: 4
--------------------------------------------------------------------------
[[11854,1],0]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:

Module: OpenFabrics (openib)
  Host: wpsvm.apgrid.org

Another transport will be used instead, although this may result in
lower performance.
--------------------------------------------------------------------------
librdmacm: couldn't read ABI version.
librdmacm: assuming: 4
--------------------------------------------------------------------------
[[11851,1],0]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:

Module: OpenFabrics (openib)
  Host: wpsvm.apgrid.org

Another transport will be used instead, although this may result in
lower performance.
--------------------------------------------------------------------------

KVM VM -> Rocks VM hosting server

  • Create a VM
[root@fiji ~]# rocks add host vm fiji.rocksclusters.org membership="Hosted VM" name=fmotif
added VM fmotif on physical node fiji
  • Set VM network interface to public
[root@fiji ~]# rocks set host interface subnet fmotif eth0 public
  • Get VM MAC address and disk image path
[root@fiji ~]# rocks list host interface fmotif
SUBNET IFACE MAC               IP NETMASK MODULE NAME   VLAN OPTIONS CHANNEL
public eth0  b6:58:ca:00:00:60 -- ------- ------ fmotif ---- ------- -------
[root@fiji ~]# rocks list host vm fmotif showdisks=y
SLICE MEM   CPUS MAC               HOST DISK                                              DISKSIZE
14    1024  1    b6:58:ca:00:00:60 fiji file:/state/partition1/xen/disks/fmotif.hda,hda,w 36
  • Copy disk image file to new VM file location
[root@fiji ~]# cd /state/partition1/xen/disks
[root@fiji disks]# wget http://snowfox.nchc.org.tw/download/ubuntu-10.04-amd64-fmotif.img.gz
[root@fiji disks]# mv ubuntu-10.04-amd64-fmotif.img.gz fmotif.hda.gz
[root@fiji disks]# gunzip fmotif.hda.gz
  • Modify disk image before boot
[root@fiji disks]# lomount -diskimage fmotif.hda -partition 1 /media
[root@fiji disks]# vi /media/etc/network/interfaces
[root@fiji disks]# cat /media/etc/network/interfaces
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
        address 198.202.88.204
#       hwaddress ether 00:55:60:48:18:60
        netmask 255.255.255.0
        network 198.202.88.0
        broadcast 198.202.88.255
        gateway 198.202.88.20
  • Add your ssh public key to /media/home/nchc/.ssh/authorized_keys file
  • Find the Ubuntu kernel and ramdisk image file paths in /media/boot/grub/grub.cfg, in lines like these
linux /boot/vmlinuz-2.6.32-21-generic root=UUID=85376c09-2271-4ab0-b43e-70d2598610e3 ro quiet splash
initrd /boot/initrd.img-2.6.32-21-generic
  • Copy the image files to your VM hosting server
[root@fiji disks]# cp /media/boot/vmlinuz-2.6.32-21-generic /var/www/html/ubuntu
[root@fiji disks]# cp /media/boot/initrd.img-2.6.32-21-generic /var/www/html/ubuntu
  • Add a new bootaction for Ubontu VM
[root@fiji disks]# rocks add bootaction action=ubuntu args="root=/dev/xvda1 ro quiet" kernel=http://10.1.1.1/ubuntu/vmlinuz-2.6.32-21-generic ramdisk=http://10.1.1.1/ubuntu/initrd.img-2.6.32-21-generic
[root@fiji disks]# rocks set host runaction fmotif ubuntu
  • Boot VM
[root@fiji disks]# umount /media
[root@fiji disks]# rocks set host boot action=os fmotif
[root@fiji disks]# rocks list host boot fmotif
ACTION
os
[root@fiji disks]# rocks start host vm fmotif
  • Test ssh access
$ ssh nchc@rocks-204.sdsc.edu
The authenticity of host 'rocks-204.sdsc.edu (198.202.88.204)' can't be established.
RSA key fingerprint is c6:12:33:95:ed:e1:de:a4:62:75:66:1c:ff:b3:d5:6c.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'rocks-204.sdsc.edu,198.202.88.204' (RSA) to the list of known hosts.
Enter passphrase for key '/home/zhengc/.ssh/id_rsa':
Linux bio002 2.6.32-21-generic #32-Ubuntu SMP Fri Apr 16 08:09:38 UTC 2010 x86_64 GNU/Linux
Ubuntu 10.04.2 LTS

Welcome to Ubuntu!
 * Documentation:  https://help.ubuntu.com/

Last login: Thu Feb 17 16:00:51 2011 from node01.local
nchc@bio002:~$
  • Test web interface

Point your browser to http://rocks-204.sdsc.edu

Rocks (Xen) VM -> KVM hosting server

  • On Rocks VM hosting server, replace DomU Xen Kernel with a kernel that understands KVM virtio drivers
[ota@compute-0-15]% sudo lomount -diskimage frontend-0-0-9.pywps.hda -partition 1 /mnt/
[ota@compute-0-15]% sudo lomount -diskimage frontend-0-0-9.pywps.hda -partition 2 /mnt/var
[ota@compute-0-15]% sudo chroot /mnt
[root@compute-0-15]/# yum install kernel.x86_64
...
====================================================================================================
 Package            Arch               Version                            Repository           Size
====================================================================================================
Installing:
 kernel             x86_64             2.6.18-194.32.1.el5                updates              19 M

Transaction Summary
====================================================================================================
Install      1 Package(s)         
Update       0 Package(s)         
Remove       0 Package(s)         
...
  • Modify grub.conf to point to the new (KVM) kernel
[root@compute-0-15]/# vi /boot/grub/grub.conf
(Add the following entry)
+title CentOS 5.5 (2.6.18-194.32.1.el5)
+      root (hd0,0)
+      kernel /boot/vmlinuz-2.6.18-194.32.1.el5 ro root=LABEL=/ rhgb quiet console=tty0 console=ttyS0,115200
+      initrd /boot/initrd-2.6.18-194.32.1.el5.img

And modify "default=" properly.
  • Edit modprobe.conf to alias the disk and network devices properly
[root@compute-0-15]/# vi /etc/modprobe.conf
(modify as follows)
-alias scsi_hostadapter xenblk
-alias eth0 xennet
-alias eth1 xennet
+alias scsi_hostadapter ata_piix
+alias eth0 virtio_net
+alias eth1 virtio_net
  • Boot this image by KVM as follows:
[ota@compute-0-15]% sudo /opt/qemu-kvm/bin/qemu-system-x86_64 \
-m 1024 \
-net nic,model=virtio,macaddr=00:16:3e:1d:ff:0f \
-net tap,ifname=tap0,script=no,downscript=no \
-vnc :0 \
-serial stdio \
-drive file=./frontend-0-0-9.pywps.hda,boot=on 

Note: This example is using qemu-kvm version 0.13.50 compiled from source of git repository.

  • On KVM hosting server, modify network parameters in the VM image, such as hostname, ip address, gateway, DNS server.