Network overlay setup at SDSC

From PRAGMA wiki
Revision as of 12:59, 4 April 2012 by Cindysdsc (Talk)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

OVS structure diagram

Ovs-sdsc.jpg

On rocks-68

Build Open vSwitch

  • Get openswitch package
$ wget http://openvswitch.org/releases/openvswitch-1.4.0.tar.gz
$ cp openvswitch-1.4.0.tar.gz /path/to/rpmbuild_dir/SOURCES/
$ tar xvzf openvswitch-1.4.0.tar.gz
$ cd openvswitch-1.4.0
  • Modify rhel/openvswitch-kmod-rhel5.spec
# uname -a
Linux rocks-68.sdsc.edu 2.6.18-274.18.1.el5 #1 SMP Thu Feb 9 12:45:44 EST 2012 x86_64 x86_64 x86_64 GNU/Linux
# diff rhel/openvswitch-kmod-rhel5-original.spec rhel/openvswitch-kmod-rhel5.spec
13c13
< %{!?kversion: %define kversion 2.6.18-238.12.1.el5}
---
> %{!?kversion: %define kversion 2.6.18-274.18.1.el5}
36,38d35
< %ifarch i686 x86_64
< %define xenvar xen
< %endif
41c38
< %{!?kvariants: %define kvariants %{?basevar} %{?xenvar} %{?paevar}}
---
> %{!?kvariants: %define kvariants %{?basevar} %{?paevar}}
46,48d42
< # Disable the building of the debug package(s).
< %define debug_package %{nil}
  • Build RPMs
$ rpmbuild -bb rhel/openvswitch.spec
$ cp rhel/kmodtool-openvswitch-rhel5.sh ..
$ rpmbuild -bb -D "kversion 2.6.18-274.18.1.el5" --target x86_64 rhel/openvswitch-kmod-rhel5.spec

The above commands produces the following packages in your RPMS dir.

# ls -l ../../RPMS/x86_64/
total 2596
-rw-r--r-- 1 root root   49915 Mar 19 12:13 kmod-openvswitch-1.4.0-1.x86_64.rpm
-rw-r--r-- 1 root root 2594813 Mar 19 12:13 openvswitch-1.4.0-1.x86_64.rpm

Install and setup

  • Install these packages.
# rpm -ivh openvswitch-1.4.0-1.x86_64.rpm kmod-openvswitch-1.4.0-1.x86_64.rpm kmod-openvswitch-xen-1.4.0-1.x86_64.rpm
  • Enable Linux bridge device compatibility mode by uncomment the line of "BRCOMPAT=yes" in /etc/sysconfig/openvswitch.
  • Start Open vSwitch services.
# service openvswitch start
Inserting openvswitch module                               [  OK  ]
Inserting brcompat module                                  [  OK  ]
/etc/openvswitch/conf.db does not exist ... (warning).
Creating empty database /etc/openvswitch/conf.db           [  OK  ]
Starting ovsdb-server                                      [  OK  ]
Configuring Open vSwitch system IDs                        [  OK  ]
Starting ovs-vswitchd                                      [  OK  ]
Starting ovs-brcompatd                                     [  OK  ]
Enabling gre with iptables                                 [  OK  ]
  • Check if Default bridge device is removed
# lsmod | grep bridge

Should yield no output.

  • Check if /etc/openvswitch/conf.db is generated.
  • Check if ovsdb-server ovs-vswitchd ovs-brcompatd is running.
# ps -ef | grep ovsdb-server
root      3621     1  0 12:22 pts/1    00:00:00 ovsdb-server: monitoring pid 3622 (healthy)
root      3622  3621  0 12:22 ?        00:00:00 ovsdb-server /etc/openvswitch/conf.db -vANY:CONSOLE:EMER -vANY:SYSLOG:ERR vANY:FILE:INFO --remote=punix:/var/run/openvswitch/db.sock --remote=db:Open_vSwitch,manager_options --private-key=db:SSL,private_key --certificate=db:SSL,certificate --bootstrap-ca-cert=db:SSL,ca_cert --no-chdir --log-file=/var/log/openvswitch/ovsdb-server.log --pidfile=/var/run/openvswitch/ovsdb-server.pid --detach --monitor
# ps -ef | grep ovs-vswitchd
root      3630     1  0 12:22 pts/1    00:00:00 ovs-vswitchd: monitoring pid 3631 (healthy)
root      3631  3630  0 12:22 ?        00:00:00 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vANY:CONSOLE:EMER -vANY:SYSLOG:ERR -vANY:FILE:INFO --mlockall --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor
# ps -ef | grep ovs-brcompatd
root      3633     1  0 12:22 pts/1    00:00:00 ovs-brcompatd: monitoring pid 3634 (healthy)
root      3634  3633  0 12:22 ?        00:00:00 ovs-brcompatd -vANY:CONSOLE:EMER -vANY:SYSLOG:ERR -vANY:FILE:INFO --no-chdir --log-file=/var/log/openvswitch/ovs -brcompatd.log --pidfile=/var/run/openvswitch/ovs-brcompatd.pid --detach --monitor
  • Add a policy to pass GRE protocol in iptables.

'service openvswitch start' automatically adds a policy to pass GRE in iptables. But, to ensure that the policy is always enabled, add the policy in your iptables rule file.

# rocks add firewall host=frontend network=all protocol=gre service=all chain=INPUT action=ACCEPT rulename=A70-ALL-PUBLIC-INPUT
# rocks add firewall host=frontend network=all protocol=gre service=all chain=OUTPUT action=ACCEPT rulename=A70-ALL-PUBLIC-OUTPUT
# rocks sync host firewall rocks-68
# grep gre /etc/sysconfig/iptables
-A INPUT -p gre -j ACCEPT
-A OUTPUT -p gre -j ACCEPT
# service iptables restart
  • Create an Open vSwitch and set a Openflow controller
# ovs-vsctl add-br br0
# ovs-vsctl set-controller br0 tcp:133.1.134.167
  • Confirm the vSwitch status
# ovs-vsctl show
<pre>
...
    Bridge "br0"
        Controller "tcp:133.1.134.167"
            is_connected: true
        Port "br0"
            Interface "br0"
                type: internal
...

If you find the lines, 'Controller "tcp:133.1.134.167"' and ' is_connected: true', it means that your Open vSwitch connects properly to the Openflow controller at Osaka University.

  • On vm-container-0-14, create a GRE connection to rocks-68
# ovs-vsctl add-port br0 gre1 -- set interface gre1 type=gre options:remote_ip=10.1.255.68
  • On rocks-68, create a GRE connection to vm-container-0-14
# ovs-vsctl add-port br0 gre14 -- set interface gre14 type=gre options:remote_ip=10.1.255.240

Let Ichikawa-san know the global IP address of the host where the vSwitch is hosted. From Osaka university, Ichikawa-san will make a reverse GRE connection to your vSwitch.

Setup private network

  • Join eth0 of rocks-68 to the vm-containers' 10.1.255.x address space.
[root@rocks-68 x86_64]# rocks list host interface
SUBNET  IFACE MAC               IP            NETMASK       MODULE NAME     VLAN
 OPTIONS CHANNEL
private eth0  00:12:3F:20:C6:81 10.1.1.1      255.255.0.0   ------ rocks-68 ----
 ------- -------
public  eth1  00:12:3F:20:C6:82 198.202.88.68 255.255.255.0 ------ rocks-68 ----
 ------- -------
[root@rocks-68 x86_64]# rocks set host interface ip rocks-68 eth0 10.1.255.68
[root@rocks-68 x86_64]# rocks list host interface   rocks-
SUBNET  IFACE MAC               IP            NETMASK       MODULE NAME     VLAN
 OPTIONS CHANNEL
private eth0  00:12:3F:20:C6:81 10.1.255.68   255.255.0.0   ------ rocks-68 ----
 ------- -------
public  eth1  00:12:3F:20:C6:82 198.202.88.68 255.255.255.0 ------ rocks-68 ----
 ------- -------
  • Make sure that rocks-68 eth0 is connected to right private net switch

On vm-container

Build Open vSwitch

  • Get openswitch package
$ wget http://openvswitch.org/releases/openvswitch-1.4.0.tar.gz
$ cp openvswitch-1.4.0.tar.gz /path/to/rpmbuild_dir/SOURCES/
$ tar xvzf openvswitch-1.4.0.tar.gz
$ cd openvswitch-1.4.0
  • Modify rhel/openvswitch-kmod-rhel5.spec
# uname -a
Linux vm-container-0-14.local 2.6.18-238.19.1.el5xen #1 SMP Fri Jul 15 08:16:59 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux
# diff rhel/openvswitch-kmod-rhel5-original.spec rhel/openvswitch-kmod-rhel5.spec
13c13
< %{!?kversion: %define kversion 2.6.18-238.12.1.el5}
---
< %{!?kversion: %define kversion 2.6.18-238.19.1.el5}
  • Build RPMs

Run "rpmbuild -bb rhel/openvswitch.spec", output indicate many missing dependencies. Get the missing rpm files from frontend:/export/rocks/install/roll...

# ls
e2fsprogs-devel-1.39-23.el5_5.1.i386.rpm
e2fsprogs-devel-1.39-23.el5_5.1.x86_64.rpm
kernel-2.6.18-238.19.1.el5.x86_64.rpm
keyutils-libs-devel-1.2-1.el5.i386.rpm
keyutils-libs-devel-1.2-1.el5.x86_64.rpm
krb5-devel-1.6.1-55.el5.i386.rpm
krb5-devel-1.6.1-55.el5.x86_64.rpm
krb5-libs-1.6.1-55.el5.i386.rpm
krb5-libs-1.6.1-55.el5.x86_64.rpm
libselinux-devel-1.33.4-5.7.el5.i386.rpm
libselinux-devel-1.33.4-5.7.el5.x86_64.rpm
libsepol-devel-1.15.2-3.el5.i386.rpm
libsepol-devel-1.15.2-3.el5.x86_64.rpm
openssl-devel-0.9.8e-12.el5_5.7.i386.rpm
openssl-devel-0.9.8e-12.el5_5.7.x86_64.rpm
zlib-devel-1.2.3-3.i386.rpm
zlib-devel-1.2.3-3.x86_64.rpm
# rpm -ivh *.x86_64.rpm
Preparing...                ########################################### [100%]
        package krb5-libs-1.6.1-55.el5_6.2.x86_64 (which is newer than krb5-libs-1.6.1-55.el5.x86_64) is already installed
        package krb5-libs-1.6.1-55.el5_6.2.i386 (which is newer than krb5-libs-1.6.1-55.el5.x86_64) is already installed
        file /usr/lib64/libdes425.so.3.0 from install of krb5-libs-1.6.1-55.el5.x86_64 conflicts with file from package krb5-libs-1.6.1-55.el5_6.2.x86_64
        file /usr/lib64/libgssapi_krb5.so.2.2 from install of krb5-libs-1.6.1-55.el5.x86_64 conflicts with file from package krb5-libs-1.6.1-55.el5_6.2.x86_64
        file /usr/lib64/libk5crypto.so.3.1 from install of krb5-libs-1.6.1-55.el5.x86_64 conflicts with file from package krb5-libs-1.6.1-55.el5_6.2.x86_64
        file /usr/lib64/libkadm5clnt.so.5.1 from install of krb5-libs-1.6.1-55.el5.x86_64 conflicts with file from package krb5-libs-1.6.1-55.el5_6.2.x86_64
        file /usr/lib64/libkadm5srv.so.5.1 from install of krb5-libs-1.6.1-55.el5.x86_64 conflicts with file from package krb5-libs-1.6.1-55.el5_6.2.x86_64
        file /usr/lib64/libkdb5.so.4.0 from install of krb5-libs-1.6.1-55.el5.x86_64 conflicts with file from package krb5-libs-1.6.1-55.el5_6.2.x86_64
        file /usr/lib64/libkrb4.so.2.0 from install of krb5-libs-1.6.1-55.el5.x86_64 conflicts with file from package krb5-libs-1.6.1-55.el5_6.2.x86_64
        file /usr/lib64/libkrb5.so.3.3 from install of krb5-libs-1.6.1-55.el5.x86_64 conflicts with file from package krb5-libs-1.6.1-55.el5_6.2.x86_64
        file /usr/lib64/libkrb5support.so.0.1 from install of krb5-libs-1.6.1-55.el5.x86_64 conflicts with file from package krb5-libs-1.6.1-55.el5_6.2.x86_64
# rpm -Uvh --force *.x86_64.rpm
Preparing...                ########################################### [100%]
   1:krb5-libs              ########################################### [ 13%]
   2:zlib-devel             ########################################### [ 25%]
   3:libsepol-devel         ########################################### [ 38%]
   4:libselinux-devel       ########################################### [ 50%]
   5:keyutils-libs-devel    ########################################### [ 63%]
   6:e2fsprogs-devel        ########################################### [ 75%]
   7:krb5-devel             ########################################### [ 88%]
   8:openssl-devel          ########################################### [100%]
$ rpmbuild -bb rhel/openvswitch.spec
$ cp rhel/kmodtool-openvswitch-rhel5.sh ..
$ rpmbuild -bb -D "kversion 2.6.18-238.19.1.el5" --target x86_64 rhel/openvswitch-kmod-rhel5.spec

The above commands produces the following packages in your RPMS dir.

# ls -l ../../RPMS/x86_64
total 2652
-rw-r--r-- 1 root root   49896 Mar 27 15:24 kmod-openvswitch-1.4.0-1.x86_64.rpm
-rw-r--r-- 1 root root   50223 Mar 27 15:24 kmod-openvswitch-xen-1.4.0-1.x86_64.rpm
-rw-r--r-- 1 root root 2594731 Mar 27 15:10 openvswitch-1.4.0-1.x86_64.rpm

Install and setup

  • If kernel rpm is not installed, get it from the frontend and install it.
# rpm -ivh /root/openswitch/rocks/kernel-2.6.18-238.19.1.el5.x86_64.rpm
Preparing...                ########################################### [100%]
   1:kernel                 ########################################### [100%]
Modulefile is /etc/modprobe.conf
  • Install openvswitch rpm's
# cd /usr/src/redhat/RPMS/x86_64
# ls
kmod-openvswitch-1.4.0-1.x86_64.rpm      openvswitch-1.4.0-1.x86_64.rpm
kmod-openvswitch-xen-1.4.0-1.x86_64.rpm
# rpm -ivh *
Preparing...                ########################################### [100%]
   1:kmod-openvswitch       ########################################### [ 33%]
Modulefile is /etc/modprobe.conf
   2:kmod-openvswitch-xen   ########################################### [ 67%]
Modulefile is /etc/modprobe.conf
   3:openvswitch            ########################################### [100%]
  • Enable Linux bridge device compatibility mode by uncomment the line of "BRCOMPAT=yes" in /etc/sysconfig/openvswitch.
  • Start Open vSwitch services.
service openvswitch start

If you have bridge setup/running previously, you may get "invalid module format" error. In such case, reboot the vm-container before try to start openvswitch service again.

  • Check if Default bridge device is removed
# lsmod | grep bridge

Should yield no output.

  • Check if /etc/openvswitch/conf.db is generated.
  • Check if ovsdb-server ovs-vswitchd ovs-brcompatd is running.
# ps -ef | grep ovsdb-server
root     19346     1  0 11:03 ?        00:00:00 ovsdb-server: monitoring pid 19347 (healthy)
root     19347 19346  0 11:03 ?        00:00:00 ovsdb-server /etc/openvswitch/conf.db -vANY:CONSOLE:EMER -vANY:SYSLOG:ERR -vANY:FILE:INFO --remote=punix:/var/run/openvswitch/db.sock --remote=db:Open_vSwitch,manager_options --private-key=db:SSL,private_key --certificate=db:SSL,certificate --bootstrap-ca-cert=db:SSL,ca_cert --no-chdir --log-file=/var/log/openvswitch/ovsdb-server.log --pidfile=/var/run/openvswitch/ovsdb-server.pid --detach --monitor
# ps -ef | grep ovs-vswitchd
root     19355     1  0 11:03 ?        00:00:00 ovs-vswitchd: monitoring pid 19356 (healthy)
root     19356 19355  0 11:03 ?        00:00:22 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vANY:CONSOLE:EMER -vANY:SYSLOG:ERR -vANY:FILE:INFO --mlockall --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/ope
nvswitch/ovs-vswitchd.pid --detach --monitor
# ps -ef | grep ovs-brcompatd
root     19358     1  0 11:03 ?        00:00:00 ovs-brcompatd: monitoring pid 19359 (healthy)
root     19359 19358  0 11:03 ?        00:00:00 ovs-brcompatd -vANY:CONSOLE:EMER -vANY:SYSLOG:ERR -vANY:FILE:INFO --no-chdir --log-file=/var/log/openvswitch/ovs-brcompatd.log --pidfile=/var/run/openvswitch/ovs-brcompatd.pid --detach --monitor
  • Add a policy to pass GRE protocol in iptables.

'service openvswitch start' automatically adds a policy to pass GRE in iptables. But, to ensure that the policy is always enabled, add the policy in your iptables rule file.
On the VM hosting server frontend, do

# rocks add firewall host=vm-container-0-14 network=all protocol=gre service=all chain=INPUT action=ACCEPT rulename=A70-ALL-PUBLIC-INPUT
# rocks add firewall host=vm-container-0-14 network=all protocol=gre service=all chain=OUTPUT action=ACCEPT rulename=A70-ALL-PUBLIC-OUTPUT
# rocks sync host firewall vm-container-0-14

On vm-container-0-14, check

# grep gre /etc/sysconfig/iptables
-A INPUT -p gre -j ACCEPT
-A OUTPUT -p gre -j ACCEPT
# service iptables restart
  • Create an Open vSwitch and set a Openflow controller
# ovs-vsctl add-br br0
# ovs-vsctl set-controller br0 tcp:133.1.134.167
  • Confirm the vSwitch status
# ovs-vsctl show
<pre>
...
    Bridge "br0"
        Controller "tcp:133.1.134.167"
            is_connected: true
        Port "br0"
            Interface "br0"
                type: internal
...

If you find the lines, 'Controller "tcp:133.1.134.167"' and ' is_connected: true', it means that your Open vSwitch connects properly to the Openflow controller at Osaka University.

  • Create GRE connections between Open vSwitchs
# ovs-vsctl add-port br0 gre14 -- set interface gre14 type=gre options:remote_ip=10.1.255.240

Testing with a VM

  • Create a VM
[root@fiji ~]# rocks add host vm vm-container-0-14 membership="Hosted VM" name=worker_qq
added VM worker_qq on physical node vm-container-0-14
[root@fiji ~]# rocks list host vm worker_qq
SLICE MEM   CPUS MAC               HOST              VIRT-TYPE
3     1024  1    b6:58:ca:00:00:89 vm-container-0-14 para
[root@fiji ~]# rocks set host interface subnet worker_qq eth0 public
[root@fiji ~]# rocks list host interface worker_qq
SUBNET IFACE MAC               IP NETMASK MODULE NAME      VLAN OPTIONS CHANNEL
public eth0  b6:58:ca:00:00:89 -- ------- ------ worker_qq ---- ------- -------
[root@fiji ~]# rocks set host boot action=os worker_qq
[root@fiji ~]# rocks list host boot worker_qq
  • Turn on hardware virtualization in vm-container-0-14's BIO
  • Generate hvm start-up xml file for worker_qq
[root@fiji ~]# rocks set host vm worker_qq virt-type-hvm
[root@fiji ~]# rocks report host vm config worker_qq | tee worker_qq-hvm.xml
<domain type='xen'>
<name>worker_qq</name>
<os>
<type>hvm</type>
<loader>/usr/lib/xen/boot/hvmloader</loader>
<boot dev='network'/>
<boot dev='hd'/>
<bootmenu enable='yes'/>
</os>
<memory>1048576</memory>
<vcpu>1</vcpu>
<features>
        <acpi/>
        <apic/>
        <pae/>
</features>
<devices>
  <emulator>/usr/lib64/xen/bin/qemu-dm</emulator>
  <interface type='bridge'>
    <source bridge='None'/>
    <mac address='b6:58:ca:00:00:89'/>
    <script path='vif-bridge'/>
  </interface>
<disk type='file' device='disk'>
<driver name='file'/>
<source file='/state/partition1/xen/disks/worker_qq.hda'/>
<target dev='hda'/>
</disk>
<graphics type='vnc' port='-1'/>
<console tty='/dev/pts/0'/>
</devices>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>preserve</on_crash>
</domain>

Testing instruction from Osaka

  • Launch a VM and bridge to the Open vSwitch

The following is an example for launching a KVM based VM

# /usr/libexec/qemu-kvm -no-kvm-pit-reinjection -hda centos_5_x86_64-kvm.img -boot c -m 512 -k ja -localtime -net nic,macaddr=54:52:00:12:34:10,model=virtio -net tap,ifname=tap10,script=qemu-ifup,downscript=qemu-ifdown -no-reboot -vnc :10 -serial pty -parallel none -monitor pty -daemonize

The MAC address must be an unique address in our project. Osaka university members use 54:52:00:12:34:xx for the purpose of this feasibility test. Please let me know the MAC address used for your VM.

The samples of qemu-ifup and qemu-ifdown scripts are the below. This script bridges the tap device of KVM to the Open vSwitch(br0) If you plan to use Xen, please edit your xen-bridge script. qemu-ifup script:

#!/bin/sh

switch=br0
echo "Bringing up $1 for bridged mode..."
/sbin/ifconfig $1 0.0.0.0 promisc up
echo "Adding $1 to ${switch}..."
ovs-vsctl add-port ${switch} $1

qemu-ifdown script:

#!/bin/sh

switch=br0
echo "Removing $1 from ${switch}..."
ovs-vsctl del-port ${switch} $1
echo "Shutting down $1..."
/sbin/ifconfig $1 0.0.0.0 down


IP address of the VM:

Osaka members use 10.2.1.1-99 For the purpose of this feasibility test, please use following addresses. 10.2.1.100-199/24 for AIST members. 10.2.1.200-254/24 for UCSD members.


After launched your VM, please let me know the MAC address and IP address of your VM and IP address of your vSwitch. I will make GRE connection from our vSwitch to your vSwitch, and add your VM in the same slice with Osaka university's VMs.

In this way, our VMs can establish an isolated virtual L2 network.