Hello my friend,

Some time ago we have discussed how Ansible could be used as VNF manager for onboarding and deleting the VMs in Linux. I have got some feedbacks, that not everything is easy or could be reproduced, so I decided to find the easier way, how to help you with the creation of virtual labs to train your networking and programming skills (assuming you are using Ansible).

Brief description

The whole OpenConfig series, I have written recently took me a lot of time and efforts. In total part 1, part 2 and part 3 uses 167 A4 pages in Microsoft Word, just like a small book. That’s why I’ll try to write something shorter, but as useful at least as OpenConfig articles.

The second important point for writing this article for me was the fact that I’m using different hypervisors for different VMs. On the one hand, there is a very easy explanation for that, what is the balancing of the used resources versus flexibility. Nevertheless, it prevents me of deploying full automation for lab lifecycle starting from creation till making the resource free again. As I generally find Linux very easy in terms of automation and as it has amazing CLI, I decided to use it as main platform for all kinds of virtualization. In order not to lose performance of VMs, I’m looking in direction of pure KVM without QEMU.

So this blogpost is dedicated to explanation, how different virtual network functions could be deployed on KVM.

What we are going to test?

We are focusing on deployment of VMs with network functions on KVM hypervisor with Linux CentOS 7 as host OS. So I want to find the easiest way to do it via CLI so that I can automate it with Ansible either using specific module for that or generic module for Linux commands.

We’ll deploy 4 different VNFs, details are coming in the next chapter.

Software version

The following infrastructure is used in my lab:

  • CentOS 7 with python 2.7.
  • Ansible 2.6.0
  • Arista EOS 4.20.6F [guest VNF]
  • Nokia SR OS 16.0.R1 [guest VNF]
  • Cisco IOS XR 6.1.2 [guest VNF]
  • Cumulus Linux 3.6.2 [guest VNF]

See the previous article to get details how to build the lab (https://goo.gl/hKhtvX)

Topology

There is no particular topology for this lab, as we are speaking about VNF onboarding. The generic image, how any VNF should be connected you might find on the image below:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
           +                      +                       +
           |ens33                 |ens35                  |ens35
           |ip: 192.168.44.17/24  |                       |
+---------------------------------------------------------------------+
|                                 |                       |           |
|                                 |                       |           |
| +------------------+   +--------+---------+   +---------+--------+  |
| |    Management    |   |     Data 1       |   |     Data 2       |  |
| |       br0        |   |       br1        |   |       br2        |  |
| |ip: 192.168.1.1/24|   |ip: 10.0.0.254/24 |   |ip: 10.0.1.254/24 |  |
| +--------+---------+   +--------+---------+   +---------+--------+  |
|          |                      |                       |           |
|          |eth0                  |eth1                   |eth2       |
|          |                      |                       |           |
| +--------+----------------------+-----------------------+--------+  |
| |                                                                |  |
| |                        New VNF with XX                         |  |
| |                                                                |  |
| +----------------------------------------------------------------+  |
|                                                                     |
|                      Host OS (CentOS) w/ KVM                        |
|                                                                     |
+---------------------------------------------------------------------+

 

The topology above also provides the overview of the logical connectivity, as there is nothing particular, just IP connectivity to Linux bridges.

There is no default configuration files to any of VNFs, as there is no VNFs yet.:

General algorithm for VNF creation and prerequisites

We’ll start with the second point, hence, prerequisites.:

To convert file from “vmdk” to “qcow2” use specific tool “qemu-img convert -f vmdk -O qcow2 source.vmdk target.qcow2”

If both prerequisites are fulfilled, then the rest is quite easy. We do the following steps:

  • Copy VNF image in “qcow2” format to folder with folder dedicated to VNF images, which is “/var/lib/libvirt/images/”.
  • Install VNF using “virt-install” tool according to the physical topology provided above.
  • Update hosts tables for Linux and Ansible

As you see, the set of activities we need to do is quite straightforward, so let’s start with the execution.

#1.1. Onboarding Arista vEOS

You can download the Arista vEOS files from the official website. There also you find official guidelines how to deploy it as VM with QEMU example. Earlier I have explained, how you can install it on VMware Player, so now we streamline things with KVM.

So, we briefly check source folder:

1
2
3
$ ls -l /tmp/ | grep '\.iso\|.vmdk'
-rwxrw-rw-. 1 aaa  aaa    5242880 May 12 12:04 Aboot-veos-8.0.0.iso
-rwxrw-rw-. 1 aaa  aaa  657588224 Jul 18 22:44 vEOS-lab-4.20.7M.vmdk

 

And destination folder:

1
2
$ sudo ls -l /var/lib/libvirt/images/
total 0

 

Now we start deployment of or VNF with Arista vEOS. First of all, we convert “vmdk” to “qcow2”:

1
2
3
4
5
6
7
$ qemu-img convert -f vmdk -O qcow2 vEOS-lab-4.20.7M.vmdk vEOS-lab.qcow2

$ rm vEOS-lab-4.20.7M.vmdk

$ ls -l /tmp/ | grep '\.iso\|\.vmdk\|\.qcow2'
-rwxrw-rw-. 1 aaa  aaa    5242880 May 12 12:04 Aboot-veos-8.0.0.iso
-rw-r--r--. 1 aaa  aaa  657391616 Sep  6 18:45 vEOS-lab.qcow2

 

Next, we copy disc of VNF, what is “qcow2” file to standard folder with VNFs:

1
2
3
4
5
$ sudo cp /tmp/vEOS-lab.qcow2 /var/lib/libvirt/images/vEOS-lab.qcow2

[aaa@sand7 ~]$ sudo ls -l /var/lib/libvirt/images/
total 641984
-rw-r--r--. 1 root root 657391616 Sep  6 18:50 vEOS-lab.qcow2

 

So we have reached the core activity that is installation of VNF with Arista vEOS:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
$ sudo virt-install \
>   --name=vEOS1 \
>   --description "vEOS VM" \
>   --os-type=Linux \
>   --ram=2048 \
>   --vcpus=2 \
>   --boot cdrom,hd,menu=on \
>   --cdrom /tmp/Aboot-veos-8.0.0.iso \
>   --livecd \
>   --disk path=/var/lib/libvirt/images/vEOS1.qcow2,bus=ide,size=4 \
>   --graphics vnc \
>   --serial tcp,host=0.0.0.0:2281,mode=bind,protocol=telnet \
>   --network=bridge:br0,mac=52:54:00:01:02:00,model=virtio \
>   --network=bridge:br1,mac=52:54:00:01:02:01,model=virtio \
>   --network=bridge:br2,mac=52:54:00:01:02:02,model=virtio



WARNING  No operating system detected, VM performance may suffer. Specify an OS with --os-variant for optimal results.

Starting install...

(virt-viewer:39116): Gtk-WARNING **: Allocating size to VncDisplay 0x56146adec250 without calling gtk_widget_get_preferred_width/height(). How does the code know the size to allocate?

 

At this time graphical VNC console starts, so you can see the process of boot there.

As it’s explained in the official documentation from Arista, the file called “Aboot-veos” should be installed in CD-ROM of VM as it’s used during boot. In order to sustain this after reboot, we instruct KVM to treat it as “livecd”, as by default the CD will be removed after the first reboot.

In the command above we define the most critical parameters of the VM, as available resources, connection points and so on. They all are hardcoded there, but for sure they might be variables read from somewhere (environment variables directly in Linux or from file with variables in Ansible).

If we check the defined serial port, we see a lot information there as well:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
$ telnet 0.0.0.0 2281
Trying 0.0.0.0...
Connected to 0.0.0.0.
Escape character is '^]'.
[    8.072503] Starting new kernel
[    1.096026] Running dosfsck on: /mnt/flash
fsck.fat 3.0.23 (2013-10-15)
/dev/sda1: 4 files, 80086/966346 clusters
Data in /mnt/flash/vEOS-lab.swi differs from previous boot image on /mnt/flash.
Saving new boot image to /mnt/flash...
Switching rootfs

Welcome to Arista Networks EOS 4.20.7M
method return sender=:1.0 -> dest=:1.2 reply_serial=2
New seat seat0.
RTNETLINK answers: No such process
[  OK  ] ConnMgr: Starting TimeAgent: [  OK  ]
!
! FURTHER OUTPUT IS OMITTED

 

After the Arista vEOS is booted, it’s available for configuration on CLI:

1
2
3
4
5
6
7
8
9
10
$ telnet 0.0.0.0 2281
Trying 0.0.0.0...
Connected to 0.0.0.0.
Escape character is '^]'.
!
! OUTPUT IS OMITTED
!
localhost login: admin
localhost>en
localhost#

 

As we have defined 3 ports, we see exactly these ports available for configuration (the first one is always management port, hence we connect it to br0):

1
2
3
4
5
localhost#show int des
Interface                      Status         Protocol           Description
Et1                            up             up                
Et2                            up             up                
Ma1                            up             up

 

Checking that ports are really connected to Linux bridges:

1
2
3
4
5
6
7
8
$ brctl show
bridge name bridge id       STP enabled interfaces
br0     8000.fe5400010200   no      vnet0
br1     8000.000c298ba1d3   no      ens34
                            vnet1
br2     8000.000c298ba1dd   no      ens35
                            vnet2
virbr0      8000.525400267f78   yes     virbr0-nic

 

The final step in verification is to configure IP address on management interface and check connectivity to host OS:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
localhost#conf t
localhost(config)#int management 1
localhost(config-if-Ma1)#ip add 192.168.1.81/24
localhost(config-if-Ma1)#
localhost#ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1) 72(100) bytes of data.
80 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.000 ms
80 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=0.000 ms
80 bytes from 192.168.1.1: icmp_seq=3 ttl=64 time=0.000 ms
80 bytes from 192.168.1.1: icmp_seq=4 ttl=64 time=0.000 ms
80 bytes from 192.168.1.1: icmp_seq=5 ttl=64 time=0.000 ms

--- 192.168.1.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4016ms
rtt min/avg/max/mdev = 0.000/0.000/0.000/0.000 ms, ipg/ewma 1004.062/0.000 ms

 

The last, but not least, we add the vEOS to hosts in Linux and Ansible, so that it’s available for further automation:

1
2
3
4
5
$ sudo cat /etc/hosts | grep 'vEOS1'
192.168.1.81 vEOS1

$ sudo cat /etc/ansible/hosts | grep 'vEOS1'
vEOS1

 

We are done with the onboarding of the VNF with Arista vEOS.

#1.2. Removing Arista vEOS

In case we need to free resources (like disk space), or we want to upgrade vEOS to another version, or for any other reason we want to delete it, we do the following:

  • Remove VM from KVM
  • Delete VNF’s disc from working folders
  • Clear hosts

The first two points are self-explanatory. The last one is not always necessary, as if you just want to rebuild VNF with new image, you could think about retaining the hostname and management IP.

Here how it looks like with commands. First of all we shoot down VNF with Aritsa vEOS and deleting it from KVM:

1
2
3
4
5
$ sudo virsh destroy vEOS1
Domain vEOS1 destroyed

$ sudo virsh undefine vEOS1
Domain vEOS1 has been undefined

 

Then we remove it from HDD:

1
2
3
4
$ sudo rm /var/lib/libvirt/images/vEOS1.qcow2

$ sudo ls -l /var/lib/libvirt/images/
total 0

 

And the last point is related to clearing hosts:

1
2
3
$ sudo cat /etc/hosts | grep 'vEOS1'

[aaa@sand7 temp]$ sudo cat /etc/ansible/hosts | grep 'vEOS1'

 

So, the lifecycle of the first VNF is finished. Let’s replicate, what we have done, to other VNFs.

#2.1. Onboarding Cisco IOS XR

It was very easy to download Cisco IOS XRv demo as it was available on the official website without any restrictions. Now there is only one image, which I can’t use, as I’m not using GNS3. But there is the mail you could try to contact or just look for the image somewhere in internet, it should be available somewhere.

The point with the creation of the “qcow2” image from “vmdk” I’m omitting, as it’s exactly the same as we’ve just seen. Moreover, I’ve already converted it earlier (link), so I’m using result of that job. Let’s briefly check current status of source and destination folders:

1
2
3
4
5
$ ls -l /tmp/ | grep '\.iso\|\.vmdk\|\.qcow2'
-rwxrw-rw-. 1 aaa  aaa  961740800 Feb 17  2018 cisco-iosxr-6.1.2.qcow2

$ sudo ls -l /var/lib/libvirt/images/
total 0

 

Now we copy that image to working folder for KVM:

1
2
3
4
5
$ sudo cp /tmp/cisco-iosxr-6.1.2.qcow2 /var/lib/libvirt/images/XR1.qcow2

$ sudo ls -l /var/lib/libvirt/images/
total 939200
-rwxr--r--. 1 root root 961740800 Sep  7 07:51 XR1.qcow2

 

By now all the preparation work is done, so we can onboard VNF with Cisco IOS XRv:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
$ sudo virt-install \
>   --name=XR1 \
>   --description "XR1 VM" \
>   --os-type=Linux \
>   --ram=3072 \
>   --vcpus=1 \
>   --boot hd,cdrom,menu=on \
>   --disk path=/var/lib/libvirt/images/XR1.qcow2,bus=ide,size=4 \
>   --import \
>   --graphics vnc \
>   --serial tcp,host=0.0.0.0:2251,mode=bind,protocol=telnet \
>   --network=bridge:br0,mac=52:54:00:04:02:00,model=virtio \
>   --network=bridge:br1,mac=52:54:00:04:02:01,model=virtio \
>   --network=bridge:br2,mac=52:54:00:04:02:02,model=virtio



WARNING  No operating system detected, VM performance may suffer. Specify an OS with --os-variant for optimal results.

Starting install...

(virt-viewer:5103): Gtk-WARNING **: Allocating size to VncDisplay 0x5632ce134250 without calling gtk_widget_get_preferred_width/height(). How does the code know the size to allocate?

 

Exactly in the same manner, as it was for Arista vEOS, graphical console is launched, though we see there only one message “Booting ‘IOS XRv’”.

You can see in the installation process above the same set of parameters for Cisco IOS XRv, as we have had for Arista vEOS, with the exception that XRv doesn’t require any “livecd” or anything else to boot, so configuraton instruction is easier.

On serial console we see much more information, which ends with login prompt, after it’s booted:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
$ telnet 0.0.0.0 2251
Trying 0.0.0.0...
Connected to 0.0.0.0.
Escape character is '^]'.
   Startup-code initializing, running on XRVR
   BSP date: Jun  3 2016 10:04:58
   Debugging enabled; debug_flag = 3, vendor_debug_flag = 0
   Page Address Extension mode (PAE) enabled with 36bit physical addressing.
   Reading back MTRR config from boot processor.
LAPIC id: 0000
LAPIC addr: 0x00000000fee00000
   IOSXRV: Initialized PIC
   IOSXRV: Initialized APIC LINT for CPU0
!
! OUTPUT IS OMITTED
!
ios con0/0/CPU0 is now available





Press RETURN to get started.

RP/0/0/CPU0:Sep  7 05:56:43.768 : sam_server[362]: %SECURITY-SAM-4-WARNING : Failed to initialize nvram digest




This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply third-party
authority to import, export, distribute or use encryption. Importers,
exporters, distributors and users are responsible for compliance with
U.S. and local country laws. By using this product you agree to comply
with applicable laws and regulations. If you are unable to comply with
U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be
found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.





IMPORTANT:  READ CAREFULLY
Welcome to the Demo Version of Cisco IOS XRv (the "Software").
The Software is subject to and governed by the terms and conditions
of the End User License Agreement and the Supplemental End User
License Agreement accompanying the product, made available at the
time of your order, or posted on the Cisco website at
www.cisco.com/go/terms (collectively, the "Agreement").
As set forth more fully in the Agreement, use of the Software is
strictly limited to internal use in a non-production environment
solely for demonstration and evaluation purposes.  Downloading,
installing, or using the Software constitutes acceptance of the
Agreement, and you are binding yourself and the business entity
that you represent to the Agreement.  If you do not agree to all
of the terms of the Agreement, then Cisco is unwilling to license
the Software to you and (a) you may not download, install or use the
Software, and (b) you may return the Software as more fully set forth
in the Agreement.


Please login with any configured user/password, or cisco/cisco


User Access Verification

Username: cisco
Password:


RP/0/0/CPU0:ios#

 

Let’s check the created interfaces on Cisco IOS XRv:

1
2
3
4
5
6
7
RP/0/0/CPU0:ios#show ip int br
Fri Sep  7 06:00:40.232 UTC

Interface                      IP-Address      Status          Protocol Vrf-Name
MgmtEth0/0/CPU0/0              unassigned      Shutdown        Down     default
GigabitEthernet0/0/0/0         unassigned      Shutdown        Down     default
GigabitEthernet0/0/0/1         unassigned      Shutdown        Down     default

 

And how they are connected to Linux bridges:

1
2
3
4
5
6
7
8
$ brctl show
bridge name bridge id       STP enabled interfaces
br0     8000.fe5400040200   no      vnet0
br1     8000.000c298ba1d3   no      ens34
                            vnet1
br2     8000.000c298ba1dd   no      ens35
                            vnet2
virbr0      8000.525400267f78   yes     virbr0-nic

 

Now we bring management interface UP and check connectivity to management bridge on host OS:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
RP/0/0/CPU0:ios(config)#show conf
Fri Sep  7 06:04:11.637 UTC
Building configuration...
!! IOS XR Configuration 6.1.2
vrf mgmt
 address-family ipv4 unicast
 !
!
interface MgmtEth0/0/CPU0/0
 vrf mgmt
 ipv4 address 192.168.1.51 255.255.255.0
 no shutdown
!
end

RP/0/0/CPU0:ios(config)#commit
Fri Sep  7 06:04:17.357 UTC
RP/0/0/CPU0:Sep  7 06:04:17.407 : ifmgr[228]: %PKT_INFRA-LINK-3-UPDOWN : Interface MgmtEth0/0/CPU0/0, changed state to Down
RP/0/0/CPU0:Sep  7 06:04:17.447 : ifmgr[228]: %PKT_INFRA-LINK-3-UPDOWN : Interface MgmtEth0/0/CPU0/0, changed state to Up
RP/0/0/CPU0:ios(config)#
RP/0/0/CPU0:ios#ping 192.168.1.1 vrf mgmt  
Fri Sep  7 06:06:00.080 UTC
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/20/89 ms

 

So, connectivityy is there and our VNF with Cisco IOS XRv is fully working. The last point is just make it available for all kinds of automation with Ansible by adding to hosts:

1
2
3
4
5
$ sudo cat /etc/hosts | grep 'XR1'
192.168.1.51 XR1

$ sudo cat /etc/ansible/hosts | grep 'XR1'
XR1

 

So, the VNF with Cisco IOS XRv is up and running and is ready to serve you.

#2.2. Removing Cisco IOS XR

The process of decommissioning is exactly the same as followed it for Arista vEOS:

  • Remove VM definition in KVM
  • Delete image with VNF
  • Clear hosts in Linux and Ansible

Here we go:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ sudo virsh destroy XR1
Domain XR1 destroyed

$ sudo virsh undefine XR1
Domain XR1 has been undefined

$ sudo rm /var/lib/libvirt/images/XR1.qcow2

$ sudo ls -l /var/lib/libvirt/images/
total 0

$ sudo cat /etc/hosts | grep 'XR1'

[aaa@sand7 temp]$ sudo cat /etc/ansible/hosts | grep 'XR1'

 

You might have seen that the process for Cisco IOS XRv and Arista vEOS is exactly the same, what already make me thinking about automation. What do you think? How about other VNFs?

#3.1. Onboarding Cumulus Linux VX

Here the situation is even more easy, as we don’t need to convert the image from one format to another, as we can download “qcow2” image from official website.

After the image is obtained, we put it the “/tmp/” folder and checking the status of the destination one:

1
2
3
4
5
6
$ ls -l /tmp/ | grep '\.iso\|\.vmdk\|\.qcow2'
-rwxrw-r--. 1 aaa  aaa  1157169152 Sep  7 17:49 cumulus-linux-3.6.2-vx-amd64.qcow2

$ sudo ls -l /var/lib/libvirt/images/
[sudo] password for aaa:
total 0

 

The second step is to move Cumulus VX image to working folder of KVM:

1
2
3
4
5
$ sudo cp /tmp/cumulus-linux-3.6.2-vx-amd64.qcow2 /var/lib/libvirt/images/VX1.qcow2

$ sudo ls -l /var/lib/libvirt/images/
total 1130048
-rwxr--r--. 1 root root 1157169152 Sep  7 17:53 VX1.qcow2

 

The preparation activities are done, so we can start the onboarding activity for Cumulus Networks VX:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
$ sudo virt-install \
>   --name=VX1 \
>   --description "VX1 VM" \
>   --os-type=Linux \
>   --ram=1024 \
>   --vcpus=1 \
>   --boot hd,cdrom,menu=on \
>   --disk path=/var/lib/libvirt/images/VX1.qcow2,bus=ide,size=4 \
>   --import \
>   --graphics vnc \
>   --serial tcp,host=0.0.0.0:2261,mode=bind,protocol=telnet \
>   --network=bridge:br0,mac=52:54:00:03:02:00,model=virtio \
>   --network=bridge:br1,mac=52:54:00:03:02:01,model=virtio \
>   --network=bridge:br2,mac=52:54:00:03:02:02,model=virtio



WARNING  No operating system detected, VM performance may suffer. Specify an OS with --os-variant for optimal results.

Starting install...

(virt-viewer:15664): Gtk-WARNING **: Allocating size to VncDisplay 0x565390fe0250 without calling gtk_widget_get_preferred_width/height(). How does the code know the size to allocate?

 

Surprise, surprise! There is no diffrences are all in provisioning of VNF with Cumulus VX comparing to Cisco IOS XRv. So the set of parameters is exactly the same, and we even provide lower amount of resources (just 1 GB RAM instead of 3 GB RAM).

The major difference comparing to Cisco IOS XRv is that VNF with Cumulus VX has a working graphical terminal, just like Arista vEOS has. But, we also have an working serial CLI, so it means that initial provisioning could be done in the similar way as well, if we want:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
$ telnet 0.0.0.0 2261
Trying 0.0.0.0...
Connected to 0.0.0.0.
Escape character is '^]'.

Loading Linux 4.1.0-cl-7-amd64 ...
Loading initial ramdisk ...
Loading, please wait...
Scanning for Btrfs filesystems

Welcome to Cumulus Linux!
!
! OUTPUT IS OMITTED
!
[  OK  ] Started Cumulus Linux Multi-Chassis LACP Bonding Daemon.

Debian GNU/Linux 8 cumulus ttyS0

cumulus login: cumulus
Password:
Linux cumulus 4.1.0-cl-7-amd64 #1 SMP Debian 4.1.33-1+cl3u14 (2018-07-05) x86_64

Welcome to Cumulus VX (TM)

Cumulus VX (TM) is a community supported virtual appliance designed for
experiencing, testing and prototyping Cumulus Networks' latest technology.
For any questions or technical support, visit our community site at:
http://community.cumulusnetworks.com

The registered trademark Linux (R) is used pursuant to a sublicense from LMI,
the exclusive licensee of Linus Torvalds, owner of the mark on a world-wide
basis.
cumulus@cumulus:~$

 

The next step is to check, which interfaces are connected where. On Cumulus Linux we see interesting output, like the interfaces are visible in one output, but not in another one:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
cumulus@cumulus:~$ net show interface
State  Name  Spd  MTU    Mode      LLDP  Summary
-----  ----  ---  -----  --------  ----  ---------------
UP     lo    N/A  65536  Loopback        IP: 127.0.0.1/8
       lo                                IP: ::1/128
UP     eth0  1G   1500   Mgmt

cumulus@cumulus:~$ ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 52:54:00:03:02:00 brd ff:ff:ff:ff:ff:ff
3: swp1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 52:54:00:03:02:01 brd ff:ff:ff:ff:ff:ff
4: swp2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 52:54:00:03:02:02 brd ff:ff:ff:ff:ff:ff

 

In Linux bridges on host OS everything looks OK so far:

1
2
3
4
5
6
7
8
$ brctl show
bridge name bridge id       STP enabled interfaces
br0     8000.fe5400030200   no      vnet0
br1     8000.000c298ba1d3   no      ens34
                            vnet1
br2     8000.000c298ba1dd   no      ens35
                            vnet2
virbr0      8000.525400267f78   yes     virbr0-nic

 

So, I will check connectivity on management and one data plane interface, to make sure both are working:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
cumulus@cumulus:~$ net commit
--- /etc/network/interfaces 2018-07-15 06:53:10.000000000 +0000
+++ /run/nclu/ifupdown2/interfaces.tmp  2018-09-07 16:08:56.781035636 +0000
@@ -2,11 +2,22 @@
 # and how to activate them. For more information, see interfaces(5).
 
 source /etc/network/interfaces.d/*.intf
 
 # The loopback network interface
 auto lo
 iface lo inet loopback
 
 # The primary network interface
 auto eth0
-iface eth0 inet dhcp
+iface eth0
+    address 192.168.1.61/24
+    vrf mgmt
+
+auto swp1
+iface swp1
+    address 10.0.0.61/24
+
+auto mgmt
+iface mgmt
+    address 127.0.0.1/8
+    vrf-table auto



net add/del commands since the last "net commit"
================================================

User     Timestamp                   Command
-------  --------------------------  -------------------------------------------------
cumulus  2018-09-07 16:07:40.566185  net add vrf mgmt
cumulus  2018-09-07 16:08:29.285741  net add interface eth0 ip address 192.168.1.61/24
cumulus  2018-09-07 16:08:43.031818  net add interface swp1 ip address 10.0.0.61/24












cumulus@cumulus:~$ ping -I mgmt 192.168.1.1
ping: Warning: source address might be selected on device other than mgmt.
PING 192.168.1.1 (192.168.1.1) from 192.168.1.61 mgmt: 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=1.88 ms
^C
--- 192.168.1.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.880/1.880/1.880/0.000 ms
cumulus@cumulus:~$ ping 10.0.0.254
PING 10.0.0.254 (10.0.0.254) 56(84) bytes of data.
64 bytes from 10.0.0.254: icmp_seq=1 ttl=64 time=3.74 ms
64 bytes from 10.0.0.254: icmp_seq=2 ttl=64 time=0.992 ms
^C
--- 10.0.0.254 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1003ms
rtt min/avg/max/mdev = 0.992/2.366/3.740/1.374 ms

 

Both management and data plane interfaces are up and running. And now we see this configured interface in “netd” managed output:

1
2
3
4
5
6
7
8
9
cumulus@cumulus:~$ net show interface
State  Name  Spd  MTU    Mode          LLDP  Summary
-----  ----  ---  -----  ------------  ----  -------------------
UP     lo    N/A  65536  Loopback            IP: 127.0.0.1/8
       lo                                    IP: ::1/128
UP     eth0  1G   1500   Mgmt                Master: mgmt(UP)
       eth0                                  IP: 192.168.1.61/24
UP     swp1  1G   1500   Interface/L3        IP: 10.0.0.61/24
UP     mgmt  N/A  65536  Interface/L3        IP: 127.0.0.1/8

 

The last point, we update hosts’ tables:

1
2
3
4
5
$ sudo cat /etc/hosts | grep 'VX1'
192.168.1.61 VX1

$ sudo cat /etc/ansible/hosts | grep 'VX1'
VX1

 

Onboarding for VNF with Cumulus Networks VX is done.

#3.2. Removing Cumulus Linux VX

The process of deconstructing of Cumulus Linux VX on KVM is exactly the same as already done 2 times:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ sudo virsh destroy VX1
Domain VX1 destroyed

$ sudo virsh undefine VX1
Domain VX1 has been undefined

$ sudo rm /var/lib/libvirt/images/VX1.qcow2

$ sudo ls -l /var/lib/libvirt/images/
total 0

$ sudo cat /etc/hosts | grep 'VX1'

$ sudo cat /etc/ansible/hosts | grep 'VX1'

 

Bye, Cumulus VX, see you by next onboarding.

#4.1. Onboarding Nokia (Alcatel-Lucent) VSR

The next in the queue for the onboarding is Nokia (Alcatel-Lucent) VSR. The reason, there are two reasons, why it stands here:

  • We are using just alphabetical order
  • It has a bit more complicated structure of installation instruction

Just to highlight the last point, we need to define additional information about VSR structure and license upon creation of VNF, so we need to somehow to note this information. After some time spent on searching how to do it, I’ve found the solution. In a minute I will show you how it works together with “virt-install”. But for now, we are traditionally checking files:

1
2
3
4
5
$ ls -l /tmp/ | grep '\.qcow2\|\.vmdk\|\.iso'
-rwxrw-r--. 1 aaa  aaa  287309824 Sep 10 07:40 nokia-sros-16.0R1.qcow2

$ sudo ls -l /var/lib/libvirt/images/
total 0

 

“Copy as usual”:

1
2
3
4
5
$ sudo cp /tmp/nokia-sros-16.0R1.qcow2 /var/lib/libvirt/images/SR1.qcow2

$ sudo ls -l /var/lib/libvirt/images/
total 280576
-rwxr--r--. 1 root root 287309824 Sep 10 07:45 SR1.qcow2

 

And now we are reaching the key activity, where we onboard the VNF with Nokia VSR on KVM:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
$ sudo virt-install \
>   --name=SR1 \
>   --description "SR1 VM" \
>   --os-type=Linux \
>   --sysinfo type='smbios',system_product='TIMOS:address=192.168.1.101/24@active static-route=192.168.0.0/16@192.168.1.1 license-file=ftp://nokia:nokianokia@192.168.1.1/sros16.lic slot=A chassis=SR-1 card=iom-1 mda/1=me6-100gb-qsfp28'\
>   --ram=4096 \
>   --vcpus=2 \
>   --boot hd \
>   --disk path=/var/lib/libvirt/images/SR1.qcow2,bus=virtio,size=4 \
>   --import \
>   --graphics vnc \
>   --serial tcp,host=0.0.0.0:2271,mode=bind,protocol=telnet \
>   --network=bridge:br0,mac=52:54:00:02:02:00,model=virtio \
>   --network=bridge:br1,mac=52:54:00:02:02:01,model=virtio \
>   --network=bridge:br2,mac=52:54:00:02:02:02,model=virtio



WARNING  No operating system detected, VM performance may suffer. Specify an OS with --os-variant for optimal results.

Starting install...

(virt-viewer:4891): Gtk-WARNING **: Allocating size to VncDisplay 0x56061b056250 without calling gtk_widget_get_preferred_width/height(). How does the code know the size to allocate?

 

Comparing to three previous installations, we have new key “–sysinfo”, where this additional information about Nokia (Alcatel-Lucent) VSR structure is stored. The rest of the actions is exactly the same. So we check the serial console:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
$ telnet 0.0.0.0 2271
Trying 0.0.0.0...
Connected to 0.0.0.0.
Escape character is '^]'.
!
! OUTPUT IS OMITTED
!
TiMOS-B-16.0.R1 both/x86_64 Nokia 7750 SR Copyright (c) 2000-2018 Nokia.
All rights reserved. All use subject to applicable license agreements.
Built on Thu May 31 16:23:56 PDT 2018 by builder in /builds/160B/R1/panos/main


Login: admin
Password:

 SR OS Software
 Copyright (c) Nokia 2018.  All Rights Reserved.
 
 Trademarks
 
 Nokia and the Nokia logo are registered trademarks of Nokia. All other
 trademarks are the property of their respective owners.
 
 IMPORTANT: READ CAREFULLY
 
 The SR OS Software (the "Software") is proprietary to Nokia and is subject
 to and governed by the terms and conditions of the End User License
 Agreement accompanying the product, made available at the time of your order,
 or posted on the Nokia website (collectively, the "EULA").  As set forth
 more fully in the EULA, use of the Software is strictly limited to your
 internal use.  Downloading, installing, or using the Software constitutes
 acceptance of the EULA and you are binding yourself and the business entity
 that you represent to the EULA.  If you do not agree to all of the terms of
 the EULA, then Nokia is unwilling to license the Software to you and (a) you
 may not download, install or use the Software, and (b) you may return the
 Software as more fully set forth in the EULA.
 
 This product contains cryptographic features and is subject to United States
 and local country laws governing import, export, transfer and use. Delivery
 of Nokia cryptographic products does not imply third-party authority to
 import, export, distribute or use encryption.
 
 If you require further assistance please contact us by sending an email
 to support@nokia.com.

A:vSIM#

 

There is an official documentation regarding VSR onboarding available, but I’m not able to reach is I don’t have corresponding rights. So I will explain one key topic briefly myself. In the setup above the 100G MDA is used, but to connect it somewhere, the breakout cables are used. This configuration was already shown during OpenConfig journey (link) without explanation. On the other hand, we define ports in VM in the time of creation, so the world outside VNF doesn’t know anything about breakout ports and is looking good already now.

1
2
3
4
5
6
7
8
$ brctl show
bridge name bridge id       STP enabled interfaces
br0     8000.fe5400020200   no      vnet0
br1     8000.000c298ba1d3   no      ens34
                            vnet1
br2     8000.000c298ba1dd   no      ens35
                            vnet2
virbr0      8000.525400267f78   yes     virbr0-nic

 

So, only proper configuration on VSR is needed. By the time of the VNF onboarding we have defined 3 ports, what translates in 1 management in BOF and 2 data planes: 1st breakout in first two ports on MDA. The configuration below shows how the Nokia VSR ports should be configured (in case we use SR-1 with breakout ports) in order to have connectivity to Linux bridges:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
*(gl)[]
A:admin@vSIM# compare
+    configure {
+       card 1 {
+           card-type iom-1
+           mda 1 {
+               mda-type me6-100gb-qsfp28
+           }
+       }
+       port 1/1/c1 {
+           admin-state enable
+           connector {
+               breakout c4-10g
+           }
+       }
+       port 1/1/c1/1 {
+           admin-state enable
+           ethernet {
+               mode network
+           }
+       }
+       port 1/1/c2 {
+           admin-state enable
+           connector {
+               breakout c4-10g
+           }
+       }
+       port 1/1/c2/1 {
+           admin-state enable
+           ethernet {
+               mode network
+           }
+       }
+       router "Base" {
+           interface "p1" {
+               admin-state enable
+               port 1/1/c1/1
+               ipv4 {
+                   primary {
+                       address 10.0.0.71
+                       prefix-length 24
+                   }
+               }
+           }
+           interface "p2" {
+               admin-state enable
+               port 1/1/c2/1
+               ipv4 {
+                   primary {
+                       address 10.0.1.71
+                       prefix-length 24
+                   }
+               }
+           }
+       }
    }

*(gl)[]
A:admin@vSIM# commit

 

Let’s check that connectivity between Nokia (Alcatel-Lucent) VSR and Linux bridges are working in reality:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
(gl)[]
A:admin@vSIM# //ping 10.0.0.254
INFO: CLI #2051: Switching to the classic CLI engine
A:vSIM# /ping 10.0.0.254
PING 10.0.0.254 56 data bytes
64 bytes from 10.0.0.254: icmp_seq=1 ttl=64 time=12.4ms.
64 bytes from 10.0.0.254: icmp_seq=2 ttl=64 time=2.47ms.
^C
ping aborted by user

---- 10.0.0.254 PING Statistics ----
2 packets transmitted, 2 packets received, 0.00% packet loss
round-trip min = 2.47ms, avg = 7.43ms, max = 12.4ms, stddev = 0.000ms
INFO: CLI #2052: Switching to the MD-CLI engine


(gl)[]
A:admin@vSIM# //ping 10.0.1.254
INFO: CLI #2051: Switching to the classic CLI engine
A:vSIM# /ping 10.0.1.254
PING 10.0.1.254 56 data bytes
64 bytes from 10.0.1.254: icmp_seq=1 ttl=64 time=6.99ms.
64 bytes from 10.0.1.254: icmp_seq=2 ttl=64 time=1.35ms.
^C
ping aborted by user

---- 10.0.1.254 PING Statistics ----
2 packets transmitted, 2 packets received, 0.00% packet loss
round-trip min = 1.35ms, avg = 4.17ms, max = 6.99ms, stddev = 0.000ms
INFO: CLI #2052: Switching to the MD-CLI engine

 

Wait, we have forgotten to test management interfaces reachability. Ok, there is no problem, we check it together with hosts table update:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
$ sudo cat /etc/hosts | grep 'SR1'
192.168.1.101 SR1

$ sudo cat /etc/ansible/hosts | grep 'SR1'
SR1

$ ping SR1
PING SR1 (192.168.1.101) 56(84) bytes of data.
64 bytes from SR1 (192.168.1.101): icmp_seq=1 ttl=64 time=6.88 ms
64 bytes from SR1 (192.168.1.101): icmp_seq=2 ttl=64 time=0.375 ms
64 bytes from SR1 (192.168.1.101): icmp_seq=3 ttl=64 time=0.946 ms
64 bytes from SR1 (192.168.1.101): icmp_seq=4 ttl=64 time=10.6 ms
64 bytes from SR1 (192.168.1.101): icmp_seq=5 ttl=64 time=1.29 ms
64 bytes from SR1 (192.168.1.101): icmp_seq=6 ttl=64 time=24.5 ms
64 bytes from SR1 (192.168.1.101): icmp_seq=7 ttl=64 time=2.83 ms
^C
--- SR1 ping statistics ---
7 packets transmitted, 7 received, 0% packet loss, time 6028ms
rtt min/avg/max/mdev = 0.375/6.792/24.559/8.034 ms

 

So the onboarding of the VNF with Nokia (Alcatel-Lucent) VSR is done.

#4.1. Removing Nokia (Alcatel-Lucent) VSR

The process of the removing of the VNF is already very well known, so I won’t even explain you the steps and will just proceed with some commands, how you do it:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ sudo virsh destroy SR1
Domain SR1 destroyed

$ sudo virsh undefine SR1
Domain SR1 has been undefined

$ sudo rm /var/lib/libvirt/images/SR1.qcow2

$ sudo ls -l /var/lib/libvirt/images/
total 0

$ sudo cat /etc/hosts | grep 'SR1'

$ sudo cat /etc/ansible/hosts | grep 'SR1'

 

This process is more than straightforward, as you can see. The VNF with Nokia (Alcatel-Lucent) VSR is successfully undeployed.

As a summary, you can download all 4 “virt-install” examples: 134_create_KVM_Cisco_IOS_XRv134_create_KVM_Cumulus_Linux134_create_KVM_Nokia_VSR134_create_KVM_Arista_vEOS

Consideration on automated deployment with Ansible

All these manual operations in terms of VNF deployment or undeployment could be automated using Ansible. Earlier we have explained how Ansible could be used as VNF-M, so that you can easily template all mandatory parameters and rollout the necessary amount of VNFs. All you need is to create proper jinja2 template per VNF type and hold it consistent with variables. I won’t create new Ansible playbook in the current article, as it doesn’t bring any additional value.

Stick to that article for an example of fully automated deployment.

Lessons learned

Devil is hidden into details. You might have seen that all installation is similar. But here “similar” is the key word, it isn’t “equal”. As a result, I need to test different sets of the parameters for installation, as I haven’t found any single documents that explains that process for KVM. So, trying and redoing is the only working approach to build any system.

The second lessons learned is that sometimes you need to take much more efforts in order to get things done. For instance, I haven’t managed to get Cisco Nexus 9000v working on KVM, though I have tried so hard that I have broken my lab and had to rebuild it from scratches. The good thing is that I have recorded how to do that.

Conclusion

Making the army of clones is always good when we start speaking about CI/CD DevOps approach. For instance, if you want to make any test on the network setup, the only working approach is to rebuild the lab, destroy it, repeat it. So it’s quite often it’s necessary to use that kind of templating. Coupling this lab and the VNF-M with Ansible gives you that tools.. Take care and good bye!

Support us





P.S.

If you have further questions or you need help with your networks, I’m happy to assist you, just send me message (http://karneliuk.com/contact/). Also don’t forget to share the article on your social media, if you like it.

BR,

Anton Karneliuk