Hello my friend,
Some time ago we have discussed how Ansible could be used as VNF manager for onboarding and deleting the VMs in Linux. I have got some feedbacks, that not everything is easy or could be reproduced, so I decided to find the easier way, how to help you with the creation of virtual labs to train your networking and programming skills (assuming you are using Ansible).
2
3
4
5
retrieval system, or transmitted in any form or by any
means, electronic, mechanical or photocopying, recording,
or otherwise, for commercial purposes without the
prior permission of the author.
Brief description
The whole OpenConfig series, I have written recently took me a lot of time and efforts. In total part 1, part 2 and part 3 uses 167 A4 pages in Microsoft Word, just like a small book. That’s why I’ll try to write something shorter, but as useful at least as OpenConfig articles.
The second important point for writing this article for me was the fact that I’m using different hypervisors for different VMs. On the one hand, there is a very easy explanation for that, what is the balancing of the used resources versus flexibility. Nevertheless, it prevents me of deploying full automation for lab lifecycle starting from creation till making the resource free again. As I generally find Linux very easy in terms of automation and as it has amazing CLI, I decided to use it as main platform for all kinds of virtualization. In order not to lose performance of VMs, I’m looking in direction of pure KVM without QEMU.
So this blogpost is dedicated to explanation, how different virtual network functions could be deployed on KVM.
What we are going to test?
We are focusing on deployment of VMs with network functions on KVM hypervisor with Linux CentOS 7 as host OS. So I want to find the easiest way to do it via CLI so that I can automate it with Ansible either using specific module for that or generic module for Linux commands.
We’ll deploy 4 different VNFs, details are coming in the next chapter.
Software version
The following infrastructure is used in my lab:
- CentOS 7 with python 2.7.
- Ansible 2.6.0
- Arista EOS 4.20.6F [guest VNF]
- Nokia SR OS 16.0.R1 [guest VNF]
- Cisco IOS XR 6.1.2 [guest VNF]
- Cumulus Linux 3.6.2 [guest VNF]
See the previous article to get details how to build the lab (https://goo.gl/hKhtvX)
Topology
There is no particular topology for this lab, as we are speaking about VNF onboarding. The generic image, how any VNF should be connected you might find on the image below:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
|ens33 |ens35 |ens35
|ip: 192.168.44.17/24 | |
+---------------------------------------------------------------------+
| | | |
| | | |
| +------------------+ +--------+---------+ +---------+--------+ |
| | Management | | Data 1 | | Data 2 | |
| | br0 | | br1 | | br2 | |
| |ip: 192.168.1.1/24| |ip: 10.0.0.254/24 | |ip: 10.0.1.254/24 | |
| +--------+---------+ +--------+---------+ +---------+--------+ |
| | | | |
| |eth0 |eth1 |eth2 |
| | | | |
| +--------+----------------------+-----------------------+--------+ |
| | | |
| | New VNF with XX | |
| | | |
| +----------------------------------------------------------------+ |
| |
| Host OS (CentOS) w/ KVM |
| |
+---------------------------------------------------------------------+
The topology above also provides the overview of the logical connectivity, as there is nothing particular, just IP connectivity to Linux bridges.
There is no default configuration files to any of VNFs, as there is no VNFs yet.:
General algorithm for VNF creation and prerequisites
We’ll start with the second point, hence, prerequisites.:
- You have KVM and all related components installed.
- All VNF images are in “qcow2”
- All the source files with VNFs are located in “/tmp/” They could be anywhere else, but in my examples I’ll show such case.
To convert file from “vmdk” to “qcow2” use specific tool “qemu-img convert -f vmdk -O qcow2 source.vmdk target.qcow2”
If both prerequisites are fulfilled, then the rest is quite easy. We do the following steps:
- Copy VNF image in “qcow2” format to folder with folder dedicated to VNF images, which is “/var/lib/libvirt/images/”.
- Install VNF using “virt-install” tool according to the physical topology provided above.
- Update hosts tables for Linux and Ansible
As you see, the set of activities we need to do is quite straightforward, so let’s start with the execution.
#1.1. Onboarding Arista vEOS
You can download the Arista vEOS files from the official website. There also you find official guidelines how to deploy it as VM with QEMU example. Earlier I have explained, how you can install it on VMware Player, so now we streamline things with KVM.
So, we briefly check source folder:
2
3
-rwxrw-rw-. 1 aaa aaa 5242880 May 12 12:04 Aboot-veos-8.0.0.iso
-rwxrw-rw-. 1 aaa aaa 657588224 Jul 18 22:44 vEOS-lab-4.20.7M.vmdk
And destination folder:
2
total 0
Now we start deployment of or VNF with Arista vEOS. First of all, we convert “vmdk” to “qcow2”:
2
3
4
5
6
7
$ rm vEOS-lab-4.20.7M.vmdk
$ ls -l /tmp/ | grep '\.iso\|\.vmdk\|\.qcow2'
-rwxrw-rw-. 1 aaa aaa 5242880 May 12 12:04 Aboot-veos-8.0.0.iso
-rw-r--r--. 1 aaa aaa 657391616 Sep 6 18:45 vEOS-lab.qcow2
Next, we copy disc of VNF, what is “qcow2” file to standard folder with VNFs:
2
3
4
5
$ sudo ls -l /var/lib/libvirt/images/
total 641984
-rw-r--r--. 1 root root 657391616 Sep 6 18:50 vEOS-lab.qcow2
So we have reached the core activity that is installation of VNF with Arista vEOS:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
> --name=vEOS1 \
> --description "vEOS VM" \
> --os-type=Linux \
> --ram=2048 \
> --vcpus=2 \
> --boot cdrom,hd,menu=on \
> --cdrom /tmp/Aboot-veos-8.0.0.iso \
> --livecd \
> --disk path=/var/lib/libvirt/images/vEOS1.qcow2,bus=ide,size=4 \
> --graphics vnc \
> --serial tcp,host=0.0.0.0:2281,mode=bind,protocol=telnet \
> --network=bridge:br0,mac=52:54:00:01:02:00,model=virtio \
> --network=bridge:br1,mac=52:54:00:01:02:01,model=virtio \
> --network=bridge:br2,mac=52:54:00:01:02:02,model=virtio
WARNING No operating system detected, VM performance may suffer. Specify an OS with --os-variant for optimal results.
Starting install…
At this time graphical VNC console starts, so you can see the process of boot there.
As it’s explained in the official documentation from Arista, the file called “Aboot-veos” should be installed in CD-ROM of VM as it’s used during boot. In order to sustain this after reboot, we instruct KVM to treat it as “livecd”, as by default the CD will be removed after the first reboot.
In the command above we define the most critical parameters of the VM, as available resources, connection points and so on. They all are hardcoded there, but for sure they might be variables read from somewhere (environment variables directly in Linux or from file with variables in Ansible).
If we check the defined serial port, we see a lot information there as well:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
Trying 0.0.0.0...
Connected to 0.0.0.0.
Escape character is '^]'.
[ 8.072503] Starting new kernel
[ 1.096026] Running dosfsck on: /mnt/flash
fsck.fat 3.0.23 (2013-10-15)
/dev/sda1: 4 files, 80086/966346 clusters
Data in /mnt/flash/vEOS-lab.swi differs from previous boot image on /mnt/flash.
Saving new boot image to /mnt/flash...
Switching rootfs
Welcome to Arista Networks EOS 4.20.7M
method return sender=:1.0 -> dest=:1.2 reply_serial=2
New seat seat0.
RTNETLINK answers: No such process
[ OK ] ConnMgr: Starting TimeAgent: [ OK ]
!
! FURTHER OUTPUT IS OMITTED
After the Arista vEOS is booted, it’s available for configuration on CLI:
2
3
4
5
6
7
8
9
10
Trying 0.0.0.0...
Connected to 0.0.0.0.
Escape character is '^]'.
!
! OUTPUT IS OMITTED
!
localhost login: admin
localhost>en
localhost#
As we have defined 3 ports, we see exactly these ports available for configuration (the first one is always management port, hence we connect it to br0):
2
3
4
5
Interface Status Protocol Description
Et1 up up
Et2 up up
Ma1 up up
Checking that ports are really connected to Linux bridges:
2
3
4
5
6
7
8
bridge name bridge id STP enabled interfaces
br0 8000.fe5400010200 no vnet0
br1 8000.000c298ba1d3 no ens34
vnet1
br2 8000.000c298ba1dd no ens35
vnet2
virbr0 8000.525400267f78 yes virbr0-nic
The final step in verification is to configure IP address on management interface and check connectivity to host OS:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
localhost(config)#int management 1
localhost(config-if-Ma1)#ip add 192.168.1.81/24
localhost(config-if-Ma1)#
localhost#ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1) 72(100) bytes of data.
80 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.000 ms
80 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=0.000 ms
80 bytes from 192.168.1.1: icmp_seq=3 ttl=64 time=0.000 ms
80 bytes from 192.168.1.1: icmp_seq=4 ttl=64 time=0.000 ms
80 bytes from 192.168.1.1: icmp_seq=5 ttl=64 time=0.000 ms
--- 192.168.1.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4016ms
rtt min/avg/max/mdev = 0.000/0.000/0.000/0.000 ms, ipg/ewma 1004.062/0.000 ms
The last, but not least, we add the vEOS to hosts in Linux and Ansible, so that it’s available for further automation:
2
3
4
192.168.1.81 vEOS1
$ sudo cat /etc/ansible/hosts | grep 'vEOS1'
vEOS1
We are done with the onboarding of the VNF with Arista vEOS.
#1.2. Removing Arista vEOS
In case we need to free resources (like disk space), or we want to upgrade vEOS to another version, or for any other reason we want to delete it, we do the following:
- Remove VM from KVM
- Delete VNF’s disc from working folders
- Clear hosts
The first two points are self-explanatory. The last one is not always necessary, as if you just want to rebuild VNF with new image, you could think about retaining the hostname and management IP.
Here how it looks like with commands. First of all we shoot down VNF with Aritsa vEOS and deleting it from KVM:
2
3
4
5
Domain vEOS1 destroyed
$ sudo virsh undefine vEOS1
Domain vEOS1 has been undefined
Then we remove it from HDD:
2
3
4
$ sudo ls -l /var/lib/libvirt/images/
total 0
And the last point is related to clearing hosts:
2
$ sudo cat /etc/ansible/hosts | grep 'vEOS1'
So, the lifecycle of the first VNF is finished. Let’s replicate, what we have done, to other VNFs.
#2.1. Onboarding Cisco IOS XR
It was very easy to download Cisco IOS XRv demo as it was available on the official website without any restrictions. Now there is only one image, which I can’t use, as I’m not using GNS3. But there is the mail you could try to contact or just look for the image somewhere in internet, it should be available somewhere.
The point with the creation of the “qcow2” image from “vmdk” I’m omitting, as it’s exactly the same as we’ve just seen. Moreover, I’ve already converted it earlier (link), so I’m using result of that job. Let’s briefly check current status of source and destination folders:
2
3
4
5
6
-rwxrw-rw-. 1 aaa aaa 961740800 Feb 17 2018 cisco-iosxr-6.1.2.qcow2
$ sudo ls -l /var/lib/libvirt/images/
total 0
Now we copy that image to working folder for KVM:
2
3
4
5
6
$ sudo ls -l /var/lib/libvirt/images/
total 939200
-rwxr--r--. 1 root root 961740800 Sep 7 07:51 XR1.qcow2
By now all the preparation work is done, so we can onboard VNF with Cisco IOS XRv:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
> --name=XR1 \
> --description "XR1 VM" \
> --os-type=Linux \
> --ram=3072 \
> --vcpus=1 \
> --boot hd,cdrom,menu=on \
> --disk path=/var/lib/libvirt/images/XR1.qcow2,bus=ide,size=4 \
> --import \
> --graphics vnc \
> --serial tcp,host=0.0.0.0:2251,mode=bind,protocol=telnet \
> --network=bridge:br0,mac=52:54:00:04:02:00,model=virtio \
> --network=bridge:br1,mac=52:54:00:04:02:01,model=virtio \
> --network=bridge:br2,mac=52:54:00:04:02:02,model=virtio
WARNING No operating system detected, VM performance may suffer. Specify an OS with --os-variant for optimal results.
Starting install…
Exactly in the same manner, as it was for Arista vEOS, graphical console is launched, though we see there only one message “Booting ‘IOS XRv’”.
You can see in the installation process above the same set of parameters for Cisco IOS XRv, as we have had for Arista vEOS, with the exception that XRv doesn’t require any “livecd” or anything else to boot, so configuraton instruction is easier.
On serial console we see much more information, which ends with login prompt, after it’s booted:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
Trying 0.0.0.0...
Connected to 0.0.0.0.
Escape character is '^]'.
Startup-code initializing, running on XRVR
BSP date: Jun 3 2016 10:04:58
Debugging enabled; debug_flag = 3, vendor_debug_flag = 0
Page Address Extension mode (PAE) enabled with 36bit physical addressing.
Reading back MTRR config from boot processor.
LAPIC id: 0000
LAPIC addr: 0x00000000fee00000
IOSXRV: Initialized PIC
IOSXRV: Initialized APIC LINT for CPU0
!
! OUTPUT IS OMITTED
!
ios con0/0/CPU0 is now available
Press RETURN to get started.
RP/0/0/CPU0:Sep 7 05:56:43.768 : sam_server[362]: %SECURITY-SAM-4-WARNING : Failed to initialize nvram digest
This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply third-party
authority to import, export, distribute or use encryption. Importers,
exporters, distributors and users are responsible for compliance with
U.S. and local country laws. By using this product you agree to comply
with applicable laws and regulations. If you are unable to comply with
U.S. and local laws, return this product immediately.
A summary of U.S. laws governing Cisco cryptographic products may be
found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html
If you require further assistance please contact us by sending email to export@cisco.com.
IMPORTANT: READ CAREFULLY
Welcome to the Demo Version of Cisco IOS XRv (the "Software").
The Software is subject to and governed by the terms and conditions
of the End User License Agreement and the Supplemental End User
License Agreement accompanying the product, made available at the
time of your order, or posted on the Cisco website at
www.cisco.com/go/terms (collectively, the "Agreement").
As set forth more fully in the Agreement, use of the Software is
strictly limited to internal use in a non-production environment
solely for demonstration and evaluation purposes. Downloading,
installing, or using the Software constitutes acceptance of the
Agreement, and you are binding yourself and the business entity
that you represent to the Agreement. If you do not agree to all
of the terms of the Agreement, then Cisco is unwilling to license
the Software to you and (a) you may not download, install or use the
Software, and (b) you may return the Software as more fully set forth
in the Agreement.
Please login with any configured user/password, or cisco/cisco
User Access Verification
Username: cisco
Password:
RP/0/0/CPU0:ios#
Let’s check the created interfaces on Cisco IOS XRv:
2
3
4
5
6
Fri Sep 7 06:00:40.232 UTC
Interface IP-Address Status Protocol Vrf-Name
MgmtEth0/0/CPU0/0 unassigned Shutdown Down default
GigabitEthernet0/0/0/0 unassigned Shutdown Down default
GigabitEthernet0/0/0/1 unassigned Shutdown Down default
And how they are connected to Linux bridges:
2
3
4
5
6
7
8
bridge name bridge id STP enabled interfaces
br0 8000.fe5400040200 no vnet0
br1 8000.000c298ba1d3 no ens34
vnet1
br2 8000.000c298ba1dd no ens35
vnet2
virbr0 8000.525400267f78 yes virbr0-nic
Now we bring management interface UP and check connectivity to management bridge on host OS:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
Fri Sep 7 06:04:11.637 UTC
Building configuration...
!! IOS XR Configuration 6.1.2
vrf mgmt
address-family ipv4 unicast
!
!
interface MgmtEth0/0/CPU0/0
vrf mgmt
ipv4 address 192.168.1.51 255.255.255.0
no shutdown
!
end
RP/0/0/CPU0:ios(config)#commit
Fri Sep 7 06:04:17.357 UTC
RP/0/0/CPU0:Sep 7 06:04:17.407 : ifmgr[228]: %PKT_INFRA-LINK-3-UPDOWN : Interface MgmtEth0/0/CPU0/0, changed state to Down
RP/0/0/CPU0:Sep 7 06:04:17.447 : ifmgr[228]: %PKT_INFRA-LINK-3-UPDOWN : Interface MgmtEth0/0/CPU0/0, changed state to Up
RP/0/0/CPU0:ios(config)#
RP/0/0/CPU0:ios#ping 192.168.1.1 vrf mgmt
Fri Sep 7 06:06:00.080 UTC
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/20/89 ms
So, connectivityy is there and our VNF with Cisco IOS XRv is fully working. The last point is just make it available for all kinds of automation with Ansible by adding to hosts:
2
3
4
5
6
192.168.1.51 XR1
$ sudo cat /etc/ansible/hosts | grep 'XR1'
XR1
So, the VNF with Cisco IOS XRv is up and running and is ready to serve you.
#2.2. Removing Cisco IOS XR
The process of decommissioning is exactly the same as followed it for Arista vEOS:
- Remove VM definition in KVM
- Delete image with VNF
- Clear hosts in Linux and Ansible
Here we go:
2
3
4
5
6
7
8
9
10
11
12
13
14
Domain XR1 destroyed
$ sudo virsh undefine XR1
Domain XR1 has been undefined
$ sudo rm /var/lib/libvirt/images/XR1.qcow2
$ sudo ls -l /var/lib/libvirt/images/
total 0
$ sudo cat /etc/hosts | grep 'XR1'
$ sudo cat /etc/ansible/hosts | grep 'XR1'
You might have seen that the process for Cisco IOS XRv and Arista vEOS is exactly the same, what already make me thinking about automation. What do you think? How about other VNFs?
#3.1. Onboarding Cumulus Linux VX
Here the situation is even more easy, as we don’t need to convert the image from one format to another, as we can download “qcow2” image from official website.
After the image is obtained, we put it the “/tmp/” folder and checking the status of the destination one:
2
3
4
5
6
-rwxrw-r--. 1 aaa aaa 1157169152 Sep 7 17:49 cumulus-linux-3.6.2-vx-amd64.qcow2
$ sudo ls -l /var/lib/libvirt/images/
password for aaa:
total 0
The second step is to move Cumulus VX image to working folder of KVM:
2
3
4
5
$ sudo ls -l /var/lib/libvirt/images/
total 1130048
-rwxr--r--. 1 root root 1157169152 Sep 7 17:53 VX1.qcow2
The preparation activities are done, so we can start the onboarding activity for Cumulus Networks VX:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
> --name=VX1 \
> --description "VX1 VM" \
> --os-type=Linux \
> --ram=1024 \
> --vcpus=1 \
> --boot hd,cdrom,menu=on \
> --disk path=/var/lib/libvirt/images/VX1.qcow2,bus=ide,size=4 \
> --import \
> --graphics vnc \
> --serial tcp,host=0.0.0.0:2261,mode=bind,protocol=telnet \
> --network=bridge:br0,mac=52:54:00:03:02:00,model=virtio \
> --network=bridge:br1,mac=52:54:00:03:02:01,model=virtio \
> --network=bridge:br2,mac=52:54:00:03:02:02,model=virtio
WARNING No operating system detected, VM performance may suffer. Specify an OS with --os-variant for optimal results.
Starting install…
Surprise, surprise! There is no diffrences are all in provisioning of VNF with Cumulus VX comparing to Cisco IOS XRv. So the set of parameters is exactly the same, and we even provide lower amount of resources (just 1 GB RAM instead of 3 GB RAM).
The major difference comparing to Cisco IOS XRv is that VNF with Cumulus VX has a working graphical terminal, just like Arista vEOS has. But, we also have an working serial CLI, so it means that initial provisioning could be done in the similar way as well, if we want:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
Trying 0.0.0.0...
Connected to 0.0.0.0.
Escape character is '^]'.
Loading Linux 4.1.0-cl-7-amd64 ...
Loading initial ramdisk ...
Loading, please wait...
Scanning for Btrfs filesystems
Welcome to Cumulus Linux!
!
! OUTPUT IS OMITTED
!
[ OK ] Started Cumulus Linux Multi-Chassis LACP Bonding Daemon.
Debian GNU/Linux 8 cumulus ttyS0
cumulus login: cumulus
Password:
Linux cumulus 4.1.0-cl-7-amd64 #1 SMP Debian 4.1.33-1+cl3u14 (2018-07-05) x86_64
Welcome to Cumulus VX (TM)
Cumulus VX (TM) is a community supported virtual appliance designed for
experiencing, testing and prototyping Cumulus Networks' latest technology.
For any questions or technical support, visit our community site at:
http://community.cumulusnetworks.com
The registered trademark Linux (R) is used pursuant to a sublicense from LMI,
the exclusive licensee of Linus Torvalds, owner of the mark on a world-wide
basis.
cumulus@cumulus:~$
The next step is to check, which interfaces are connected where. On Cumulus Linux we see interesting output, like the interfaces are visible in one output, but not in another one:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
State Name Spd MTU Mode LLDP Summary
----- ---- --- ----- -------- ---- ---------------
UP lo N/A 65536 Loopback IP: 127.0.0.1/8
lo IP: ::1/128
UP eth0 1G 1500 Mgmt
cumulus@cumulus:~$ ip link show
1: lo: <loopback,up,lower_up> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
link/ether 52:54:00:03:02:00 brd ff:ff:ff:ff:ff:ff
3: swp1: <broadcast,multicast> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 52:54:00:03:02:01 brd ff:ff:ff:ff:ff:ff
4: swp2: <broadcast,multicast> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 52:54:00:03:02:02 brd ff:ff:ff:ff:ff:ff
In Linux bridges on host OS everything looks OK so far:
2
3
4
5
6
7
8
bridge name bridge id STP enabled interfaces
br0 8000.fe5400030200 no vnet0
br1 8000.000c298ba1d3 no ens34
vnet1
br2 8000.000c298ba1dd no ens35
vnet2
virbr0 8000.525400267f78 yes virbr0-nic
So, I will check connectivity on management and one data plane interface, to make sure both are working:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
--- /etc/network/interfaces 2018-07-15 06:53:10.000000000 +0000
+++ /run/nclu/ifupdown2/interfaces.tmp 2018-09-07 16:08:56.781035636 +0000
@@ -2,11 +2,22 @@
# and how to activate them. For more information, see interfaces(5).
source /etc/network/interfaces.d/*.intf
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto eth0
-iface eth0 inet dhcp
+iface eth0
+ address 192.168.1.61/24
+ vrf mgmt
+
+auto swp1
+iface swp1
+ address 10.0.0.61/24
+
+auto mgmt
+iface mgmt
+ address 127.0.0.1/8
+ vrf-table auto
net add/del commands since the last "net commit"
================================================
User Timestamp Command
------- -------------------------- -------------------------------------------------
cumulus 2018-09-07 16:07:40.566185 net add vrf mgmt
cumulus 2018-09-07 16:08:29.285741 net add interface eth0 ip address 192.168.1.61/24
cumulus 2018-09-07 16:08:43.031818 net add interface swp1 ip address 10.0.0.61/24
cumulus@cumulus:~$ ping -I mgmt 192.168.1.1
ping: Warning: source address might be selected on device other than mgmt.
PING 192.168.1.1 (192.168.1.1) from 192.168.1.61 mgmt: 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=1.88 ms
^C
--- 192.168.1.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.880/1.880/1.880/0.000 ms
cumulus@cumulus:~$ ping 10.0.0.254
PING 10.0.0.254 (10.0.0.254) 56(84) bytes of data.
64 bytes from 10.0.0.254: icmp_seq=1 ttl=64 time=3.74 ms
64 bytes from 10.0.0.254: icmp_seq=2 ttl=64 time=0.992 ms
^C
--- 10.0.0.254 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1003ms
rtt min/avg/max/mdev = 0.992/2.366/3.740/1.374 ms
Both management and data plane interfaces are up and running. And now we see this configured interface in “netd” managed output:
2
3
4
5
6
7
8
9
State Name Spd MTU Mode LLDP Summary
----- ---- --- ----- ------------ ---- -------------------
UP lo N/A 65536 Loopback IP: 127.0.0.1/8
lo IP: ::1/128
UP eth0 1G 1500 Mgmt Master: mgmt(UP)
eth0 IP: 192.168.1.61/24
UP swp1 1G 1500 Interface/L3 IP: 10.0.0.61/24
UP mgmt N/A 65536 Interface/L3 IP: 127.0.0.1/8
The last point, we update hosts’ tables:
2
3
4
5
192.168.1.61 VX1
$ sudo cat /etc/ansible/hosts | grep 'VX1'
VX1
Onboarding for VNF with Cumulus Networks VX is done.
#3.2. Removing Cumulus Linux VX
The process of deconstructing of Cumulus Linux VX on KVM is exactly the same as already done 2 times:
2
3
4
5
6
7
8
9
10
11
12
13
14
Domain VX1 destroyed
$ sudo virsh undefine VX1
Domain VX1 has been undefined
$ sudo rm /var/lib/libvirt/images/VX1.qcow2
$ sudo ls -l /var/lib/libvirt/images/
total 0
$ sudo cat /etc/hosts | grep 'VX1'
$ sudo cat /etc/ansible/hosts | grep 'VX1'
Bye, Cumulus VX, see you by next onboarding.
#4.1. Onboarding Nokia (Alcatel-Lucent) VSR
The next in the queue for the onboarding is Nokia (Alcatel-Lucent) VSR. The reason, there are two reasons, why it stands here:
- We are using just alphabetical order
- It has a bit more complicated structure of installation instruction
Just to highlight the last point, we need to define additional information about VSR structure and license upon creation of VNF, so we need to somehow to note this information. After some time spent on searching how to do it, I’ve found the solution. In a minute I will show you how it works together with “virt-install”. But for now, we are traditionally checking files:
2
3
4
5
-rwxrw-r--. 1 aaa aaa 287309824 Sep 10 07:40 nokia-sros-16.0R1.qcow2
$ sudo ls -l /var/lib/libvirt/images/
total 0
“Copy as usual”:
2
3
4
5
$ sudo ls -l /var/lib/libvirt/images/
total 280576
-rwxr--r--. 1 root root 287309824 Sep 10 07:45 SR1.qcow2
And now we are reaching the key activity, where we onboard the VNF with Nokia VSR on KVM:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
> --name=SR1 \
> --description "SR1 VM" \
> --os-type=Linux \
> --sysinfo type='smbios',system_product='TIMOS:address=192.168.1.101/24@active static-route=192.168.0.0/16@192.168.1.1 license-file=ftp://nokia:nokianokia@192.168.1.1/sros16.lic slot=A chassis=SR-1 card=iom-1 mda/1=me6-100gb-qsfp28'\
> --ram=4096 \
> --vcpus=2 \
> --boot hd \
> --disk path=/var/lib/libvirt/images/SR1.qcow2,bus=virtio,size=4 \
> --import \
> --graphics vnc \
> --serial tcp,host=0.0.0.0:2271,mode=bind,protocol=telnet \
> --network=bridge:br0,mac=52:54:00:02:02:00,model=virtio \
> --network=bridge:br1,mac=52:54:00:02:02:01,model=virtio \
> --network=bridge:br2,mac=52:54:00:02:02:02,model=virtio
WARNING No operating system detected, VM performance may suffer. Specify an OS with --os-variant for optimal results.
Starting install…
Comparing to three previous installations, we have new key “–sysinfo”, where this additional information about Nokia (Alcatel-Lucent) VSR structure is stored. The rest of the actions is exactly the same. So we check the serial console:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
Trying 0.0.0.0...
Connected to 0.0.0.0.
Escape character is '^]'.
!
! OUTPUT IS OMITTED
!
TiMOS-B-16.0.R1 both/x86_64 Nokia 7750 SR Copyright (c) 2000-2018 Nokia.
All rights reserved. All use subject to applicable license agreements.
Built on Thu May 31 16:23:56 PDT 2018 by builder in /builds/160B/R1/panos/main
Login: admin
Password:
SR OS Software
Copyright (c) Nokia 2018. All Rights Reserved.
Trademarks
Nokia and the Nokia logo are registered trademarks of Nokia. All other
trademarks are the property of their respective owners.
IMPORTANT: READ CAREFULLY
The SR OS Software (the "Software") is proprietary to Nokia and is subject
to and governed by the terms and conditions of the End User License
Agreement accompanying the product, made available at the time of your order,
or posted on the Nokia website (collectively, the "EULA"). As set forth
more fully in the EULA, use of the Software is strictly limited to your
internal use. Downloading, installing, or using the Software constitutes
acceptance of the EULA and you are binding yourself and the business entity
that you represent to the EULA. If you do not agree to all of the terms of
the EULA, then Nokia is unwilling to license the Software to you and (a) you
may not download, install or use the Software, and (b) you may return the
Software as more fully set forth in the EULA.
This product contains cryptographic features and is subject to United States<br>and local country laws governing import, export, transfer and use. Delivery<br>of Nokia cryptographic products does not imply third-party authority to<br>import, export, distribute or use encryption.
If you require further assistance please contact us by sending an email
to support@nokia.com.
A:vSIM#
There is an official documentation regarding VSR onboarding available, but I’m not able to reach is I don’t have corresponding rights. So I will explain one key topic briefly myself. In the setup above the 100G MDA is used, but to connect it somewhere, the breakout cables are used. This configuration was already shown during OpenConfig journey (link) without explanation. On the other hand, we define ports in VM in the time of creation, so the world outside VNF doesn’t know anything about breakout ports and is looking good already now.
2
3
4
5
6
7
8
bridge name bridge id STP enabled interfaces
br0 8000.fe5400020200 no vnet0
br1 8000.000c298ba1d3 no ens34
vents
br2 8000.000c298ba1dd no ens35
vnet2
virbr0 8000.525400267f78 yes virbr0-nic
So, only proper configuration on VSR is needed. By the time of the VNF onboarding we have defined 3 ports, what translates in 1 management in BOF and 2 data planes: 1st breakout in first two ports on MDA. The configuration below shows how the Nokia VSR ports should be configured (in case we use SR-1 with breakout ports) in order to have connectivity to Linux bridges:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
A:admin@vSIM# compare
+ configure {
+ card 1 {
+ card-type iom-1
+ mda 1 {
+ mda-type me6-100gb-qsfp28
+ }
+ }
+ port 1/1/c1 {
+ admin-state enable
+ connector {
+ breakout c4-10g
+ }
+ }
+ port 1/1/c1/1 {
+ admin-state enable
+ ethernet {
+ mode network
+ }
+ }
+ port 1/1/c2 {
+ admin-state enable
+ connector {
+ breakout c4-10g
+ }
+ }
+ port 1/1/c2/1 {
+ admin-state enable
+ ethernet {
+ mode network
+ }
+ }
+ router "Base" {
+ interface "p1" {
+ admin-state enable
+ port 1/1/c1/1
+ ipv4 {
+ primary {
+ address 10.0.0.71
+ prefix-length 24
+ }
+ }
+ }
+ interface "p2" {
+ admin-state enable
+ port 1/1/c2/1
+ ipv4 {
+ primary {
+ address 10.0.1.71
+ prefix-length 24
+ }
+ }
+ }
+ }
}
*(gl)[]
A:admin@vSIM# commit
Let’s check that connectivity between Nokia (Alcatel-Lucent) VSR and Linux bridges are working in reality:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
A:admin@vSIM# //ping 10.0.0.254
INFO: CLI #2051: Switching to the classic CLI engine
A:vSIM# /ping 10.0.0.254
PING 10.0.0.254 56 data bytes
64 bytes from 10.0.0.254: icmp_seq=1 ttl=64 time=12.4ms.
64 bytes from 10.0.0.254: icmp_seq=2 ttl=64 time=2.47ms.
^C
ping aborted by user
---- 10.0.0.254 PING Statistics ----
2 packets transmitted, 2 packets received, 0.00% packet loss
round-trip min = 2.47ms, avg = 7.43ms, max = 12.4ms, stddev = 0.000ms
INFO: CLI #2052: Switching to the MD-CLI engine
(gl)[]
A:admin@vSIM# //ping 10.0.1.254
INFO: CLI #2051: Switching to the classic CLI engine
A:vSIM# /ping 10.0.1.254
PING 10.0.1.254 56 data bytes
64 bytes from 10.0.1.254: icmp_seq=1 ttl=64 time=6.99ms.
64 bytes from 10.0.1.254: icmp_seq=2 ttl=64 time=1.35ms.
^C
ping aborted by user
---- 10.0.1.254 PING Statistics ----
2 packets transmitted, 2 packets received, 0.00% packet loss
round-trip min = 1.35ms, avg = 4.17ms, max = 6.99ms, stddev = 0.000ms
INFO: CLI #2052: Switching to the MD-CLI engine
Wait, we have forgotten to test management interfaces reachability. Ok, there is no problem, we check it together with hosts table update:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
192.168.1.101 SR1
$ sudo cat /etc/ansible/hosts | grep 'SR1'
SR1
$ ping SR1
PING SR1 (192.168.1.101) 56(84) bytes of data.
64 bytes from SR1 (192.168.1.101): icmp_seq=1 ttl=64 time=6.88 ms
64 bytes from SR1 (192.168.1.101): icmp_seq=2 ttl=64 time=0.375 ms
64 bytes from SR1 (192.168.1.101): icmp_seq=3 ttl=64 time=0.946 ms
64 bytes from SR1 (192.168.1.101): icmp_seq=4 ttl=64 time=10.6 ms
64 bytes from SR1 (192.168.1.101): icmp_seq=5 ttl=64 time=1.29 ms
64 bytes from SR1 (192.168.1.101): icmp_seq=6 ttl=64 time=24.5 ms
64 bytes from SR1 (192.168.1.101): icmp_seq=7 ttl=64 time=2.83 ms
^C
--- SR1 ping statistics ---
7 packets transmitted, 7 received, 0% packet loss, time 6028ms
rtt min/avg/max/mdev = 0.375/6.792/24.559/8.034 ms
So the onboarding of the VNF with Nokia (Alcatel-Lucent) VSR is done.
#4.1. Removing Nokia (Alcatel-Lucent) VSR
The process of the removing of the VNF is already very well known, so I won’t even explain you the steps and will just proceed with some commands, how you do it:
2
3
4
5
6
7
8
9
10
11
12
13
14
Domain SR1 destroyed
$ sudo virsh undefine SR1
Domain SR1 has been undefined
$ sudo rm /var/lib/libvirt/images/SR1.qcow2
$ sudo ls -l /var/lib/libvirt/images/
total 0
$ sudo cat /etc/hosts | grep 'SR1'
$ sudo cat /etc/ansible/hosts | grep 'SR1'
This process is more than straightforward, as you can see. The VNF with Nokia (Alcatel-Lucent) VSR is successfully undeployed.
As a summary, you can download all 4 “virt-install” examples: 134_create_KVM_Cisco_IOS_XRv134_create_KVM_Cumulus_Linux134_create_KVM_Nokia_VSR134_create_KVM_Arista_vEOS
Consideration on automated deployment with Ansible
All these manual operations in terms of VNF deployment or undeployment could be automated using Ansible. Earlier we have explained how Ansible could be used as VNF-M, so that you can easily template all mandatory parameters and rollout the necessary amount of VNFs. All you need is to create proper jinja2 template per VNF type and hold it consistent with variables. I won’t create new Ansible playbook in the current article, as it doesn’t bring any additional value.
Stick to that article for an example of fully automated deployment.
Lessons learned
Devil is hidden into details. You might have seen that all installation is similar. But here “similar” is the key word, it isn’t “equal”. As a result, I need to test different sets of the parameters for installation, as I haven’t found any single documents that explains that process for KVM. So, trying and redoing is the only working approach to build any system.
The second lessons learned is that sometimes you need to take much more efforts in order to get things done. For instance, I haven’t managed to get Cisco Nexus 9000v working on KVM, though I have tried so hard that I have broken my lab and had to rebuild it from scratches. The good thing is that I have recorded how to do that.
Conclusion
Making the army of clones is always good when we start speaking about CI/CD DevOps approach. For instance, if you want to make any test on the network setup, the only working approach is to rebuild the lab, destroy it, repeat it. So it’s quite often it’s necessary to use that kind of templating. Coupling this lab and the VNF-M with Ansible gives you that tools.. Take care and good bye!
Support us
P.S.
If you have further questions or you need help with your networks, I’m happy to assist you, just send me message (https://karneliuk.com/contact/). Also don’t forget to share the article on your social media, if you like it.
BR,
Anton Karneliuk