DC. Part 6. EVPN/VXLAN for Data Centre with Nokia (Alcatel-Lucent) SR OS and Arista EOS

Hello my friend,

Probably you are already bored with data center stuff, but I’m writing about it again. The reason for that is very easy: it’s data center era now. With growth of cloud services and so on, we need to have more data centers, much more. That’s why I’m introducing interoperability between Nokia (Alcatel-Lucent) SR OS and new product in my lab, which is Arista EOS.

Brief description

Recently I was testing some features in Cumulus Linux and found out that they aren’t supported now, but they are on road map and will be supported in near future. So I started looking for another DC vendor in order to make the most use of my time. Very quickly I’ve found Arista, which seems to be good in enterprise and data center fields, according to the information from their website. Moreover, they have fully virtual switch or router vEOS, whatever we call it. There are two options available: for the lab and for the production. The lab vEOS is free of charge and you can download it from the website (link is provided in the bottom of the page) without a problem, you just need to create account at Arista website.

Some words about Arista EOS (after I have played with it a couple of hours). In a nutshell it’s Linux, but it also has its own CLI (for example, like Cumulus has), which is VERY Cisco-like. Especially Cisco IOS / IOS XE, rather than Cisco IOS XR or Cisco NX-OS.

If you have previous experience with Cisco IOS / IOS XE, this experience is very useful for Arista EOS.

Also Arista claims to support deployment over white boxes, what is also available for Cumulus Linux, but isn’t yet available for Nokia. I hope the latter could change.

Apart from that, it’s new networking vendor, we are going to get naked into our interoperability tests. Let’s get rock!

What we are going to test?

To save the time and give a kind of comparison, we’ll rebuild data centre, we have built with Cisco spines and pure Nokia (Alcatel-Lucent) SR OS based leafs or mixed Nokia (Alcatel-Lucent) SR OS with Cumulus Linux. Hence, we’ll deploy two scenarios:

• EVPN/VXLAN for pure L2 switching within broadcast domain
• EVPN/VXLAN for IRB (integrated routing and bridging) between 2 broadcast domains with anycast gateway. To be honest, anycast or not will be seen further, when we reach configuration of corresponding feature.

Software version

The following infrastructure is used in my lab:

• CentOS 7 with python 2.7.
• Ansible 2.4.2
• Nokia (Alcatel-Lucent) SR OS 15.0.R7
• Arista EOS 4.20.5F
• Cisco IOS XR 6.2.1

See the previous article to get details how to build the lab.

Topology

Physical topology doesn’t change from the previous lab; we just add additional node to the lab environment:

The following nodes are active: SR1, vEOS2, XR3 and XR4

In the logical topology, we follow exactly the same topology we have used previously with two spines (Cisco IOS XR) and two leafs (Nokia (Alcatel-Lucent) SR OS and Arista EOS):

Like in the previous lab for Cumulus Linux, we start with preconfigured basic part on spine switches operated by Cisco IOS XR and basic configuration on the leaf switch operated by Nokia (Alcatel-Lucent) SR OS and no configuration on the second leaf by Arista EOS: 120_config_initial_linux 118_config_initial_SR1 118_config_initial_XR3 118_config_initial_XR4

Deployment of VM with Arista EOS and its initial configuration

As some times ago in the article about Cumulus Linux and Nokia SR OS interoperability for data centers, we start today with the installation of vEOS. You can download fully working Arista vEOS for lab absolutely officially from their website, you need just to create account with guest access:

I have highlighted the files, I have downloaded myself and generally they are enough to launch Arista vEOS in our virtual data center testbed.

There are some guidelines in the official documentation, which we need to follow upon VM creation:

In this example I’m using VMware Workstation 12 Player, though it can be launched on KVM as well.

So, let’s create the Arista vEOS.

#1. Create VM.

#2. Select install OS later.

#3. Select OS type

#4. Set name for VM

#5. Set disk type

At this stage I can’t choose the existing VMDK image or even drive’s type. So I just point to create a single file, so that later I will replace it.

#6. Finish VM draft

At this point we have created default VM (really default, as we haven’t chosen any Arista vEOS related files). Now we need to tune it and make it workable. So we make finish and then chose “Edit virtual machine settings” from main panel.

#7. Fine tuning – updating available memory

In the official documentation Arista proposes to have 2GB RAM per vEOS, so let’s provide it:

#8. Fine tuning – CPU virtualization

Don’t forget to enable VT-x:

#9. Fine tuning – proper hard disk

Here we do several steps:

• In the folder with VM we copy downloaded in the beginning VMDK file
• Here in configuration we remove created drive
• Add new hard drive of IDE type and choosing copied VMDK

I don’t show the process of copying or removing as it’s self-explanatory. Here is the creation:

Choose of proper hard drive type:

We’ll take the existing drive:

Finally we chose the copied disk:

#10. Fine tuning – inserting disk in CD

As it’s said in the official documentation, we need to insert additional file into CD:

#11. Fine tuning – Network interfaces

The last point before the launch is the extending number of network interfaces. By default, only one interface is created, which is management one, much the same way we had in Cumulus Linux or even Nokia (Alcatel-Lucent) SR OS or Cisco IOS XR:

Our VM is created and ready for the first launch. I’m excited, and what about you?

If everything was done properly (according to the manual above), you will see Arista vEOS started booting:

It takes a couple of minutes to get started (longer that Cumulus VX but less than Nokia (Alcatel-Lucent) VSR and much less than Cisco IOS XRv). In a couple of minutes, you will see the invitation to login:

#12. Configuring username and management interface

Default login name is “admin” without password. It won’t work for SSH access, so we’ll have to create it as part of initial configuration. Let’s start with it and with the configuration if the management port, so that we can connect to Arista EOS with SSH:

Cisco fans, are you ready?!

Don’t forget to save configuration by “write memory” after all changes.

Now we are able to connect to Arista EOS leaf switch vEOS2 using SSH client, like Putty:

#13. Configuring connectivity with the rest of the data centre fabric

The last topic on initial agenda is the configuration of connectivity to other devices in our data centre, fabric, more precisely two spine switches based on Cisco IOS XR:

Only relevant part of the configuration is shown

Let’s check the reachability of spine switches:

As basic connectivity is established, we go further with BGP configuration.

BGP configuration of leaf switches (Nokia (Alcatel-Lucent) SR OS and Cumulus Linux)

Following the same approach, we have for building data center fabric with Cumulus Linux and Nokia (Alcatel-Lucent) SR OS, we have to major building blocks for BGP:

• Configuration of BGP for underlay IP fabric
• Configuration of BGP for overlay EVPN

BGP-based underlay IP fabric

We build already well-known topology for BGP:

Configuration of BGP for Nokia (Alcatel-Lucent) SR OS is coming from previous articles (link), so we hope you understand it well already. The new one is coming for Arista EOS:

SR1 – Nokia (Alcatel-Lucent) VSR

vEOS2 – Arista EOS

A:SR1>edit-cfg# candidate view ========================= configure router policy-option begin prefix-list PL_BGP_LO prefix 10.0.0.0/24 prefix-length-range 32-32 exit policy-statement RP_BGP_IPV4_UNICAST default-action drop exit entry 10 from prefix-list PL_BGP_LO exit action accept exit exit exit commit exit autonomous-system 65011 router-id 10.0.0.11 ecmp 2 bgp multipath 2 next-hop-resolution use-bgp-routes group FABRIC family ipv4 export RP_BGP_IPV4_UNICAST authentication-key FABRIC neighbor 10.11.33.33 peer-as 65001 exit neighbor 10.11.44.44 peer-as 65002 exit exit exit exit exit =========================

vEOS2#show run ! Command: show running-config ! device: vEOS2 (vEOS, EOS-4.20.1F) ! service routing protocols model multi-agent ! interface Loopback0 ip address 10.0.0.22/32 ! ip routing ! router bgp 65012 router-id 10.0.0.22 maximum-paths 4 ecmp 4 neighbor 10.22.33.33 remote-as 65001 neighbor 10.22.33.33 password FABRIC neighbor 10.22.33.33 maximum-routes 12000 neighbor 10.22.44.44 remote-as 65002 neighbor 10.22.44.44 password FABRIC neighbor 10.22.44.44 maximum-routes 12000 ! address-family ipv4 neighbor 10.22.33.33 activate neighbor 10.22.44.44 activate network 10.0.0.22/32 ! end

Command “service routing protocols model multi-agent” is of particular importance. If don’t enter it, you won’t be able to configure EVPN address family

Actually configuration of BGP in Arista EOS is very similar to one in Cisco, even in Cisco IOS XR (link), so it’s easy for us, as we have configured it many times. In the example, we just create eBGP peering on interface level between leaf and spine switches. Let’s check these BGP sessions. We start at Arista EOS leaf switch vEOS1:

At Nokia (Alcatel-Lucent) SR OS:

BGP peering is established as proper and we see some routes being propagated. It’s very easy to guess that these are our loopbacks, but let’s double check that:

And from Nokia prospective we will also check BGP RIB, but you remember locally advertised routes are shown there:

So, control plane looks fine for our underlay data centre fabric, and before we move further we check that data plane works as well. How will we do that? By utilizing one of the most powerful tool in network engineer’s toolkit:

BGP-based overlay EVPN

The next step in our journey is to establish multihop eBGP peering between Nokia (Alcatel-Lucent) SR OS leaf SR1 and Arista EOS lead vEOS2 with the following topology:

To do that we implement the configuration below:

SR1 – Nokia (Alcatel-Lucent) VSR

vEOS2 – Arista EOS

A:SR1>edit-cfg# candidate view ========================= configure router bgp rapid-update rapid-withdrawal group OVERLAY family evpn authentication-key OVERLAY neighbor 10.0.0.22 local-address 10.0.0.11 multihop 5 peer-as 65012 exit exit exit exit exit =========================

vEOS2#show run ! Command: show running-config ! device: vEOS2 (vEOS, EOS-4.20.1F) ! router bgp 65012 neighbor 10.0.0.11 remote-as 65011 neighbor 10.0.0.11 update-source Loopback0 neighbor 10.0.0.11 ebgp-multihop 5 neighbor 10.0.0.11 password OVERLAY neighbor 10.0.0.11 send-community extended ! address-family evpn neighbor 10.0.0.11 activate ! address-family ipv4 no neighbor 10.0.0.11 activate ! end

Also the configuration is relatively easy. There is one point specific to Arista EOS (at least it’s shown in the official config guide), which is activation of extended communities so that they are sent to neighbour. In Nokia (Alcatel-Lucent) or Cisco IOS XR or Cumulus Linux it’s done automatically, though in Cisco IOS you need to manually activate that.

Different vendors mean different default behaviour, just know what exactly needed technologywise and implement that.

For now we can check only the status of BGP peering for EVPN, as we haven’t advertised anything yet. In Arista EOS the command is the following:

In Nokia (Alcatel-Lucent) we just use the same command we have used previously for checking BGP session for IPv4 unicast address family, but this time we extend it with address family:

At this point we consider that data centre fabric, both underlay and overlay, is fully ready and we can proceed with customer’s use cases.

Case #1. Switching within L2 domain (L2 option of EVPN with VXLAN)

We start out journey to EVPN/VXLAN use cases with building switching domain across two leafs, hence pure L2 use case with the following topology:

The configuration below is necessary to be applied at both our leaf switches by Nokia (Alcatel-Lucent) SR OS and Arista EOS:

SR1 – Nokia (Alcatel-Lucent) VSR

vEOS2 – Arista EOS

A:SR1>edit-cfg# candidate view ========================= configure service customer 2 create description TEST_EVPN_TENNANT exit vpls 1000789 customer 2 create vxlan vni 789 create exit bgp route-distinguisher 10.0.0.11:789 route-target export target:65000:789 import target:65000:789 exit bgp-evpn vxlan no shutdown exit mpls shutdown exit exit stp shutdown exit sap 1/1/2:555 create no shutdown exit proxy-arp no shutdown exit proxy-nd evpn-nd-advertise router no shutdown exit no shutdown exit exit exit =========================

vEOS2#show run ! Command: show running-config ! device: vEOS2 (vEOS, EOS-4.20.1F) ! vlan 666 name Tennant_L2 ! interface Vxlan1 vxlan source-interface Loopback0 vxlan udp-port 4789 vxlan vlan 666 vni 789 ! router bgp 65012 vlan 666 rd 10.0.0.22:789 route-target both 65000:789 redistribute learned ! ! end

Here we can see at a glance two difference logic of the configuration. In Nokia (Alcatel-Lucent) SR OS all the configuration related to particular service, like here we speak about particular EVPN service, is contained without single configuration context. In Arista EOS, much like we’ve seen recently in Cumulus Linux (link), the configuration of EVPN service is splitted across VLAN, interface VXLAN, which contains both VTEP address and VNI to VLAN mapping, and BGP context, where we assign RD/RT and inform BGP to distribute learned MAC addresses.

On control plane we check the BGP RIB for EVPN. On Arista EOS we use the following command:

We see 5 routes: 2 of them are type 3 routes, which are announced by VTEPs and used for BUM traffic, 3 of them are host routes. Why do we have 3 type-2 routes? 2 of them, marked with bold are MAC addresses of our VMs, whereas third one comes into play-based on the fact we have enabled proxy-arp as this MAC relates to SR 7750 chassis:

At Nokia (Alcatel-Lucent) SR OS we can see BGP RIB for incoming routes and separately BGP-RIB-OUT for routes being sent to neighbors.

Control plane looks OK, what is good sign. Now we see into data plane to double check that. At Arista EOS we have two commands for that, which show a bit different but mainly similar information:

As Nokia (Alcatel-Lucent) follows service-oriented model, so we check the forwarding database per service:

The outputs from different commands looks nice and we need to do the last thing on this verification path

Case #1. Verification

As usual we test our data plane by issuing ping command between our VMs:

Configuration of Cisco IOS XR to emulate VMs as well as packet capture is provided in one of the previous articles

Case #2. Inter-VXLAN routing (L3 option of EVPN with VXLAN – distributed gateway)

Going further, we deploy asymmetric IRB, or inter VXLAN routing, with the aim to deploy anycast gateway at both Nokia (Alcatel-Lucent) SR OS and Arista EOS leafs to provide possibility of two broadcast domains talk to each other. The image below shows broadcast domain 1:

And the next one image reflects domain 2:

The following configuration must be implemented:

SR1 – Nokia (Alcatel-Lucent) VSR

vEOS2 – Arista EOS

A:SR1>edit-cfg# candidate view ========================= configure service vpls 1000123 customer 2 create allow-ip-int-bind exit vxlan vni 123 create exit bgp route-distinguisher 10.0.0.11:123 route-target export target:65000:123 import target:65000:123 exit bgp-evpn vxlan no shutdown exit mpls shutdown exit exit stp shutdown exit sap 1/1/2:111 create no shutdown exit proxy-arp shutdown exit proxy-nd shutdown exit service-name “L2_DOMAIN_1” no shutdown exit vpls 1000456 customer 2 create allow-ip-int-bind exit vxlan vni 456 create exit bgp route-distinguisher 10.0.0.11:456 route-target export target:65000:456 import target:65000:456 exit bgp-evpn vxlan no shutdown exit mpls shutdown exit exit stp shutdown exit sap 1/1/2:333 create no shutdown exit proxy-arp shutdown exit proxy-nd shutdown exit service-name “L2_DOMAIN_2” no shutdown exit vprn 20000000 customer 2 create route-distinguisher 10.0.0.11:2000 vrf-target export target:65011:456 import target:65012:456 interface “IRB_VXLAN1” create address 192.168.0.253/24 vrrp 1 passive backup 192.168.0.250 ping-reply traceroute-reply mac 00:00:5e:00:00:01 exit local-proxy-arp mac 00:20:00:01:23:01 vpls “L2_DOMAIN_1” exit exit interface “IRB_VXLAN2” create address 192.168.1.253/24 vrrp 2 passive backup 192.168.1.250 ping-reply traceroute-reply mac 00:00:5e:00:00:01 exit local-proxy-arp mac 00:20:00:04:56:01 vpls “L2_DOMAIN_2” exit exit no shutdown exit exit exit =========================

vEOS2#show run ! Command: show running-config ! device: vEOS2 (vEOS, EOS-4.20.1F) ! vlan 222 name CUST_L3_DOMAIN_1 ! vlan 444 name CUST_L3_DOMAIN_2 ! vrf definition CUST ! interface Ethernet2 load-interval 30 switchport trunk allowed vlan 222,444,666 switchport mode trunk ! interface Vlan222 vrf forwarding CUST ip proxy-arp ip local-proxy-arp ip address 192.168.0.254/24 ip virtual-router address 192.168.0.250 ! interface Vlan444 vrf forwarding CUST ip proxy-arp ip local-proxy-arp ip address 192.168.1.254/24 ip virtual-router address 192.168.1.250 ! interface Vxlan1 vxlan source-interface Loopback0 vxlan udp-port 4789 vxlan vlan 222 vni 123 vxlan vlan 444 vni 456 vxlan vlan 666 vni 789 ! ip virtual-router mac-address 00:00:5e:00:00:01 ! ip routing vrf CUST ! router bgp 65012 ! vlan 222 rd 10.0.0.22:123 route-target both 65000:123 redistribute learned ! vlan 444 rd 10.0.0.22:456 route-target both 65000:456 redistribute learned ! ! end

In Nokia (Alcatel-Lucent) SR OS we create two L2 domains within VPLS context and map them to L3 instance, which is called VPRN in Nokia (Alcatel-Lucent) SR OS.

More information about Nokia (Alcatel-Lucent) VPRN you will find in the dedicated article.

What is important, we deploy distributed anycast GW configuration, shared by @Jorge Rabadan, and you will see later on, that this configuration works interoperable with distributed anycast GW from Arista EOS! That’s really amazing!

In Arista EOS, comparing to configuration of L2 EVPN service, we add VLAN interfaces with IP addresses and virtual-address for anycast GW as well as corresponding VRF, where we map the customer. To make distributed anycast GW working we configure the same MAC address for virtual instances both at Nokia (Alcatel-Lucent) SR OS and Arista EOS leaf switches.

We begin our verifications traditionally with BGP RIB for EVPN:

Based on route distinguisher we can understand, which routes are related to particular VNIs, though we can extend command, like “show bgp evpn vni xxx”, to see all the routes belonging to interested VRF.

In Nokia (Alcatel-Lucent) SR OS we check to BGP RIB for EVPN as well:

As you see, the amount of routes are less due to Nokia (Alcatel-Lucent) SR OS behaviour, we have explained previously,

We won’t check the MAC address table in this case, as we did in in the first case and it shew correct output. Here we’ll check two more things. The first one is related to ARP to check that all the tables are build correctly.

Nokia (Alcatel-Lucent) SR OS:

Arista EOS:

And the last verification, before we jump to actual packets’ transmission, is to show the operation of component related to distributed anycast GW.

Arista EOS:

Nokia (Alcatel-Lucent) SR OS:

So far the outputs look fabulous and we proceed further.

Case #2. Verification

First of all, we check connectivity between all VMs within these two domains:

As you see, everything is working, what is a step forward comparing to interoperability tests between Nokia (Alcatel-Lucent) SR OS and Cumulus Linux, where we had some problems with ARP.

The most interesting case for me is that distributed anycast GW works into interoperable scenario. I’ve done two tests for that.

The first one is to ping VM3 from VM1 as both of them are connected to the same leaf switch SR1 and check what is going on in the wire:

During this test we monitor with Wireshark all the links and see this traffic only on VLAN 111 and 333:

The second test touches VM2 and VM4, which are connected to vEOS2:

And again Wireshark shows the traffic only on interconnected links, which are VLANs 222 and 444:

Trace for ICMP over VXLAN is provided in article about Cumulus Linux interoperability

Here are the final configuration files of this lab: 120_config_final_XR3 120_config_final_XR4 120_config_final_SR1 120_config_final_VEOS2

Lessons learned

It was my first time working with Arista EOS, same as earlier with Cumulus Linux, and my main observations are the next ones.

First, CLI in Arista EOS is quite user friendly, so if I have forgotten to enter critical commands, it informs about it:

Second. another nice feature in Arista EOS allows us to check the configuration in the exiting context, exactly how we do in Nokia (Alcatel-Lucent) SR OS by issuing “info”:

Third, Arista also doesn’t share a lot of documentations, just as Nokia does. A lot of links I was trying to open at Arista website was telling me that I can’t access requested information as my level of the access is guest. In the time, when industry is moving into open-source networking, such approach looks weird for me.

Conclusion

Arista EOS is relatively new comparing to old giants like Nokia (merged with Alcatel-Lucent) or Cisco. And this is good, as they have great ideas and moving much quicker than mentioned companies. The same is relevant for Cumulus Linux, which is small and therefore moving forward at a good pace. But, what is more important for this particular article, we have reached the interoperability for all use cases between Nokia (Alcatel-Lucent) SR OS and Arista EOS, which looks very promising for data centre world. Take care and good bye!

P.S.

If you have further questions or you need help with your networks, I’m happy to assist you, just send me message. Also don’t forget to share the article on your social media, if you like it.

Support us

BR,

Anton Karneliuk

Useful links

During preparation of the article I have found very good article at Arista website about data centre fabric configuration, which I have used to study EVPN in Arista EOS.