Hello my friend,

Probably you are already bored with data center stuff, but I’m writing about it again. The reason for that is very easy: it’s data center era now. With growth of cloud services and so on, we need to have more data centers, much more. That’s why I’m introducing interoperability between Nokia (Alcatel-Lucent) SR OS and new product in my lab, which is Arista EOS.

Brief description

Recently I was testing some features in Cumulus Linux and found out that they aren’t supported now, but they are on road map and will be supported in near future. So I started looking for another DC vendor in order to make the most use of my time. Very quickly I’ve found Arista, which seems to be good in enterprise and data center fields, according to the information from their website. Moreover, they have fully virtual switch or router vEOS, whatever we call it. There are two options available: for the lab and for the production. The lab vEOS is free of charge and you can download it from the website (link is provided in the bottom of the page) without a problem, you just need to create account at Arista website.

Some words about Arista EOS (after I have played with it a couple of hours). In a nutshell it’s Linux, but it also has its own CLI (for example, like Cumulus has), which is VERY Cisco-like. Especially Cisco IOS / IOS XE, rather than Cisco IOS XR or Cisco NX-OS.

If you have previous experience with Cisco IOS / IOS XE, this experience is very useful for Arista EOS.

Also Arista claims to support deployment over white boxes, what is also available for Cumulus Linux, but isn’t yet available for Nokia. I hope the latter could change.

Apart from that, it’s new networking vendor, we are going to get naked into our interoperability tests. Let’s get rock!

What we are going to test?

To save the time and give a kind of comparison, we’ll rebuild data centre, we have built with Cisco spines and pure Nokia (Alcatel-Lucent) SR OS based leafs or mixed Nokia (Alcatel-Lucent) SR OS with Cumulus Linux. Hence, we’ll deploy two scenarios:

  • EVPN/VXLAN for pure L2 switching within broadcast domain
  • EVPN/VXLAN for IRB (integrated routing and bridging) between 2 broadcast domains with anycast gateway. To be honest, anycast or not will be seen further, when we reach configuration of corresponding feature.

Software version

The following infrastructure is used in my lab:

  • CentOS 7 with python 2.7.
  • Ansible 2.4.2
  • Nokia (Alcatel-Lucent) SR OS 15.0.R7
  • Arista EOS 4.20.5F
  • Cisco IOS XR 6.2.1

See the previous article to get details how to build the lab.

Topology

Physical topology doesn’t change from the previous lab; we just add additional node to the lab environment:

The following nodes are active: SR1, vEOS2, XR3 and XR4

In the logical topology, we follow exactly the same topology we have used previously with two spines (Cisco IOS XR) and two leafs (Nokia (Alcatel-Lucent) SR OS and Arista EOS):

Like in the previous lab for Cumulus Linux, we start with preconfigured basic part on spine switches operated by Cisco IOS XR and basic configuration on the leaf switch operated by Nokia (Alcatel-Lucent) SR OS and no configuration on the second leaf by Arista EOS: 120_config_initial_linux 118_config_initial_SR1 118_config_initial_XR3 118_config_initial_XR4

Deployment of VM with Arista EOS and its initial configuration

As some times ago in the article about Cumulus Linux and Nokia SR OS interoperability for data centers, we start today with the installation of vEOS. You can download fully working Arista vEOS for lab absolutely officially from their website, you need just to create account with guest access:

I have highlighted the files, I have downloaded myself and generally they are enough to launch Arista vEOS in our virtual data center testbed.

There are some guidelines in the official documentation, which we need to follow upon VM creation:

1
2
3
4
5
Hypervisor-agnostic Configuration
.
The Aboot-veos iso must be set as a CD-ROM image on the IDE bus, and the EOS vmdk must
be a hard drive image on the same IDE bus. The simulated hardware cannot contain a SATA
controller or vEOS will fail to fully boot.

In this example I’m using VMware Workstation 12 Player, though it can be launched on KVM as well.

So, let’s create the Arista vEOS.

#1. Create VM.

#2. Select install OS later.

#3. Select OS type

#4. Set name for VM

#5. Set disk type

At this stage I can’t choose the existing VMDK image or even drive’s type. So I just point to create a single file, so that later I will replace it.

#6. Finish VM draft

At this point we have created default VM (really default, as we haven’t chosen any Arista vEOS related files). Now we need to tune it and make it workable. So we make finish and then chose “Edit virtual machine settings” from main panel.

#7. Fine tuning – updating available memory

In the official documentation Arista proposes to have 2GB RAM per vEOS, so let’s provide it:

#8. Fine tuning – CPU virtualization

Don’t forget to enable VT-x:

#9. Fine tuning – proper hard disk

Here we do several steps:

  • In the folder with VM we copy downloaded in the beginning VMDK file
  • Here in configuration we remove created drive
  • Add new hard drive of IDE type and choosing copied VMDK

I don’t show the process of copying or removing as it’s self-explanatory. Here is the creation:

Choose of proper hard drive type:

We’ll take the existing drive:

Finally we chose the copied disk:

#10. Fine tuning – inserting disk in CD

As it’s said in the official documentation, we need to insert additional file into CD:

#11. Fine tuning – Network interfaces

The last point before the launch is the extending number of network interfaces. By default, only one interface is created, which is management one, much the same way we had in Cumulus Linux or even Nokia (Alcatel-Lucent) SR OS or Cisco IOS XR:

Our VM is created and ready for the first launch. I’m excited, and what about you?

If everything was done properly (according to the manual above), you will see Arista vEOS started booting:

It takes a couple of minutes to get started (longer that Cumulus VX but less than Nokia (Alcatel-Lucent) VSR and much less than Cisco IOS XRv). In a couple of minutes, you will see the invitation to login:

1
localhost login: _

#12. Configuring username and management interface

Default login name is “admin” without password. It won’t work for SSH access, so we’ll have to create it as part of initial configuration. Let’s start with it and with the configuration if the management port, so that we can connect to Arista EOS with SSH:

Cisco fans, are you ready?!

1
2
3
4
5
6
7
8
localhost> enable
localhost# configure
localhost(config)# username aaa privilege 15 secret aaa
localhost(config)# interface management 1
localhost(config-if-Ma1)# ip address 192.168.44.82/24
localhost(config-if-Ma1)# hostname vEOS2
localhost(config)# end
localhost# write memory

Don’t forget to save configuration by “write memory” after all changes.

Now we are able to connect to Arista EOS leaf switch vEOS2 using SSH client, like Putty:

1
2
3
4
login as: aaa
Using keyboard-interactive authentication.
Password:
vEOS2>

#13. Configuring connectivity with the rest of the data centre fabric

The last topic on initial agenda is the configuration of connectivity to other devices in our data centre, fabric, more precisely two spine switches based on Cisco IOS XR:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
.vEOS2#show run
.! Command: show running-config
.! device: vEOS2 (vEOS, EOS-4.20.1F)
.!
.interface Ethernet1
.   no switchport
.!
.interface Ethernet1.23
.   logging event link-status
.   encapsulation dot1q vlan 23
.   ip address 10.22.33.22/24
.!
.interface Ethernet1.24
.   logging event link-status
.   encapsulation dot1q vlan 24
.   ip address 10.22.44.22/24
.!

Only relevant part of the configuration is shown

Let’s check the reachability of spine switches:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
vEOS2#ping  10.22.33.33
PING 10.22.33.33 (10.22.33.33) 72(100) bytes of data.
80 bytes from 10.22.33.33: icmp_seq=1 ttl=255 time=15.5 ms
80 bytes from 10.22.33.33: icmp_seq=2 ttl=255 time=10.4 ms
80 bytes from 10.22.33.33: icmp_seq=3 ttl=255 time=8.37 ms
80 bytes from 10.22.33.33: icmp_seq=4 ttl=255 time=9.03 ms
80 bytes from 10.22.33.33: icmp_seq=5 ttl=255 time=8.28 ms
--- 10.22.33.33 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 58ms
rtt min/avg/max/mdev = 8.289/10.333/15.533/2.712 ms, ipg/ewma 14.590/12.805 ms
.
.
vEOS2#ping  10.22.44.44
PING 10.22.44.44 (10.22.44.44) 72(100) bytes of data.
80 bytes from 10.22.44.44: icmp_seq=1 ttl=255 time=13.2 ms
80 bytes from 10.22.44.44: icmp_seq=2 ttl=255 time=11.1 ms
80 bytes from 10.22.44.44: icmp_seq=3 ttl=255 time=9.37 ms
80 bytes from 10.22.44.44: icmp_seq=4 ttl=255 time=9.22 ms
80 bytes from 10.22.44.44: icmp_seq=5 ttl=255 time=7.30 ms
--- 10.22.44.44 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 58ms
rtt min/avg/max/mdev = 7.302/10.211/13.234/2.712 ms, ipg/ewma 14.590/12.805 ms

As basic connectivity is established, we go further with BGP configuration.

BGP configuration of leaf switches (Nokia (Alcatel-Lucent) SR OS and Cumulus Linux)

Following the same approach, we have for building data center fabric with Cumulus Linux and Nokia (Alcatel-Lucent) SR OS, we have to major building blocks for BGP:

  • Configuration of BGP for underlay IP fabric
  • Configuration of BGP for overlay EVPN

BGP-based underlay IP fabric

We build already well-known topology for BGP:

Configuration of BGP for Nokia (Alcatel-Lucent) SR OS is coming from previous articles (link), so we hope you understand it well already. The new one is coming for Arista EOS:

SR1 – Nokia (Alcatel-Lucent) VSR vEOS2 – Arista EOS

A:SR1>edit-cfg# candidate view
=========================
configure
router
policy-option
begin
prefix-list PL_BGP_LO
prefix 10.0.0.0/24 prefix-length-range 32-32
exit
policy-statement RP_BGP_IPV4_UNICAST
default-action drop
exit
entry 10
from
prefix-list PL_BGP_LO
exit
action accept
exit
exit
exit
commit
exit
autonomous-system 65011
router-id 10.0.0.11
ecmp 2
bgp
multipath 2
next-hop-resolution use-bgp-routes
group FABRIC
family ipv4
export RP_BGP_IPV4_UNICAST
authentication-key FABRIC
neighbor 10.11.33.33
peer-as 65001
exit
neighbor 10.11.44.44
peer-as 65002
exit
exit
exit
exit
exit
=========================

vEOS2#show run
! Command: show running-config
! device: vEOS2 (vEOS, EOS-4.20.1F)
!
service routing protocols model multi-agent
!
interface Loopback0
ip address 10.0.0.22/32
!
ip routing
!
router bgp 65012
router-id 10.0.0.22
maximum-paths 4 ecmp 4
neighbor 10.22.33.33 remote-as 65001
neighbor 10.22.33.33 password FABRIC
neighbor 10.22.33.33 maximum-routes 12000
neighbor 10.22.44.44 remote-as 65002
neighbor 10.22.44.44 password FABRIC
neighbor 10.22.44.44 maximum-routes 12000
!
address-family ipv4
neighbor 10.22.33.33 activate
neighbor 10.22.44.44 activate
network 10.0.0.22/32
!
end

Command “service routing protocols model multi-agent” is of particular importance. If don’t enter it, you won’t be able to configure EVPN address family

Actually configuration of BGP in Arista EOS is very similar to one in Cisco, even in Cisco IOS XR (link), so it’s easy for us, as we have configured it many times. In the example, we just create eBGP peering on interface level between leaf and spine switches. Let’s check these BGP sessions. We start at Arista EOS leaf switch vEOS1:

1
2
3
4
5
6
7
.vEOS2#show bgp ipv4 unicast summary
.BGP summary information for VRF default
.Router identifier 10.0.0.22, local AS number 65012
.Neighbor Status Codes: m - Under maintenance
.  Neighbor         V  AS           MsgRcvd   MsgSent  InQ OutQ  Up/Down State  PfxRcd PfxAcc
.  10.22.33.33      4  65001             23        21    0    0 00:12:35 Estab  3      3
.  10.22.44.44      4  65002             22        24    0    0 00:13:11 Estab  3      3

At Nokia (Alcatel-Lucent) SR OS:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
.A:SR1# show router bgp summary
.===============================================================================
. BGP Router ID:10.0.0.11        AS:65011       Local AS:65011
.===============================================================================
.BGP Summary
.===============================================================================
.Legend : D - Dynamic Neighbor
.===============================================================================
.Neighbor
.Description
.                   AS PktRcvd InQ  Up/Down   State|Rcv/Act/Sent (Addr Family)
.                      PktSent OutQ
.-------------------------------------------------------------------------------
.10.11.33.33
.               65001       58    0 00h26m03s 2/2/4 (IPv4)
.                           62    0
.10.11.44.44
.               65002       58    0 00h24m07s 2/2/4 (IPv4)
.                           68    0
.-------------------------------------------------------------------------------

BGP peering is established as proper and we see some routes being propagated. It’s very easy to guess that these are our loopbacks, but let’s double check that:

1
2
3
4
5
6
7
8
9
10
11
12
13
.vEOS2#show bgp ipv4 unicast
.BGP routing table information for VRF default
.Router identifier 10.0.0.22, local AS number 65012
.Route status codes: s - suppressed, * - valid, > - active, # - not installed, E - ECMP head, e - ECMP
.                    S - Stale, c - Contributing to ECMP, b - backup, L - labeled-unicast
.                    % - Pending BGP convergence
.Origin codes: i - IGP, e - EGP, ? - incomplete
.AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop
.
.         Network             Next Hop         Metric  LocPref Weight Path
. * >     10.0.0.11/32        10.22.33.33      -       100     0      65001 65011 i
. * >     10.0.0.22/32        -                -       -       0      i
. * >     10.0.0.33/32        10.22.33.33      0       100     0      65001 i

And from Nokia prospective we will also check BGP RIB, but you remember locally advertised routes are shown there:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
.A:SR1# show router bgp routes ipv4
.===============================================================================
. BGP Router ID:10.0.0.11        AS:65011       Local AS:65011
.===============================================================================
. Legend -
. Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
.                 l - leaked, x - stale, > - best, b - backup, p - purge
. Origin codes  : i - IGP, e - EGP, ? - incomplete
.
.===============================================================================
.BGP IPv4 Routes
.===============================================================================
.Flag  Network                                            LocalPref   MED
.      Nexthop (Router)                                   Path-Id     Label
.      As-Path
.-------------------------------------------------------------------------------
.u*>i  10.0.0.22/32                                       None        None
.      10.11.33.33                                        None        -
.      65001 65012
.u*>i  10.0.0.33/32                                       None        0
.      10.11.33.33                                        None        -
.      65001
.-------------------------------------------------------------------------------
.Routes : 2
.===============================================================================

So, control plane looks fine for our underlay data centre fabric, and before we move further we check that data plane works as well. How will we do that? By utilizing one of the most powerful tool in network engineer’s toolkit:

1
2
3
4
5
6
7
8
9
10
vEOS2#ping 10.0.0.11 source 10.0.0.22
PING 10.0.0.11 (10.0.0.11) from 10.0.0.22 : 72(100) bytes of data.
80 bytes from 10.0.0.11: icmp_seq=1 ttl=63 time=40.7 ms
80 bytes from 10.0.0.11: icmp_seq=2 ttl=63 time=18.7 ms
80 bytes from 10.0.0.11: icmp_seq=3 ttl=63 time=18.6 ms
80 bytes from 10.0.0.11: icmp_seq=4 ttl=63 time=23.0 ms
80 bytes from 10.0.0.11: icmp_seq=5 ttl=63 time=18.7 ms
--- 10.0.0.11 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 152ms
rtt min/avg/max/mdev = 18.620/23.984/40.798/8.574 ms, ipg/ewma 38.099/32.126 ms

BGP-based overlay EVPN

The next step in our journey is to establish multihop eBGP peering between Nokia (Alcatel-Lucent) SR OS leaf SR1 and Arista EOS lead vEOS2 with the following topology:

To do that we implement the configuration below:

SR1 – Nokia (Alcatel-Lucent) VSR vEOS2 – Arista EOS

A:SR1>edit-cfg# candidate view
=========================
configure
router
bgp
rapid-update
rapid-withdrawal
group OVERLAY
family evpn
authentication-key OVERLAY
neighbor 10.0.0.22
local-address 10.0.0.11
multihop 5
peer-as 65012
exit
exit
exit
exit
exit
=========================

vEOS2#show run
! Command: show running-config
! device: vEOS2 (vEOS, EOS-4.20.1F)
!
router bgp 65012
neighbor 10.0.0.11 remote-as 65011
neighbor 10.0.0.11 update-source Loopback0
neighbor 10.0.0.11 ebgp-multihop 5
neighbor 10.0.0.11 password OVERLAY
neighbor 10.0.0.11 send-community extended
!
address-family evpn
neighbor 10.0.0.11 activate
!
address-family ipv4
no neighbor 10.0.0.11 activate
!
end

Also the configuration is relatively easy. There is one point specific to Arista EOS (at least it’s shown in the official config guide), which is activation of extended communities so that they are sent to neighbour. In Nokia (Alcatel-Lucent) or Cisco IOS XR or Cumulus Linux it’s done automatically, though in Cisco IOS you need to manually activate that.

Different vendors mean different default behaviour, just know what exactly needed technologywise and implement that.

For now we can check only the status of BGP peering for EVPN, as we haven’t advertised anything yet. In Arista EOS the command is the following:

1
2
3
4
5
6
.vEOS2#show bgp evpn summary
.BGP summary information for VRF default
.Router identifier 10.0.0.22, local AS number 65012
.Neighbor Status Codes: m - Under maintenance
.  Neighbor         V  AS           MsgRcvd   MsgSent  InQ OutQ  Up/Down State  PfxRcd PfxAcc
.  10.0.0.11        4  65011             22        23    0    0 00:06:55 Estab  0      0

In Nokia (Alcatel-Lucent) we just use the same command we have used previously for checking BGP session for IPv4 unicast address family, but this time we extend it with address family:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
.A:SR1# show router bgp summary family evpn
.===============================================================================
. BGP Router ID:10.0.0.11        AS:65011       Local AS:65011
.===============================================================================
.!
.! Output is ommited
.!
.===============================================================================
.BGP EVPN Summary
.===============================================================================
.Legend : D - Dynamic Neighbor
.===============================================================================
.Neighbor
.                   AS PktRcvd PktSent  InQ OutQ Up/Down   State|Recv/Actv/Sent
.-------------------------------------------------------------------------------
.10.0.0.22
.               65012       30      25    0    0 00h09m44s 0/0/0
.-------------------------------------------------------------------------------

At this point we consider that data centre fabric, both underlay and overlay, is fully ready and we can proceed with customer’s use cases.

Case #1. Switching within L2 domain (L2 option of EVPN with VXLAN)

We start out journey to EVPN/VXLAN use cases with building switching domain across two leafs, hence pure L2 use case with the following topology:

The configuration below is necessary to be applied at both our leaf switches by Nokia (Alcatel-Lucent) SR OS and Arista EOS:

SR1 – Nokia (Alcatel-Lucent) VSR vEOS2 – Arista EOS

A:SR1>edit-cfg# candidate view
=========================
configure
service
customer 2 create
description TEST_EVPN_TENNANT
exit
vpls 1000789 customer 2 create
vxlan vni 789 create
exit
bgp
route-distinguisher 10.0.0.11:789
route-target export target:65000:789 import target:65000:789
exit
bgp-evpn
vxlan
no shutdown
exit
mpls
shutdown
exit
exit
stp
shutdown
exit
sap 1/1/2:555 create
no shutdown
exit
proxy-arp
no shutdown
exit
proxy-nd
evpn-nd-advertise router
no shutdown
exit
no shutdown
exit
exit
exit
=========================

vEOS2#show run
! Command: show running-config
! device: vEOS2 (vEOS, EOS-4.20.1F)
!
vlan 666
name Tennant_L2
!
interface Vxlan1
vxlan source-interface Loopback0
vxlan udp-port 4789
vxlan vlan 666 vni 789
!
router bgp 65012
vlan 666
rd 10.0.0.22:789
route-target both 65000:789
redistribute learned
!
!
end

Here we can see at a glance two difference logic of the configuration. In Nokia (Alcatel-Lucent) SR OS all the configuration related to particular service, like here we speak about particular EVPN service, is contained without single configuration context. In Arista EOS, much like we’ve seen recently in Cumulus Linux (link), the configuration of EVPN service is splitted across VLAN, interface VXLAN, which contains both VTEP address and VNI to VLAN mapping, and BGP context, where we assign RD/RT and inform BGP to distribute learned MAC addresses.

On control plane we check the BGP RIB for EVPN. On Arista EOS we use the following command:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
.vEOS2#show bgp evpn vni 789
.BGP routing table information for VRF default
.Router identifier 10.0.0.22, local AS number 65012
.Route status codes: s - suppressed, * - valid, > - active, # - not installed, E - ECMP head, e - ECMP
.                    S - Stale, c - Contributing to ECMP, b - backup
.                    % - Pending BGP convergence
.Origin codes: i - IGP, e - EGP, ? - incomplete
.AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop
.
.        Network             Next Hop         Metric  LocPref Weight Path
. * >     RD: 10.0.0.11:789 mac-ip 0050.5623.b334
.                            10.0.0.11        -       100     0      65011 i
. * >     RD: 10.0.0.11:789 mac-ip 0265.ff00.033a
.                            10.0.0.11        -       100     0      65011 i
. * >     RD: 10.0.0.22:789 mac-ip faac.a614.0103
.                            -                -       -       0       i
. * >     RD: 10.0.0.11:789 imet 10.0.0.11
.                            10.0.0.11        -       100     0      65011 i
. * >     RD: 10.0.0.22:789 imet 10.0.0.22
.                            -                -       -       0       i

We see 5 routes: 2 of them are type 3 routes, which are announced by VTEPs and used for BUM traffic, 3 of them are host routes. Why do we have 3 type-2 routes? 2 of them, marked with bold are MAC addresses of our VMs, whereas third one comes into play-based on the fact we have enabled proxy-arp as this MAC relates to SR 7750 chassis:

1
2
.A:SR1# show chassis | match MAC
.  Base MAC address                  : 02:65:ff:00:00:00

At Nokia (Alcatel-Lucent) SR OS we can see BGP RIB for incoming routes and separately BGP-RIB-OUT for routes being sent to neighbors.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
.A:SR1# show router bgp routes evpn mac
.===============================================================================
. BGP Router ID:10.0.0.11        AS:65011       Local AS:65011
.===============================================================================
. Legend -
. Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
.                 l - leaked, x - stale, > - best, b - backup, p - purge
. Origin codes  : i - IGP, e - EGP, ? - incomplete
.
.===============================================================================
.BGP EVPN MAC Routes
.===============================================================================
.Flag  Route Dist.         MacAddr           ESI
.      Tag                 Mac Mobility      Label1
.                          Ip Address
.                          NextHop
.-------------------------------------------------------------------------------
.u*>i  10.0.0.22:789       fa:ac:a6:14:01:03 ESI-0
.      0                   Seq:0             VNI 789
.                          N/A
.                          10.0.0.22
.
.-------------------------------------------------------------------------------
.Routes : 1
.===============================================================================
.
.
.A:SR1# show router bgp neighbor 10.0.0.22 advertised-routes evpn mac
.===============================================================================
. BGP Router ID:10.0.0.11        AS:65011       Local AS:65011
.===============================================================================
. Legend -
. Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
.                 l - leaked, x - stale, > - best, b - backup, p - purge
. Origin codes  : i - IGP, e - EGP, ? - incomplete
.
.===============================================================================
.BGP EVPN MAC Routes
.===============================================================================
.Flag  Route Dist.         MacAddr           ESI
.      Tag                 Mac Mobility      Label1
.                          Ip Address
.                          NextHop
.-------------------------------------------------------------------------------
.i     10.0.0.11:789       00:50:56:23:b3:34 ESI-0
.      0                   Seq:0             VNI 789
.                          N/A
.                          10.0.0.11
.
.i     10.0.0.11:789       02:65:ff:00:03:3a ESI-0
.      0                   Static            VNI 789
.                          N/A
.                          10.0.0.11
.
.i     10.0.0.22:789       fa:ac:a6:14:01:03 ESI-0
.      0                   Seq:0             VNI 789
.                          N/A
.                          10.0.0.22
.
.-------------------------------------------------------------------------------
.Routes : 3
.===============================================================================

Control plane looks OK, what is good sign. Now we see into data plane to double check that. At Arista EOS we have two commands for that, which show a bit different but mainly similar information:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
.vEOS2#show mac address-table vlan 666
.          Mac Address Table
.------------------------------------------------------------------
.
.Vlan    Mac Address       Type        Ports      Moves   Last Move
.----    -----------       ----        -----      -----   ---------
. 666    0050.5623.b334    DYNAMIC     Vx1        1       0:02:34 ago
. 666    0265.ff00.033a    STATIC      Vx1
. 666    faac.a614.0103    DYNAMIC     Et2        1       0:04:06 ago
.Total Mac Addresses for this criterion: 3
.
.
.vEOS2#show vxlan address-table
.          Vxlan Mac Address Table
.----------------------------------------------------------------------
.
.VLAN  Mac Address     Type     Prt  VTEP             Moves   Last Move
.----  -----------     ----     ---  ----             -----   ---------
. 666  0050.5623.b334  EVPN     Vx1  10.0.0.11        1       0:02:54 ago
. 666  0265.ff00.033a  DYNAMIC  Vx1  10.0.0.11        1       0:02:54 ago
.Total Remote Mac Addresses for this criterion: 2

As Nokia (Alcatel-Lucent) follows service-oriented model, so we check the forwarding database per service:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
.A:SR1# show service id 1000789 fdb detail
.
.===============================================================================
.Forwarding Database, Service 1000789
.===============================================================================
.ServId    MAC               Source-Identifier        Type     Last Change
.                                                     Age
.-------------------------------------------------------------------------------
.1000789   00:50:56:23:b3:34 sap:1/1/2:555            L/135    05/13/18 09:54:58
.1000789   fa:ac:a6:14:01:03 vxlan:                   Evpn     05/13/18 09:56:25
.                            10.0.0.22:789
.-------------------------------------------------------------------------------
.No. of MAC Entries: 2
.-------------------------------------------------------------------------------
.Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static
.===============================================================================

The outputs from different commands looks nice and we need to do the last thing on this verification path

Case #1. Verification

As usual we test our data plane by issuing ping command between our VMs:

1
2
3
4
5
6
RP/0/0/CPU0:XR3#ping vrf VM5 192.168.2.2
Sat May 12 23:23:40.999 UTC
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.2.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 99/259/409 ms

Configuration of Cisco IOS XR to emulate VMs as well as packet capture is provided in one of the previous articles

Case #2. Inter-VXLAN routing (L3 option of EVPN with VXLAN – distributed gateway)

Going further, we deploy asymmetric IRB, or inter VXLAN routing, with the aim to deploy anycast gateway at both Nokia (Alcatel-Lucent) SR OS and Arista EOS leafs to provide possibility of two broadcast domains talk to each other. The image below shows broadcast domain 1:

And the next one image reflects domain 2:

The following configuration must be implemented:

SR1 – Nokia (Alcatel-Lucent) VSR vEOS2 – Arista EOS

A:SR1>edit-cfg# candidate view
=========================
configure
service
vpls 1000123 customer 2 create
allow-ip-int-bind
exit
vxlan vni 123 create
exit
bgp
route-distinguisher 10.0.0.11:123
route-target export target:65000:123 import target:65000:123
exit
bgp-evpn
vxlan
no shutdown
exit
mpls
shutdown
exit
exit
stp
shutdown
exit
sap 1/1/2:111 create
no shutdown
exit
proxy-arp
shutdown
exit
proxy-nd
shutdown
exit
service-name “L2_DOMAIN_1”
no shutdown
exit
vpls 1000456 customer 2 create
allow-ip-int-bind
exit
vxlan vni 456 create
exit
bgp
route-distinguisher 10.0.0.11:456
route-target export target:65000:456 import target:65000:456
exit
bgp-evpn
vxlan
no shutdown
exit
mpls
shutdown
exit
exit
stp
shutdown
exit
sap 1/1/2:333 create
no shutdown
exit
proxy-arp
shutdown
exit
proxy-nd
shutdown
exit
service-name “L2_DOMAIN_2”
no shutdown
exit
vprn 20000000 customer 2 create
route-distinguisher 10.0.0.11:2000
vrf-target export target:65011:456 import target:65012:456
interface “IRB_VXLAN1” create
address 192.168.0.253/24
vrrp 1 passive
backup 192.168.0.250
ping-reply
traceroute-reply
mac 00:00:5e:00:00:01
exit
local-proxy-arp
mac 00:20:00:01:23:01
vpls “L2_DOMAIN_1”
exit
exit
interface “IRB_VXLAN2” create
address 192.168.1.253/24
vrrp 2 passive
backup 192.168.1.250
ping-reply
traceroute-reply
mac 00:00:5e:00:00:01
exit
local-proxy-arp
mac 00:20:00:04:56:01
vpls “L2_DOMAIN_2”
exit
exit
no shutdown
exit
exit
exit
=========================

vEOS2#show run
! Command: show running-config
! device: vEOS2 (vEOS, EOS-4.20.1F)
!
vlan 222
name CUST_L3_DOMAIN_1
!
vlan 444
name CUST_L3_DOMAIN_2
!
vrf definition CUST
!
interface Ethernet2
load-interval 30
switchport trunk allowed vlan 222,444,666
switchport mode trunk
!
interface Vlan222
vrf forwarding CUST
ip proxy-arp
ip local-proxy-arp
ip address 192.168.0.254/24
ip virtual-router address 192.168.0.250
!
interface Vlan444
vrf forwarding CUST
ip proxy-arp
ip local-proxy-arp
ip address 192.168.1.254/24
ip virtual-router address 192.168.1.250
!
interface Vxlan1
vxlan source-interface Loopback0
vxlan udp-port 4789
vxlan vlan 222 vni 123
vxlan vlan 444 vni 456
vxlan vlan 666 vni 789
!
ip virtual-router mac-address 00:00:5e:00:00:01
!
ip routing vrf CUST
!
router bgp 65012
!
vlan 222
rd 10.0.0.22:123
route-target both 65000:123
redistribute learned
!
vlan 444
rd 10.0.0.22:456
route-target both 65000:456
redistribute learned
!
!
end

In Nokia (Alcatel-Lucent) SR OS we create two L2 domains within VPLS context and map them to L3 instance, which is called VPRN in Nokia (Alcatel-Lucent) SR OS.

More information about Nokia (Alcatel-Lucent) VPRN you will find in the dedicated article.

What is important, we deploy distributed anycast GW configuration, shared by @Jorge Rabadan, and you will see later on, that this configuration works interoperable with distributed anycast GW from Arista EOS! That’s really amazing!

In Arista EOS, comparing to configuration of L2 EVPN service, we add VLAN interfaces with IP addresses and virtual-address for anycast GW as well as corresponding VRF, where we map the customer. To make distributed anycast GW working we configure the same MAC address for virtual instances both at Nokia (Alcatel-Lucent) SR OS and Arista EOS leaf switches.

We begin our verifications traditionally with BGP RIB for EVPN:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
.vEOS2#show bgp evpn
.BGP routing table information for VRF default
.Router identifier 10.0.0.22, local AS number 65012
.Route status codes: s - suppressed, * - valid, > - active, # - not installed, E - ECMP head, e - ECMP
.                    S - Stale, c - Contributing to ECMP, b - backup
.                    % - Pending BGP convergence
.Origin codes: i - IGP, e - EGP, ? - incomplete
.AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop
.
.        Network             Next Hop         Metric  LocPref Weight Path
. * >     RD: 10.0.0.11:123 mac-ip 0000.5e00.0001 192.168.0.250
.                            10.0.0.11        -       100     0      65011 i
. * >     RD: 10.0.0.11:456 mac-ip 0000.5e00.0001 192.168.1.250
.                            10.0.0.11        -       100     0      65011 i
. * >     RD: 10.0.0.11:123 mac-ip 0020.0001.2301 192.168.0.253
.                            10.0.0.11        -       100     0      65011 i
. * >     RD: 10.0.0.11:456 mac-ip 0020.0004.5601 192.168.1.253
.                            10.0.0.11        -       100     0      65011 i
. * >     RD: 10.0.0.11:123 mac-ip 0050.5623.b334
.                            10.0.0.11        -       100     0      65011 i
. * >     RD: 10.0.0.11:456 mac-ip 0050.5623.b334
.                            10.0.0.11        -       100     0      65011 i
. * >     RD: 10.0.0.11:789 mac-ip 0050.5623.b334
.                            10.0.0.11        -       100     0      65011 i
. * >     RD: 10.0.0.11:789 mac-ip 0265.ff00.033a
.                            10.0.0.11        -       100     0      65011 i
. * >     RD: 10.0.0.22:123 mac-ip faac.a614.0103
.                            -                -       -       0       i
. * >     RD: 10.0.0.22:456 mac-ip faac.a614.0103
.                            -                -       -       0       i
. * >     RD: 10.0.0.22:123 mac-ip faac.a614.0103 192.168.0.2
.                            -                -       -       0       i
. * >     RD: 10.0.0.22:456 mac-ip faac.a614.0103 192.168.1.2
.                            -                -       -       0       i
. * >     RD: 10.0.0.11:123 imet 10.0.0.11
.                            10.0.0.11        -       100     0      65011 i
. * >     RD: 10.0.0.11:456 imet 10.0.0.11
.                            10.0.0.11        -       100     0      65011 i
. * >     RD: 10.0.0.11:789 imet 10.0.0.11
.                            10.0.0.11        -       100     0      65011 i
. * >     RD: 10.0.0.22:123 imet 10.0.0.22
.                            -                -       -       0       i
. * >     RD: 10.0.0.22:456 imet 10.0.0.22
.                            -                -       -       0       i
. * >     RD: 10.0.0.22:789 imet 10.0.0.22
.                            -                -       -       0       i

Based on route distinguisher we can understand, which routes are related to particular VNIs, though we can extend command, like “show bgp evpn vni xxx”, to see all the routes belonging to interested VRF.

In Nokia (Alcatel-Lucent) SR OS we check to BGP RIB for EVPN as well:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
.A:SR1# show router bgp routes evpn mac
.===============================================================================
. BGP Router ID:10.0.0.11        AS:65011       Local AS:65011
.===============================================================================
. Legend -
. Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
.                 l - leaked, x - stale, > - best, b - backup, p - purge
. Origin codes  : i - IGP, e - EGP, ? - incomplete
.
.===============================================================================
.BGP EVPN MAC Routes
.===============================================================================
.Flag  Route Dist.         MacAddr           ESI
.      Tag                 Mac Mobility      Label1
.                          Ip Address
.                          NextHop
.-------------------------------------------------------------------------------
.i     10.0.0.11:123       00:20:00:01:23:01 ESI-0
.      0                   Static            VNI 123
.                          192.168.0.253
.                          10.0.0.11
.
.i     10.0.0.11:456       00:20:00:04:56:01 ESI-0
.      0                   Static            VNI 456
.                          192.168.1.253
.                          10.0.0.11
.
.i     10.0.0.11:789       00:50:56:23:b3:34 ESI-0
.      0                   Seq:0             VNI 789
.                          N/A
.                          10.0.0.11
.
.i     10.0.0.11:789       02:65:ff:00:03:3a ESI-0
.      0                   Static            VNI 789
.                          N/A
.                          10.0.0.11
.
.u*>i  10.0.0.22:123       fa:ac:a6:14:01:03 ESI-0
.      0                   Seq:0             VNI 123
.                          192.168.0.2
.                          10.0.0.22
.
.u*>i  10.0.0.22:123       fa:ac:a6:14:01:03 ESI-0
.      0                   Seq:0             VNI 123
.                          N/A
.                          10.0.0.22
.
.u*>i  10.0.0.22:456       fa:ac:a6:14:01:03 ESI-0
.      0                   Seq:0             VNI 456
.                          192.168.1.2
.                          10.0.0.22
.
.u*>i  10.0.0.22:456       fa:ac:a6:14:01:03 ESI-0
.      0                   Seq:0             VNI 456
.                          N/A
.                          10.0.0.22
.
.-------------------------------------------------------------------------------
.Routes : 8
.===============================================================================

As you see, the amount of routes are less due to Nokia (Alcatel-Lucent) SR OS behaviour, we have explained previously,

We won’t check the MAC address table in this case, as we did in in the first case and it shew correct output. Here we’ll check two more things. The first one is related to ARP to check that all the tables are build correctly.

Nokia (Alcatel-Lucent) SR OS:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
.A:SR1# show router 20000000 arp
.===============================================================================
.ARP Table (Service: 20000000)
.===============================================================================
.IP Address      MAC Address       Expiry    Type   Interface
.-------------------------------------------------------------------------------
.192.168.0.1     00:50:56:23:b3:34 02h41m34s Dyn[I] IRB_VXLAN1
.192.168.0.250   00:00:5e:00:00:01 00h00m00s Oth[I] IRB_VXLAN1
.192.168.0.253   00:20:00:01:23:01 00h00m00s Oth[I] IRB_VXLAN1
.192.168.0.254   00:50:56:63:f3:eb 02h41m40s Dyn[I] IRB_VXLAN1
.192.168.1.1     00:50:56:23:b3:34 02h41m39s Dyn[I] IRB_VXLAN2
.192.168.1.250   00:00:5e:00:00:01 00h00m00s Oth[I] IRB_VXLAN2
.192.168.1.253   00:20:00:04:56:01 00h00m00s Oth[I] IRB_VXLAN2
.192.168.1.254   00:50:56:63:f3:eb 02h41m48s Dyn[I] IRB_VXLAN2
.-------------------------------------------------------------------------------
.No. of ARP Entries: 8
.===============================================================================

Arista EOS:

1
2
3
4
5
6
7
8
.vEOS2#show arp vrf CUST
.Address         Age (min)  Hardware Addr   Interface
.192.168.0.1           N/A  0050.5623.b334  Vlan222, Vxlan1
.192.168.0.2           N/A  faac.a614.0103  Vlan222, Ethernet2
.192.168.0.253           -  0020.0001.2301  Vlan222, Vxlan1
.192.168.1.1           N/A  0050.5623.b334  Vlan444, Vxlan1
.192.168.1.2           N/A  faac.a614.0103  Vlan444, Ethernet2
.192.168.1.253           -  0020.0004.5601  Vlan444, Vxlan1

And the last verification, before we jump to actual packets’ transmission, is to show the operation of component related to distributed anycast GW.

Arista EOS:

1
2
3
4
5
.vEOS2(config-if-Vl222)#show active
.interface Vlan222
.   vrf forwarding CUST
.   ip address 192.168.0.254/24
.   ip virtual-router address 192.168.0.250

Nokia (Alcatel-Lucent) SR OS:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
.A:SR1# show router 20000000 vrrp instance
.===============================================================================
.VRRP Instances
.===============================================================================
.Interface Name                   VR Id Own Adm  State       Base Pri   Msg Int
.                                 IP        Opr  Pol Id      InUse Pri  Inh Int
.-------------------------------------------------------------------------------
.IRB_VXLAN1                       1     No  Up   Master       100       1
.                                 IPv4      Up   n/a         100        No
.  Backup Addr: 192.168.0.250
.IRB_VXLAN2                       2     No  Up   Master       100       1
.                                 IPv4      Up   n/a         100        No
.  Backup Addr: 192.168.1.250
.-------------------------------------------------------------------------------
.Instances : 2
.===============================================================================

So far the outputs look fabulous and we proceed further.

Case #2. Verification

First of all, we check connectivity between all VMs within these two domains:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
RP/0/0/CPU0:XR3#ping vrf VM1 192.168.0.1
Sun May 13 12:24:52.494 UTC
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.0.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
!
!
RP/0/0/CPU0:XR3#ping vrf VM1 192.168.0.2
Sun May 13 12:24:54.923 UTC
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.0.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 159/289/409 ms
!
!
RP/0/0/CPU0:XR3#ping vrf VM1 192.168.1.1
Sun May 13 12:24:58.433 UTC
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/4/9 ms
!
!
RP/0/0/CPU0:XR3#ping vrf VM1 192.168.1.2
Sun May 13 12:24:59.553 UTC
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.1.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 249/327/399 ms

As you see, everything is working, what is a step forward comparing to interoperability tests between Nokia (Alcatel-Lucent) SR OS and Cumulus Linux, where we had some problems with ARP.

The most interesting case for me is that distributed anycast GW works into interoperable scenario. I’ve done two tests for that.

The first one is to ping VM3 from VM1 as both of them are connected to the same leaf switch SR1 and check what is going on in the wire:

1
2
3
4
5
6
RP/0/0/CPU0:XR3#ping vrf VM1 192.168.1.1
Sun May 13 12:54:30.912 UTC
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/9 ms

During this test we monitor with Wireshark all the links and see this traffic only on VLAN 111 and 333:

The second test touches VM2 and VM4, which are connected to vEOS2:

1
2
3
4
5
6
RP/0/0/CPU0:XR4#ping vrf VM2 192.168.1.2
Sun May 13 12:54:40.191 UTC
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.1.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/4/9 ms

And again Wireshark shows the traffic only on interconnected links, which are VLANs 222 and 444:

Trace for ICMP over VXLAN is provided in article about Cumulus Linux interoperability

Here are the final configuration files of this lab: 120_config_final_XR3 120_config_final_XR4 120_config_final_SR1 120_config_final_VEOS2

Lessons learned

It was my first time working with Arista EOS, same as earlier with Cumulus Linux, and my main observations are the next ones.

First, CLI in Arista EOS is quite user friendly, so if I have forgotten to enter critical commands, it informs about it:

1
2
3
4
5
6
.vEOS2(config)#router bgp 65012
.! IP routing not enabled
.vEOS2(config-router-bgp)#exit
.vEOS2(config)#ip routing
.vEOS2(config)#router bgp 65012
.vEOS2(config-router-bgp)#

Second. another nice feature in Arista EOS allows us to check the configuration in the exiting context, exactly how we do in Nokia (Alcatel-Lucent) SR OS by issuing “info”:

1
2
3
4
5
.vEOS2(config-if-Vl222)#show active
.interface Vlan222
.   vrf forwarding CUST
.   ip address 192.168.0.254/24
.   ip virtual-router address 192.168.0.250

Third, Arista also doesn’t share a lot of documentations, just as Nokia does. A lot of links I was trying to open at Arista website was telling me that I can’t access requested information as my level of the access is guest. In the time, when industry is moving into open-source networking, such approach looks weird for me.

Conclusion

Arista EOS is relatively new comparing to old giants like Nokia (merged with Alcatel-Lucent) or Cisco. And this is good, as they have great ideas and moving much quicker than mentioned companies. The same is relevant for Cumulus Linux, which is small and therefore moving forward at a good pace. But, what is more important for this particular article, we have reached the interoperability for all use cases between Nokia (Alcatel-Lucent) SR OS and Arista EOS, which looks very promising for data centre world. Take care and good bye!

P.S.

If you have further questions or you need help with your networks, I’m happy to assist you, just send me message. Also don’t forget to share the article on your social media, if you like it.

Support us





BR,

Anton Karneliuk

Useful links

During preparation of the article I have found very good article at Arista website about data centre fabric configuration, which I have used to study EVPN in Arista EOS.