Hello my friend,

We have discussed a lot how to make interoperable data center with Nokia (Alcatel-Lucent), Cumulus Networks and Arista. For routing between subnets (IRB) we have always used asymmetrical model. There is another one, which is called symmetrical IRB. Let’s review what is the difference and how to deploy it.

Brief description

Normally I don’t describe theory, as I assume you know everything and just keen to know how to connect different vendors. But today I need to explain the theory a bit, so that you understand the area of applicability for these technologies.

Asymmetrical IRB

We start with that we have already done for Nokia (Alcatel-Lucent) SR OS, Cumulus Linux and Arista EOS, meaning we start with the asymmetrical IRB.

Prerequisite: all VNIs are configured on all VTEPs for the tenant

Mode of operation: ingress VTEP performs routing and switching, egress VTEP performs only switching

Traffic flow: we take as an example the topology configured in the previous articles. Both VNI 123 and 456 are configured on both VTEPs (SR1 and SR2/VX2/vEOS2).

Assuming that our network is converged, let’s see what happens upon communication between VM1 (connected to VNI 123 on SR1) and VM4 (connected to VNI 456 at SR2/VX2/vEOS2) on the forward path:

• VM1 is going to send packet to VM4 with source IP 192.168.0.1 and destination IP 192.168.1.2
• As packet is destined to the host outside the original subnet, the packet is sent towards default gateway with IP 192.168.0.250, which is configured as distributed gateway across all date center leafs within this VNI 123. Source MAC is MAC of VM1 and destination MAC is virtual MAC of distributed gateway
• Leaf SR1 receives the packet and analyses where it should be sent
• SR1 checks that destination IP address is related to subnet on VNI 456 so that packet from VM1 to VM4 is to be sent over VNI 456
• SR1 checks the ARP for VM4 IP address to construct payload.
• SR1 sends packet to SR2:
  10. Outer IP: source 10.0.0.11, destination 10.0.0.22
  11. UDP: source 45687 (random), destination 4789
  12. VXLAN: VNI 456
  13. Inner MAC: source MAC of SR1 inside VNI 456, destination MAC VM4
  14. Inner IP: source 192.168.0.1, destination 192.168.1.2
• SR2 receives the packet from SR1, removes VXLAN header, checks the destination MAC and sends accordingly through the interface, where VM4 is attached
• VM4 receives the packet

On backward path we have the same sequence, with ingress leaf (SR2/VX2/vEOS2) performing routing:

• VM4 is going to send packet to VM1 with source IP 192.168.1.2 and destination IP 192.168.0.1
• As packet is destined to the host outside the original subnet, the packet is sent towards default gateway with IP 192.168.1.250, which is configured as distributed gateway across all date center leafs within this VNI 456. Source MAC is MAC of VM4 and destination MAC is virtual MAC of distributed gateway
• Leaf SR2 receives the packet and analyses where it should be sent
• SR2 checks that destination IP address is related to subnet on VNI 123 so that packet from VM4 to VM1 is to be sent over VNI 123
• SR2 checks the ARP for VM1 IP address to construct payload.
• SR2 sends packet to SR1:
  10. Outer IP: source 10.0.0.22, destination 10.0.0.11
  11. UDP: source 47891 (random), destination 4789
  12. VXLAN: VNI 123
  13. Inner MAC: source MAC of SR2 inside VNI 123, destination MAC VM1
  14. Inner IP: source 192.168.1.2, destination 192.168.0.1
• SR1 receives the packet from SR2, removes VXLAN header, checks the destination MAC and sends accordingly through the interface, where VM1 is attached
• VM1 receives the packet

Based on the described traffic flow you can easily spot, why this IRB mode is called asymmetrical.

Symmetrical IRB

This mode is the core topic for the current article. It differs from the previous one in several aspects. One of the key differentiator is that symmetrical IRB heavily relies on EVPN type-5 routes (IP prefix), that’s why proper redistribution of IP subnets between different address families in BGP should be done.

Prerequisite: Only one VNI (called L3 VNI) should be configured at all VTEPs.

Mode of operation: Both ingress and egress VTEPs perform switching and routing functions

Traffic flow: We take the same topology and the same VMs as in the previous example, but we modify the structure of VNIs in the following way:

• Leaf 1 has VNI 123 configured with subnet 192.168.0.0/24 and VNI 4000 as L3 VNI
• Leaf 2 has VNI 456 configured with subnet 192.168.1.0/24 and VNI 4000 as L3 VNI

In general, we have the following topology for our tenant:

On the forward path we have the following sequence of actions:

• VM1 sends packet to VM4 (IP addresses 192.168.0.1 -> 192.168.1.2)
• As IP address of VM4 lays out of the source subnet, the packets are destined to the distributed anycast gateway (leaf 1 is closest gateway), hence the destination MAC is vMAC for GW in VNI 123, whereas source MAC is MAC of VM1
• Leaf 1 receives the packet from VM1 and evaluates the destination IP address, which is learned as EVPN type-5 route and installed in routing table pointing to VNI 4000 and mac of Leaf 2 as a next-hop
• Leaf 1 sends the packet to Leaf 2 with the following structure:
  10. Outer IP: source 10.0.0.11, destination 10.0.0.22
  11. UDP: source 45687 (random), destination 4789
  12. VXLAN: VNI 4000
  13. Inner MAC: source MAC of Leaf1 inside VNI 4000, destination MAC of Leaf2 inside VNI 4000
  14. Inner IP: source 192.168.0.1, destination 192.168.1.2
• Leaf2 receives the packet form Leaf1, removes VXLAN encapsulation and inner MAC header.
• Leaf2 performs routing lookup and finds that VM4 is connected subnet in VLAN 444.
• Leaf2 performs ARP lookup to find corresponding MAC address of VM4 and sends packet to VM2 with source MAC of Leaf2 in VNI 456 and destination MAC of VM4
• VM4 receives the packet

I won’t specify the backwards path as it is completely the same: both Leaf1 and Leaf2 performs routing and switching lookup.

What we are going to test?

For the symmetrical IRB I haven’t mentioned the names of the vendors. Arista and Cumulus supports it, so I will show example of symmetrical IRB based on them. Cisco Nexus supports it as well, but I don’t have VM with Cisco Nexus.

[UPD] Nokia didn’t support L3 VNI (interface-less mode) up to latest SR OS 16.0.R1. BUT now Nokia supports L3 VNI, hurray! Thanks a lot to Nokia PLM Jorge Rabadan, who has updated me with this info. As I got this info after the whole article is ready, I will add Nokia configuration as a last chapter after verification on Arista/Cumulus.

Software version

The following infrastructure is used in my lab:

• CentOS 7 with python 2.7.
• Ansible 2.5.2
• Arista EOS 4.20.5F
• [NEW] Cumulus Linux VX 3.6.1
• [NEW] Nokia (Alcatel-Lucent) SR OS 16.0.R1

See the previous article to get details how to build the lab

Topology

The physical topology is more or less stable over the last tests, so there is no surprise for you:

The logical topology is also familiar to you, as we have used it in all the articles about data center technologies:

The major difference comparing to previous articles is that spine switches, which were previously Cisco IOS XR based, I replaced to Cumulus Linux VX and Arista vEOS. The reason for that is very simple: Cumulus Linux VX (or Arista vEOS) uses less resources than Cisco IOS XRv, whereas the simple functionality of spine switches (IP routing. BGP peering, client emulation (VRF)) is available in all vendors.

Initial configuration files for this lab: 126_config_initial_vEOS1 126_config_initial_vEOS3 126_config_initial_VX2 126_config_initial_VX4

There is no Linux file, as I’m using only VMWare based VMs in this case. For Nokia (Alcatel-Lucent) VSR there will be some additions the topology.

IP Fabric. BGP for underlay and overlay

Here we have some changes comparing to the previous topology. The difference is related to Cisco IOS XRv limitations: it doesn’t understand VXLAN in BGP updates, that’s why it can’t process routes coming from leafs. That’s why previously we have BGP peering for EVPN between Leafs as eBGP-multihop session. Now both Arista vEOS and Cumulus Linux understand such routes, so we can build reference architecture with single hop eBGP session on interface level both for IPv4 unicast and L2VPN EVPN address families:

Here is the configuration necessary for that

As you see configuration is relatively simple: we just have 2 BGP single-hop eBGP peering from each node: each Leaf is connected to two Spines, and each Spine is connected to two (all leafs). BFD is used to speedup convergence, what is particularly useful in virtual environment. Point out that this time both Spines are in the same AS, so they won’t see each other IP addresses, what is fully OK for such deployment.

There is one point that might be needed for configuration, which I haven’t configured. Normally next-hop for routes are changed by each eBGP peer in order to maintain reachability between areas. On the other hand we need to preserve originally next-hop, which is typically loopback at VTEP for EVPN routes. In some configuration guides it’s recommended to configure “next-hop-unchanged” in EVPN address-family at Spines. My observation in this lab shows that it isn’t necessary and next-hop is preserved automatically and no additional configuration is necessary. Probably, it depends on OS version, but keep in mind that it might be necessary.

The basic test is to check the reachability between loopbacks at vEOS1 and VX2, which will be later on VTEP IP addresses:

The data centre fabric is ready and we can start with building our service on top.

Symmetric IRB configuration / Arista and Cumulus

Earlier we have shown the configuration of L2 VNIs both for Cumulus Linux  and Arista EOS, that’s here we will focus on the L3 VNI solely. The following service topology we are going to implement:

And to make this service working, we need to configure both Leafs as shown below (no changes at Spines are necessary):

Configuration is quite similar across both vendors with some additions at Cumulus Linux.

In Arista EOS we create VRF, VLAN in database and corresponding VLAN interface with IP and vIP to connect VM7. Then we create VXLAN interface, where we define VTEP IP and L3 VNI ID. L3 VNI is defined as “vxlan vrf X vni Y”, what is different to L2 VNI “vxlan vlan X vni Y”, what we have configured previously for asymmetrical IRB. Under BGP context we create VRF context, where we define RD/RT and instruct it to redistribute connected IP prefixes.

For Cumulus Linux we do the same activities, plus we create additional VLAN 4000, which is purely internal to VX2 and is used to terminate L3 VNI with certain MAC in corresponding VRF. In Cumulus Linux we also explicitly configure EVPN address family also within VRF and instruct it to advertise IPv4 prefixes, whereas under IPv4 unicast address family within VRF we redistribute connected routes.

To be honest, it took me some time to collect and test this configuration, as there for Cumulus Linux it was realised recently and for Arista there are a lot of configuration unavailable without corresponding account (coupled with contract). I put some links in the end of this post, where you can learn more about L3 VNI configuration.

Control plane verification / Arista and Cumulus

Before we start testing the data plane, let’s check the control plane, how does different tables look like. First of all let’s verify that we have L3 VNIs. We start with Arista EOS:

The L3 VNI is shown under “VRF to VNI mapping”. For Cumulus Linux we have different command:

The next very useful table is BGP EVPN RIB. Here is the filtered output for Cumulus Linux based leaf switch VX2 (we filter only related EVPN type-5 routes):

On the Arista vEOS we can use either the same filtering or we can show the routes related to this VNI :

In Cumulus Linux filtering routes against VNI works for L2 VNI and not for L3 VNI, at least in VX 3.6.1.

Take a look at next hops: we haven’t specified them anywhere, but they are set correctly. I assume they are automatically copied from “VTEP source IP”. But you remember, that in the very beginning I mentioned something about MAC addresses within L3 VNI. They are transferred as extended community attribures, let’s take a look (the output will be the same on both Arista EOS and Cumulus Linux):

Finally let’s check the routing table for the tenant’s VRF. But we will do it together with the verification of BGP RIB for this VRF as well. At Cumulus Linux it looks like:

In Arista EOS the syntax is very similar to Cisco IOS (or NX-OS), what we have mentioned previously, so the command is also easy:

The control plane looks nice, so we go further with our checks.

Data plane verification / Arista and Cumulus

The data plane verification is quite easy. As usually we just use ping to generate ICMP stream between VM7 and VM8. But beforehand we need to create the emulations of such VMs using VRFs at Spine switches:

The VMs emulated using basic VRF-lite approach, so no further actions are needed. Let’s test how it works:

As both Cumulus Linux and Arista EOS has Linux as basis, we can check the packet flow using TCP dump (the command will be the same for Cumulus and for Arista if launch it from bash):

Output is reduced just for 2 packets (one input, one output).

It works!

Final configuration files for this lab for Cumulus Linux and Arista EOS: 126_config_final_vEOS3 126_config_final_vEOS1 126_config_final_VX4 126_config_final_VX2

Configuration of L3 VNI in Nokia (Alcatel-Lucent) SR OS

As we’ve highlighted in the very beginning, support of interface-less was added in the latest Nokia (Alcatel-Lucent) SR OS release, which is SR OS 16.0.R1 by the time of writing and which was released less than a month ago. That’s why I put Nokia snippet here separately, though my tests shown full interoperability with Cumulus Linux and Arista EOS.

In the example below, I’ve replaced Arista EOS based leaf vEOS1 with Nokia SR OS based leaf SR1. The reason is limited amount of RAM at my laptop, and Cumulus Linux VX utilizes less resources than Arista vEOS:

By the way, the tests with Arista EOS was done as well, just in reduced topology without spine switches, with connectivity of leaf switches back to back.

The initial configuration for Nokia SR1 is here: 126_config_initial_SR1

Assuming the rest of the topology and configuration done so far isn’t changed we have the following.

#1. BGP fabric for underlay and overlay

Remember that we are using Nokia (Alcatel-Lucent) SR OS 16.0.R1. So we use advantage of MD-CLI, and config is provided in YANG format. Pay attention that structure of BGP config is CHANGED!

Here, in the Nokia configuration there is one important point, what was so far not relevant for Arista EOS and Cumulus Linux. We need to modify the source of the BGP updates for EVPN address-family. In Cumulus Linux and Arista EOS VTEP IP address is used by default, whereas for Nokia (Alcatel-Lucent) SR OS the source IP address of BGP session. That’s why we have extended route policy, comparing to previous examples. Additionally, we need to enable “vpn-apply-export” under BGP group in oder route policy is applied also for all VPN address-families, what is not the case by default.

As I pointed previously, the BGP structure has changed:

• Up to Nokia SR OS 16.0.R1 BGP neighbours (peers) were configured within the group context
• From Nokia SR OS 16.0.R1 BGP neighbours are configured in general BGP context, whereas group is an parameter within neighbour context

I think this approach is quite good (and you can find it is similar to something else).

#2. L3 VNI in Nokia SR OS

Here we have new structure due to SR OS 16.0.R1 again

The key change is that by the command “vpls x” we configured not the service with id “x”, but rather the service with name “x”. So we don’t need (actually there is no such command anymore) to configure “service-name Y” within service context. To make interface-less L3 VNI, which is interoperable with other vendors, we disable “mac-ip” advertisement and enable “ip-prefix” advertisement under “bgp-evpn routes” context. Then under routing context “vprn 7” we create interface “L3_VNI”, which is just arbitrary name< and configure there vpls with our name “104000” and “evpn-tunnel” option enabled. Point out that we don’t configure any EVPN parameters under “vpls 2”, what in fact is just local BVI interface, which is mapped to IP 192.168.7.253 inside “vprn 7” context.

#3. Verification for L3 VNI in Nokia

This time we start the verification with data plane. We test connectivity from VM8 (emulated by VRF at Cumulus Linux VX4) to VM7 (emulated by VRF at Arista vEOS3):

As we have shown the packets on the wire previously, we do the same for this test. The “tcpdump” is performed at Cumulus Linux based leaf switch VX2:

As data plane is fully OK, it means that control plane is OK as well, but let’s review some outputs.

Here is the output of the learned EVPN routes (both type-2 MAC-IP and type-5 IP-PREFIX):

As you see, there is no single type-2 (MAC-IP) routes learned or advertised, so all customer routes are only level-5 (IP-PREFIX).

In routing table (as well as in FIB) we see that next hop to this route is pointed to MAC:

And in order for the lookup to be successful, this MAC must be reachable in VNI:

This MAC is extracted also from IP-PREFIX (EVPN type-5) route from the field “GW address”.

The final configuration for Nokia (Alcatel-Lucent) SR OS you can find here: 126_config_final_SR1

Lessons learned

Frankly speaking, by default the traffic flow from the example above won’t work. The reason for that is some caveats in data plane implementation in Arista vEOS, which is not relevant for hardware or even vEOS-Router. Actually usage of tcpdump helped me to find that issue.

I hope it will be fixed in the upcoming release of Arista vEOS

Greetings and acknowledges

Many thanks for Michael Amstelveen and Alex Nichol from Arista for answering my questions and sharing valuable tips on Arista configuration. Thanks for Pete Crocker from Cumulus Networks for pointing to useful information about Cumulus Linux. And special thanks for Jorge Rabadan from Nokia for updating on capabilities of Nokia SR OS 16.0.R1 to have interface-less mode and constantly providing feedback on features supported in Nokia SR OS.

Conclusion

Though data centres based on BGP-EVPN control plane and VXLAN data plane are already some time presents on market, this technology is being actively developed. Such type of L3 VNI is described in the one of the BESS draft. I believe that further development of RFC and standardization of features being draft now will make data centre world even more interoperable.

Support us

Support new interop and automation articles at karneliuk.com
EUR

I want to support with:
9.99 EUR4.99 EUR24.99 EUR49.99 EUR99.99 EUR199.99 EUR

P.S.

If you have further questions or you need help with your networks, I’m happy to assist you, just send me message (https://karneliuk.com/contact/). Also don’t forget to share the article on your social media, if you like it.

BR,

Anton Karneliuk

Useful links

Configuration of EVPN-VXLAN including L3VNI in Cumulus Linux

Configuration of EVPN-VXLAN for interface-less mode in Nokia (Alcatel-Lucent) SR OS

For Arista I haven’t found similar doc (L3 VNI config) doc in publically available.