Hello my friend,

with this article we’ll close our series of EVPN/VXLAN articles, we started some time ago with pure switching (L2) and distributed gateway (L3) for EVPN/VXLAN. In present article we’ll talk about so called L3 OUT functionality, where customer within tenant should speak dynamic routing with the leaf switch.

Disclaimer

Start with the L2 configuration and L3 configuration for EVPN/VXLAN to get full understanding of what is going on here.

Brief description

Related use case, where we need to make dynamic routing between customer VNF in tenant and our DC fabric, isn’t quite common. It isn’t standard web-based application or something similar. But it’s still valid. For example, distributed load balancer, which distributes customer sessions across lots of frontends, or distributed NAT/FW application, which pass the customer traffic between security zones. From example above we can distinguish two important capabilities of such flows:

  • They are statefull. So traffic flow should pass only certain device in both directions.
  • System, which traffic pass, is distributed. Probably, if you have big HW-load balancer, you might not need to have routing protocols, but of you used set of VNFs across different racks, you will need to know, which exactly VNF the traffic has pass in one direction in order to send it back through the same VNF

What we are going to test?

In order to show, how BGP peering is working between VM and Leaf, we’ll announce loopback from each VM to the cloud. The success criteria for the lab is possibility to reach any VM’s loopback from any another VM’s loopback.

Software version

The following infrastructure is used in my lab:

  • CentOS 7 with python 2.7.
  • Ansible 2.4.2
  • Nokia (Alcatel-Lucent) SR OS 14.0.R4
  • Cisco IOS XRv 6.1.2

See the previous article to get details how to build the lab

Topology

The physical topology is standard for our labs:

Logical topology is the same, as we have in previous lab. But as we are touching only service part, only respective servies will be shown. Service#1 looks like:

Service #2 has the following topology:

Initial configuration files in this lab are the finish one, from the previous one

Configuration eBGP PE-CE peering between Nokia (Alcatel-Lucent) SR OS and Cisco IOS XR

From the point of view of the activities, we need to do the following stuff:

  • Enable propagation of EVPN type-5 routes
  • Configure BGP and policies on Leaf within customer VPN
  • Configure BGP and policies at VM side

Let’s view in details, what we configure per vendor

#1. Configuration of leaf switches // Nokia (Alcatel-Lucent) SR OS

Pay attention that only incremental configuration is shown here. The rest of the configuration you can find in Part 1 and Part 2

Here we go:

SR1 SR2

A:SR1>edit-cfg# candidate view
=========================
configure
router
policy-option
begin
prefix-list “PL_CE”
prefix 192.168.0.0/16 prefix-length-range 24-24
exit
policy-statement “RP_BGP_CE_OUT”
entry 10
from
prefix-list “PL_CE”
exit
action accept
exit
exit
default-action drop
exit
exit
commit
exit
exit
service
vpls 10000123 customer 2 create
bgp-evpn
ip-route-advertisement incl-host
exit
exit
vpls 10000456 customer 2 create
bgp-evpn
ip-route-advertisement incl-host
exit
exit
vprn 20000000 customer 2 create
autonomous-system 65011
bgp
preference 165
group “CE”
family ipv4
export “RP_BGP_CE_OUT”
neighbor 192.168.0.3
peer-as 65111
exit
neighbor 192.168.1.3
peer-as 65112
exit
exit
no shutdown
exit
exit
exit
exit
=========================

A:SR2>edit-cfg# candidate view
=========================
configure
router
policy-option
begin
prefix-list “PL_CE”
prefix 192.168.0.0/16 prefix-length-range 24-24
exit
policy-statement “RP_BGP_CE_OUT”
entry 10
from
prefix-list “PL_CE”
exit
action accept
exit
exit
default-action drop
exit
exit
commit
exit
exit
service
vpls 10000123 customer 2 create
bgp-evpn
ip-route-advertisement incl-host
exit
exit
vpls 10000456 customer 2 create
bgp-evpn
ip-route-advertisement incl-host
exit
exit
vprn 20000000 customer 2 create
autonomous-system 65012
bgp
preference 165
group “CE”
family ipv4
export “RP_BGP_CE_OUT”
neighbor 192.168.0.4
peer-as 65211
exit
neighbor 192.168.1.4
peer-as 65212
exit
exit
no shutdown
exit
exit
exit
exit
=========================

In the VPLS part we need to enable type 5 routes of EVPN. Though we did it previously, I have highlighted it here once more.

Then we configure BGP for PE-CE peering. In order to advertise routes, learned from EVPN, we need to allow them in the export policy. There are several ways, how we can achieve that. Here we take just the option to control export by allowing certain prefixes in prefix lists.

The third important point is changing of preference (administrative distance – AD – in Cisco words). The reason, why we need to do that, is that by default AD of BGP routes is 170 and 169 for BGP EVPN. So, routes are getting in kind of loop, that Leaf switches shows to another Leaf, as learned back routes have better priority. When we change AD so that locally learned BGP routes are more preferable, than the loop is broken and routing works properly.

Let’s review configuration of Cisco side now.

#2. Configuration of VMs // Cisco IOS XR

Here the configuration is quite trivial:

Cisco IOS XR (XR3) Cisco IOS XR (XR4)

RP/0/0/CPU0:XR3(config-bgp-vrf-nbr-af)#show conf
!
interface Loopback111
vrf VM1
ipv4 address 192.168.111.1 255.255.255.0
!
interface Loopback112
vrf VM3
ipv4 address 192.168.112.1 255.255.255.0
!
router bgp 65001
address-family vpnv4 unicast
!
vrf VM1
rd 65001:65111
address-family ipv4 unicast
network 192.168.111.0/24
!
neighbor 192.168.0.253
remote-as 65011
local-as 65111 no-prepend replace-as
address-family ipv4 unicast
route-policy RP_PASS_ALL in
route-policy RP_PASS_ALL out
!
!
!
vrf VM3
rd 65001:65112
address-family ipv4 unicast
network 192.168.112.0/24
!
neighbor 192.168.1.253
remote-as 65011
local-as 65112 no-prepend replace-as
address-family ipv4 unicast
route-policy RP_PASS_ALL in
route-policy RP_PASS_ALL out
!
!
!
!
end

RP/0/0/CPU0:XR4(config-bgp-vrf-nbr-af)#show conf
!
interface Loopback211
vrf VM2
ipv4 address 192.168.211.1 255.255.255.0
!
interface Loopback212
vrf VM4
ipv4 address 192.168.212.1 255.255.255.0
!
router bgp 65002
address-family vpnv4 unicast
!
vrf VM2
rd 65002:65211
address-family ipv4 unicast
network 192.168.211.0/24
!
neighbor 192.168.0.254
remote-as 65012
local-as 65211 no-prepend replace-as
address-family ipv4 unicast
route-policy RP_PASS_ALL in
route-policy RP_PASS_ALL out
!
!
!
vrf VM4
rd 65002:65212
address-family ipv4 unicast
network 192.168.212.0/24
!
neighbor 192.168.1.254
remote-as 65012
local-as 65212 no-prepend replace-as
address-family ipv4 unicast
route-policy RP_PASS_ALL in
route-policy RP_PASS_ALL out
!
!
!
!
end

The only point that is worth mentioning is changes of local-as in order to simulate different customers.

See more details on BGP configuration in the corresponding article.

Verification

We start verification with routing tables. On SR1, which is Nokia (Alcatel-Lucent) SR OS based data centre leaf, we see the following picture:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
A:SR1# show router 20000000 route-table
===============================================================================
Route Table (Service: 20000000)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
Next Hop[Interface Name]                                    Metric
-------------------------------------------------------------------------------
192.168.0.0/24                                Local   Local     01h41m57s  0
IRB_VXLAN1                                                   0
192.168.0.254/32                              Remote  BGP EVPN  01h41m57s  169
192.168.1.254                                                0
192.168.1.0/24                                Local   Local     01h41m57s  0
IRB_VXLAN2                                                   0
192.168.1.254/32                              Remote  BGP EVPN  01h41m57s  169
192.168.0.254                                                0
192.168.111.0/24                              Remote  BGP       00h37m52s  165
192.168.0.3                                                  0
192.168.112.0/24                              Remote  BGP       00h38m03s  165
192.168.1.3                                                  0
192.168.211.0/24                              Remote  BGP EVPN  00h48m23s  169
192.168.1.254                                                0
192.168.212.0/24                              Remote  BGP EVPN  00h48m23s  169
192.168.0.254                                                0
-------------------------------------------------------------------------------
No. of Routes: 8
Flags: n = Number of times nexthop is repeated
B = BGP backup route available
L = LFA nexthop available
S = Sticky ECMP requested
===============================================================================

As we have changed the preference for routing table, you see the correct routing here. On connected VM, which is Cisco IOS XR router we see:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
RP/0/0/CPU0:XR3#show route vrf VM1
.
Codes: C - connected, S - static, R - RIP, B - BGP, (>) - Diversion path
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP
i - ISIS, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, su - IS-IS summary null, * - candidate default
U - per-user static route, o - ODR, L - local, G  - DAGR, l - LISP
A - access/subscriber, a - Application route
M - mobile route, r - RPL, (!) - FRR Backup path
.
Gateway of last resort is 192.168.0.253 to network 0.0.0.0
.
S* 0.0.0.0/0 [1/0] via 192.168.0.253, 01:43:03
C 192.168.0.0/24 is directly connected, 01:46:50, GigabitEthernet0/0/0/0.111
L 192.168.0.3/32 is directly connected, 01:46:50, GigabitEthernet0/0/0/0.111
B 192.168.1.0/24 [20/0] via 192.168.0.253, 00:32:18
C 192.168.111.0/24 is directly connected, 01:01:53, Loopback111
L 192.168.111.1/32 is directly connected, 01:01:53, Loopback111
B 192.168.112.0/24 [20/0] via 192.168.0.253, 00:38:54
B 192.168.211.0/24 [20/0] via 192.168.0.253, 00:32:18
B 192.168.212.0/24 [20/0] via 192.168.0.253, 00:32:18

Of the particular interest would be the BGP EVPN RIB on Nokia (Alcatel-Lucent) SR OS devices:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
A:SR1# show router bgp routes evpn ip-prefix
===============================================================================
BGP Router ID:10.0.0.11        AS:65011       Local AS:65011
===============================================================================
Legend -
Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
.               l - leaked, x - stale, > - best, b - backup, p - purge
Origin codes  : i - IGP, e - EGP, ? - incomplete
===============================================================================
BGP EVPN IP-Prefix Routes
===============================================================================
Flag  Route Dist.         Prefix
.     Tag                 Gw Address
.                         NextHop
.                         Label
-------------------------------------------------------------------------------
u*>i  10.0.0.22:123       192.168.0.253/32
.     0                   192.168.0.254
.                         10.0.0.22
.                         VNI 123
.
u*>i  10.0.0.22:123       192.168.1.254/32
.     0                   192.168.0.254
.                         10.0.0.22
.                         VNI 123
.
u*>i  10.0.0.22:123       192.168.1.0/24
.     0                   192.168.0.254
.                         10.0.0.22
.                         VNI 123
.
u*>i  10.0.0.22:123       192.168.111.0/24
.     0                   192.168.0.254
.                         10.0.0.22
.                         VNI 123
.
u*>i  10.0.0.22:123       192.168.212.0/24
.     0                   192.168.0.254
.                         10.0.0.22
.                         VNI 123
.
u*>i  10.0.0.22:456       192.168.0.254/32
.     0                   192.168.1.254
.                         10.0.0.22
.                         VNI 456
.
u*>i  10.0.0.22:456       192.168.1.253/32
.     0                   192.168.1.254
.                         10.0.0.22
.                         VNI 456
.
u*>i  10.0.0.22:456       192.168.0.0/24
.     0                   192.168.1.254
.                         10.0.0.22
.                         VNI 456
.
u*>i  10.0.0.22:456       192.168.112.0/24
.     0                   192.168.1.254
.                         10.0.0.22
.                         VNI 456
.
u*>i  10.0.0.22:456       192.168.211.0/24
.     0                   192.168.1.254
.                         10.0.0.22
.                         VNI 456
-------------------------------------------------------------------------------
Routes : 10
===============================================================================

You might see in this table, that best IP route within EVPN still points to another leaf, though routing table is correct. In this table you see all the related routes.

At Cisco IOS XR side, which is VM2, there is no something special:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
RP/0/0/CPU0:XR3#show bgp vpnv4 unicast
Sun Mar 18 19:47:52.254 UTC
BGP router identifier 10.0.0.33, local AS number 65001
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0x0   RD version: 0
BGP main routing table version 21
BGP NSR Initial initsync version 7 (Reached)
BGP NSR/ISSU Sync-Group versions 0/0
BGP scan interval 60 secs
.
Status codes: s suppressed, d damped, h history, * valid, > best
i - internal, r RIB-failure, S stale, N Nexthop-discard
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 65001:65111 (default for vrf VM1)
*> 192.168.0.0/24 192.168.0.253 0 65011 i
*> 192.168.1.0/24 192.168.0.253 0 65011 i
*> 192.168.111.0/24 0.0.0.0 0 32768 i
*> 192.168.112.0/24 192.168.0.253 0 65011 65112 i
*> 192.168.211.0/24 192.168.0.253 0 0 65011 i
*> 192.168.212.0/24 192.168.0.253 0 0 65011 i
Route Distinguisher: 65001:65112 (default for vrf VM3)
*> 192.168.0.0/24 192.168.1.253 0 65011 i
*> 192.168.1.0/24 192.168.1.253 0 65011 i
*> 192.168.111.0/24 192.168.1.253 0 65011 65111 i
*> 192.168.112.0/24 0.0.0.0 0 32768 i
*> 192.168.211.0/24 192.168.1.253 0 0 65011 i
*> 192.168.212.0/24 192.168.1.253 0 0 65011 i
Processed 12 prefixes, 12 paths

The final part of our verifications, as we have discussed so far, is ping between loopbacks.

So.. Ping time!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
RP/0/0/CPU0:XR4#ping vrf VM2 192.168.111.1 so lo211
Sun Mar 18 19:49:06.779 UTC
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.111.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/7/9 ms
RP/0/0/CPU0:XR4#ping vrf VM2 192.168.112.1 so lo211
Sun Mar 18 19:49:10.449 UTC
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.112.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/7/9 ms
RP/0/0/CPU0:XR4#ping vrf VM2 192.168.212.1 so lo211
Sun Mar 18 19:49:13.918 UTC
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.212.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/4/9 ms
RP/0/0/CPU0:XR4#ping vrf VM2 192.168.211.1 so lo211
Sun Mar 18 19:49:17.048 UTC
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.211.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

Final configuration files: 114_config_final_linux 114_config_final_SR1 114_config_final_SR2 114_config_final_XR3 114_config_final_XR4

Ansible playbook to reach that state from initial configurations:

Lessons learned

What goes around, comes around. I mean that toolset, regardless of technologies is always the same. For example, preference or administrative distance. This tool is very old, but we suddenly it comes back into game, when we start speaking about EVPN as replacement for traditional IP VPNs. So, don’t forget old school tools, even if you deal with the mega modern technologies

Conclusion

As you have seen over a couple of last articles, EVPN can do all things, like L2 VPNs, L3 VPNs and even incorporate PE-CE routing. By the end of the day, it means that this particular address-family can replace all other BGP signalling flavours. Especially, when we take into consideration, that it can works both with MPLS and VXLAN data plane, EVPN is really powerful. Take care and good bye!

P.S.

If you have further questions or you need help with your networks, I’m happy to assist you, just send me message. Also don’t forget to share the article on your social media, if you like it.

Support us





BR,

Anton Karneliuk