PBB-EVPN in Nokia (Alcatel-Lucent) SR OS and Cisco IOS XR

Anton Karneliuk

8 years ago

Hello my friend,

Today we’ll review implementation of the most advanced, scalable, flexible (and the most complex) L2VPN that exists today in networking world. It’s called EVPN (Ethernet VPN). Let’s take a look what is it and how to configure it in mixed Nokia (Alcatel-Lucent) SR OS and Cisco IOS XR environment.

You can find detailed explanation of EVPN operation in book “MPLS in the SDN Era”. There also some good documents created by Nokia and Cisco.

Brief overview

You can think about EVPN as about advanced version of VPLS, though it’s completely different technology with different rules that solves the same task that is multipoint L2VPN. There are several advantages that significantly differs EVPN from VPLS. Here are two the most important from my point of view:

MAC learning. In EVPN MAC learning is done via BGP (control-plane MAC learning), whereas in VPLS MAC learning is done by looking into source MAC address in Ethernet frame (data-plane MAC learning). The first option is much scalable and quicker in terms of convergence, what perfectly matches need of virtual machines (VMs) migration inside data center or between them.
L2/L3 traffic. In EVPN you can configure the same IP and MAC addresses at all PE routers therefore you don’t need any FHRP (like HSRP or VRRP) in order to implement redundancy for VMs inside VPLS. For VMs themselves it means that ARP cache will be valid after VM migration to another location with another PE router.

There are more advantages, but even solely these two differences makes EVPN perfect solution for data-centers and data center interconnects. If we speaking about Service Provider world, then EVPN can be used in the same areas, where VPLS is used. But Cisco IOS XR doesn’t support pure EVPN, unfortunately. In Cisco world only NX-OS (platform used for Cisco Nexus Switches) supports pure EVPN. So what we can do? We can implement PBB-EVPN then, which is supported both by Nokia (Alcatel-Lucent) SR OS and Cisco IOS XR.

PBB (provider backbone bridging) is another L2VPN technology that is called also MAC-in-MAC (802.1ah) encapsulation, because PEs add additional Ethernet header with its own MAC addresses as source and peering PE as destination. It uses concept of I-SID for customer’s demultiplexion, what has the same meaning as VLAN tag in 802.1Q frame. The main advantage of PBB over dot1Q or Q-in-Q is that in PBB the customer MAC addresses are hidden from service provider (SP) core network, therefore less resources are used and more scalability in SP network is achieved. Moreover it’s absolutely OK to transport Q-in-Q traffic over PBB so you can provide the most flexible L2 services for your clients.

The overall architecture of PBB-EVPN over MPLS looks as follows:

The main differences between PBB-EVPN and “ordinary” EVPN is that in PBB-EVPN BGP is used for MAC learning of Provider’s MAC addresses (B-Component), whereas Customer’s MAC addresses (I-Component) use data plane MAC learning.

What are we going to test?

EVPN in general is control-plane (or signaling) technology that may use different type of transport. One of the most popular for data-center is VXLAN, and both Nokia (Alcatel-Lucent) SR OS and Cisco IOS XR supports it. Unfortunately Cisco IOS XRv (at least my version 5.3.2) doesn’t support it, that’s we’ll test only PBB-EVPN with MPLS forwarding plane. If later Cisco IOS XRv will support VXLAN, we’ll such kind of lab separately.

Topology

Physical topology:

The logical topology remains as it was previously in VPWS and VPLS lab:

In previous articles about MPLS L2VPNs we have used different types of MPLS forwarding plane. So for VPWS we have used RSVP-TE for creation of MPLS LSP, whereas in VPLS we have used SPRING (Segment Routing). In this article we’ll use LDP for establishing MPLS LSPs. Routing protocol will be ISIS as it was previously.

Unfortunately Cisco IOS XRv doesn’t support data plane for L2 services, so we can test packet forwarding via PBB-EBPN only between SR1 and SR2 (XR3 can be P router in this case), though SR3 fully participates in signaling.

Initial configuration for the lab is in the following files: linux_initial sr1_initial sr2_initial xr3_initial

I recommend you to review VPLS configuration beforehand, as vpls configuration context is used for PBB and EVPN in Nokia (Alcatel-Lucent) SR OS

PBB-EVPN over MPLS configuration

All routers in our network can be PE for PBB-EVPN, so we configure all of them with corresponding PBB and EVPN components.

In order for PBB to get working, I-SID must be the same across PBB customer-facing instances. In order EVPN to get working, EVI must be the same.

Here is out service topology:

Let’s configure PBB-EVPN in our multi-vendor network build with Nokia (Alcatel-Lucent) VSR (SR 7750) and Cisco IOS XRv (ASR 9000):

Nokia (Alcatel-Lucent) VSR (SR 7750)	Cisco IOS XR (ASR 9000)
SR1	XR1
A:SR1>edit-cfg# candidate view ————————— configure router autonomous-system 65000 bgp group “iBGP” peer-as 65000 local-address 10.255.255.11 cluster 10.255.255.11 family evpn neighbor 10.255.255.22 exit neighbor 10.255.255.33 exit exit exit exit service customer 2 create description “PBB_EVPN_CUSTOMER” exit vpls 11 customer 2 b-vpls create service-mtu 2000 pbb source-bmac 00:a1:50:00:00:11 exit bgp-evpn evi 123 mpls auto-bind-tunnel resolution any exit no shutdown exit exit no shutdown exit vpls 22 customer 2 i-vpls create pbb backbone-vpls 11 isid 112233 exit exit sap 1/1/2:111 create exit no shutdown exit exit exit exit —————————	RP/0/0/CPU0:XR3(config)#show conf interface GigabitEthernet0/0/0/0.333 l2transport encapsulation dot1q 333 rewrite ingress tag pop 1 symmetric ! router bgp 65000 bgp router-id 10.255.255.33 bgp log neighbor changes detail address-family l2vpn evpn ! neighbor 10.255.255.11 remote-as 65000 update-source Loopback0 address-family l2vpn evpn ! ! ! l2vpn pbb backbone-source-mac 00c5.c000.0033 ! logging bridge-domain ! bridge group PBB_EVPN bridge-domain CORE_PART pbb core evpn evi 123 ! ! bridge-domain EDGE_PART interface GigabitEthernet0/0/0/0.333 ! pbb edge i-sid 112233 core-bridge CORE_PART ! ! ! ! end
SR2
A:SR2>edit-cfg# candidate view ————————— configure router autonomous-system 65000 bgp group “iBGP” peer-as 65000 local-address 10.255.255.22 family evpn neighbor 10.255.255.11 exit exit exit exit service customer 2 create description “PBB_EVPN_CUSTOMER” exit vpls 11 customer 2 b-vpls create service-mtu 2000 pbb source-bmac 00:a1:50:00:00:22 exit bgp-evpn evi 123 mpls auto-bind-tunnel resolution any exit no shutdown exit exit no shutdown exit vpls 22 customer 2 i-vpls create pbb backbone-vpls 11 isid 112233 exit exit sap 1/1/2:222 create exit no shutdown exit exit exit exit —————————

I’m pretty sure that you have a question “What is going on here?” This question is fully legitimate, because it isn’t easy to understand configuration at a glance.

First of all we configure BGP peering in address-family EVPN (AFI/SAFI 25/70) between our PEs. SR1 acts as BGP route-reflector, so you see command “cluster” there. As the source interfaces for BGP session, system interface (at Nokia (Alcatel-Lucent) SR OS) and loopback 0 (at Cisco IOS XR) is used in order to make connection stable and ease next-hop lookup further, when we will map MPLS LSPs to services.

The second crucial part is configuration of “B-component” of PBB, what stands for “backbone”. In Nokia (Alcatel-Lucent) SR OS you add keyword “b-vpls” upon VPLS service creation, whereas in Cisco IOS XR you configure it as “pbb core” inside of bridge-domain configuration context. Inside this “B-component” you add mapping to certain EVI. In Cisco IOS XR it’s enough, but in Nokia (Alcatel-Lucent) SR OS you have to add parameters of forwarding plane. As we use MPLS as forwarding plane for our PBB-VPLS, we add resolution to any MPLS technology. You can see that there is a significant difference comparing to VPWS/VPLS configuration, where we had to configure and map SDP to service. In PBB-EVPN we have an option to map service directly to MPLS LSP. The main reason for that, what comes to my mind, is that in SDP we have a mix of MPLS LSP and signaling information. In case of PBB-EVPN we have only BGP for signaling (like in MPLS L3 VPNs), so there is no necessity co to have SDP.

The third (and the last) part of our configuration is “I-component” of PBB, what stands for “Instance”. That is customer-facing entity, so here we configure physical interface that is used for termination of customer circuit. In Nokia (Alcatel-Lucent) SR OS we have to add keyword “i-vpls” upon creation of VPLS service. Inside the configuration context we configure mapping to B-component and I-SID to uniquely identify this particular L2VPN in IP/MPLS network.

MP-BGP parameters for EVPN (like route-distinguisher (RD) and route-targets (RTs)) are configured automatically based on EVI, AS number and router-id.

That’s it. I hope that after the provided explanation, you understand more or less the configuration. Now it’s time to see what happens in the different FIB/LFIB and other tables and make some packet capture.

In multihoming scenarios it’s necessary to configure general EVPN parameters, like ES (Ethernet-Segment) for attached circuits, but it isn’t relevant for our lab. Refer to official config guides for more information about this topic.

The first notification regarding PBB-EVPN establishment you get in logs, if they are activated:

SR OS:
51 2016/11/02 08:50:44.28 UTC MINOR: BGP #2038 Base Peer 1: 10.255.255.33
“VR 1: Group iBGP: Peer 10.255.255.33: moved into established state”
!
!
IOS XR:
RP/0/0/CPU0:Oct 19 16:30:33.922 : bgp[1053]: %ROUTING-BGP-5-ADJCHANGE_DETAIL : neighbor 10.255.255.11 Up (VRF: default; AFI/SAFI: 25/70) (AS: 65000)

This notification informs you that you BGP session in L2VPN/EVPN address-family is established successfully. The next step is to check what BGP routes are sent/received. Let’s check them in Cisco IOS XR, because we see all routers of all types just on the one page:

RP/0/0/CPU0:XR3#show bgp l2vpn evpn rd 10.255.255.33:123
BGP router identifier 10.255.255.33, local AS number 65000
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0x0 RD version: 0
BGP main routing table version 15
BGP NSR Initial initsync version 3 (Reached)
BGP NSR/ISSU Sync-Group versions 0/0
BGP scan interval 60 secs
Status codes: s suppressed, d damped, h history, * valid, > best
i – internal, r RIB-failure, S stale, N Nexthop-discard
Origin codes: i – IGP, e – EGP, ? – incomplete
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 10.255.255.33:123 (default for vrf CORE_PART)
*>i[2][0][48][00a1.5000.0011][0]/104
10.255.255.11 0 100 0 i
*>i[2][0][48][00a1.5000.0022][0]/104
10.255.255.22 0 100 0 i
*> [2][0][48][00c5.c000.0033][0]/104
0.0.0.0 0 i
*>i[3][0][32][10.255.255.11]/80
10.255.255.11 0 100 0 i
*>i[3][0][32][10.255.255.22]/80
10.255.255.22 0 100 0 i
*>i[3][112233][32][10.255.255.11]/80
10.255.255.11 0 100 0 i
*>i[3][112233][32][10.255.255.22]/80
10.255.255.22 0 100 0 i
*> [3][112233][32][10.255.255.33]/80
0.0.0.0 0 i
Processed 8 prefixes, 8 paths

I’ve filtered routes by RD in order to shorten the output, because each PE send its own routes with its own RD, which are then imported according to RT in appropriate table. You see that there are two types of routes in our table: 2 and 3. The route type 3 is used for transporting BUM traffic (Broadcast, Unknown Unicast, Multicast), whereas the type 2 is used for sending known unicast traffic. In Nokia (Acatel-Lucent) SR OS you see these routes separately:

A:SR1# show router bgp routes evpn inclusive-mcast
============================================================================
BGP Router ID:10.255.255.11 AS:65000 Local AS:65000
============================================================================
Legend –
Status codes : u – used, s – suppressed, h – history, d – decayed, * – valid
l – leaked, x – stale, > – best, b – backup, p – purge
Origin codes : i – IGP, e – EGP, ? – incomplete
============================================================================
BGP EVPN Inclusive-Mcast Routes
============================================================================
Flag Route Dist. OrigAddr
Tag NextHop
—————————————————————————-
u*>i 10.255.255.22:123 10.255.255.22
112233 10.255.255.22
u*>i 10.255.255.22:123 10.255.255.22
0 10.255.255.22
u*>i 10.255.255.33:123 10.255.255.33
112233 10.255.255.33
—————————————————————————-
Routes : 3
============================================================================
!
!
A:SR1# show router bgp routes evpn mac
============================================================================
BGP Router ID:10.255.255.11 AS:65000 Local AS:65000
============================================================================
Legend –
Status codes : u – used, s – suppressed, h – history, d – decayed, * – valid
l – leaked, x – stale, > – best, b – backup, p – purge
Origin codes : i – IGP, e – EGP, ? – incomplete
============================================================================
BGP EVPN MAC Routes
============================================================================
Flag Route Dist. MacAddr ESI
Tag Mac Mobility Label1
Ip Address
NextHop
—————————————————————————-
u*>i 10.255.255.22:123 00:a1:50:00:00:22 ESI-0
0 Static LABEL 262141
N/A
10.255.255.22
u*>i 10.255.255.33:123 00:c5:c0:00:00:33 ESI-0
0 Seq:0 LABEL 24002
N/A
10.255.255.33
—————————————————————————-
Routes : 2
============================================================================

As you may guess, each route has its own label that is used for transporting certain packets. And if the labels for unicast traffic is shown at once, labels for BUM traffic isn’t shown. To see them you need to go into details:

A:SR1# show router bgp routes evpn inclusive-mcast detail | match expression “Nexthop|Label|Tunnel|Tag”
Nexthop : 10.255.255.22
Res. Nexthop : 10.0.0.1
Tag : 112233
Route Tag : 0
PMSI Tunnel Attributes :
Tunnel-type : Ingress Replication
MPLS Label : LABEL 262141
Tunnel-Endpoint: 10.255.255.22
—
Nexthop : 10.255.255.22
Res. Nexthop : 10.0.0.1
Tag : 0
Route Tag : 0
PMSI Tunnel Attributes :
Tunnel-type : Ingress Replication
MPLS Label : LABEL 262141
Tunnel-Endpoint: 10.255.255.22
—
Nexthop : 10.255.255.33
Res. Nexthop : 10.0.0.5
Tag : 112233
Route Tag : 0
PMSI Tunnel Attributes :
Tunnel-type : Ingress Replication
MPLS Label : LABEL 24003
Tunnel-Endpoint: 10.255.255.33

The next interesting fact, which you can spot reviewing the provided outputs, is that Nokia (Acatel-Lucent) SR OS uses the same label both for BUM and known unicast traffic, whereas Cisco IOS XR uses two separate labels. Also you can notice that Nokia (Alcatel-Lucent) SR OS announces routes for configured I-SID (112233) and for default (0), whereas Cisco IOS XR announces route only for configured I-SID (112233).

Let’s review the configuration of the bridges (VPLS instances) to see what is going there. We start with the B-Component:

A:SR1# show service service-using
============================================================================
Services
============================================================================
ServiceId Type Adm Opr CustomerId Service Name
—————————————————————————-
11 b-VPLS Up Up 2
22 i-VPLS Up Up 2
2147483648 IES Up Down 1 _tmnx_InternalIesService
2147483649 intVpls Up Down 1 _tmnx_InternalVplsService
—————————————————————————-
Matching Services : 4
—————————————————————————-
!
!
============================================================================
A:SR1# show service id 11 base
============================================================================
Service Basic Information
============================================================================
Service Id : 11 Vpn Id : 0
Service Type : b-VPLS
Name : (Not Specified)
Description : (Not Specified)
Customer Id : 2 Creation Origin : manual
Last Status Change: 11/02/2016 08:49:32
Last Mgmt Change : 11/02/2016 08:49:32
Etree Mode : Disabled
Admin State : Up Oper State : Up
MTU : 2000 Def. Mesh VC Id : 11
SAP Count : 0 SDP Bind Count : 0
Snd Flush on Fail : Disabled Host Conn Verify : Disabled
SHCV pol IPv4 : None
Propagate MacFlush: Disabled Per Svc Hashing : Disabled
Allow IP Intf Bind: Disabled
Fwd-IPv4-Mcast-To*: Disabled Fwd-IPv6-Mcast-To*: Disabled
Temp Flood Time : Disabled Temp Flood : Inactive
Temp Flood Chg Cnt: 0
SPI load-balance : Disabled
TEID load-balance : Disabled
Src Tep IP : N/A
VSD Domain : <none>
Oper Backbone Src : 00:a1:50:00:00:11
Use SAP B-MAC : Disabled
i-Vpls Count : 1
Epipe Count : 0
Use ESI B-MAC : Disabled
============================================================================

There isn’t too much information that is useful for us, is it? Just point out that service-MTU in B-component must be at least 18 bytes bigger than I-Component. The most relevant information is contained in bgp-evpn section:

A:SR1# show service id 11 bgp-evpn
============================================================================
BGP EVPN Table
============================================================================
MAC Advertisement : Enabled Unknown MAC Route : Disabled
CFM MAC Advertise : Disabled
VXLAN Admin Status : Disabled Creation Origin : manual
MAC Dup Detn Moves : 5 MAC Dup Detn Window: 3
MAC Dup Detn Retry : 9 Number of Dup MACs : 0
IP Route Advert : Disabled
EVI : 123
Ing Rep Inc McastAd: Enabled
Accept IVPLS Flush : Disabled
============================================================================
BGP EVPN MPLS Information
============================================================================
Admin Status : Enabled
Force Vlan Fwding : Disabled Control Word : Disabled
Split Horizon Group: (Not Specified)
Ingress Rep BUM Lbl: Disabled Max Ecmp Routes : 0
Ingress Ucast Lbl : 262141 Ingress Mcast Lbl : 262141
Entropy Label : Disabled
RestProtSrcMacAct : none
============================================================================
BGP EVPN MPLS Auto Bind Tunnel Information
============================================================================
Resolution : any
Filter Tunnel Types: (Not Specified)
============================================================================

Pay attention that control word is disabled. It’s very important and you will see further why. To check associated I-Components with your B-Component, use the following command:

A:SR1# show service id 11 i-vpls
============================================================================
Related i-Vpls services for b-Vpls service 11
============================================================================
i-Vpls SvcId Oper ISID Admin Oper
—————————————————————————-
22 112233 Up Up
—————————————————————————-
Number of Entries : 1
—————————————————————————-
============================================================================

Ok, let’s check the I-Component itself:

A:SR1# show service id 22 base
============================================================================
Service Basic Information
============================================================================
Service Id : 22 Vpn Id : 0
Service Type : i-VPLS
Name : (Not Specified)
Description : (Not Specified)
Customer Id : 2 Creation Origin : manual
Last Status Change: 11/02/2016 08:49:32
Last Mgmt Change : 11/02/2016 08:49:32
Etree Mode : Disabled
Admin State : Up Oper State : Up
MTU : 1514 Def. Mesh VC Id : 22
SAP Count : 1 SDP Bind Count : 0
Snd Flush on Fail : Disabled Host Conn Verify : Disabled
SHCV pol IPv4 : None
Propagate MacFlush: Disabled Per Svc Hashing : Disabled
Allow IP Intf Bind: Disabled
Fwd-IPv4-Mcast-To*: Disabled Fwd-IPv6-Mcast-To*: Disabled
Temp Flood Time : Disabled Temp Flood : Inactive
Temp Flood Chg Cnt: 0
SPI load-balance : Disabled
TEID load-balance : Disabled
Src Tep IP : N/A
VSD Domain : <none>
b-Vpls Id : 11 Oper ISID : 112233
b-Vpls Status : Up
Snd Flush in bVpls: None
Flsh On bVpls Fail: Disabled Prop Flsh fr bVpls: Disabled
Force QTag Fwd : Disabled
SendBvplsEvpnFlush: Disabled
—————————————————————————-
Service Access & Destination Points
—————————————————————————-
Identifier Type AdmMTU OprMTU Adm Opr
—————————————————————————-
sap:1/1/2:111 q-tag 1518 1518 Up Up
============================================================================
* indicates that the corresponding row element may have been truncated.

As MAC addresses of B-components are already distributed via BGP, we can check FDB (forwarding database) for it. In the same time we haven’t configured so far any customers for our PBB-EVPN service, so FDB in I-Component must be empty:

A:SR1# show service id 11 fdb detail
============================================================================
Forwarding Database, Service 11
============================================================================
ServId MAC Source-Identifier Type Last Change
Age
—————————————————————————-
11 00:a1:50:00:00:22 eMpls: EvpnS:P 11/02/16 08:50:38
10.255.255.22:262141
11 00:c5:c0:00:00:33 eMpls: Evpn 11/02/16 08:50:49
10.255.255.33:24002
—————————————————————————-
No. of MAC Entries: 2
—————————————————————————-
Legend: L=Learned O=Oam P=Protected-MAC C=Conditional S=Static
============================================================================
!
!
A:SR1# show service id 22 fdb detail
============================================================================
Forwarding Database, Service 22
============================================================================
ServId MAC Source-Identifier Type Last Change
Age
—————————————————————————-
No Matching Entries
============================================================================

Let’s check Cisco side now. The main disadvantage of using Cisco IOS XRv is the absence of data plane for L2 services, therefore we can’t make any tests of its operation. The only thing that we can do is to check control plane of PBB-EVPN. Just right after configuration part we have checked the BPG RIB for L2VPN/EVPN routes. Let’s check the bridges themselves now:

RP/0/0/CPU0:XR3#show l2vpn bridge-domain
Legend: pp = Partially Programmed.
Bridge group: PBB_EVPN, bridge-domain: CORE_PART, id: 0, state: up, ShgId: 0, MSTi: 0
Type: pbb-core
Number of associated pbb-edge BDs: 1
Aging: 300 s, MAC limit: 4000, Action: none, Notification: syslog
Filter MAC addresses: 0
ACs: 0 (0 up), VFIs: 0, PWs: 0 (0 up), PBBs: 1 (1 up)
List of PBBs:
PBB Core, state: up
List of ACs:
List of Access PWs:
List of VFIs:
Bridge group: PBB_EVPN, bridge-domain: EDGE_PART, id: 1, state: up, ShgId: 0, MSTi: 0
Type: pbb-edge, I-SID: 112233
Aging: 300 s, MAC limit: 4000, Action: none, Notification: syslog
Filter MAC addresses: 0
ACs: 1 (1 up), VFIs: 0, PWs: 0 (0 up), PBBs: 1 (1 up)
List of PBBs:
PBB Edge, state: up, Static MAC addresses: 0
List of ACs:
Gi0/0/0/0.333, state: up, Static MAC addresses: 0
List of Access PWs:
List of VFIs:

Here you see the brief summary of all configured bridges. To see EVPN parameters of the B-Component in Cisco IOS XR you should display detailed information (the output is modified to leave only interesting information):

RP/0/0/CPU0:XR3#show l2vpn bridge-domain bd-name CORE_PART detail
Legend: pp = Partially Programmed.
Bridge group: PBB_EVPN, bridge-domain: CORE_PART, id: 0, state: up, ShgId: 0, MSTi: 0
Coupled state: disabled
Type: pbb-core
Number of associated pbb-edge BDs: 1
EVPN:
EVI: 123
Route Distinguisher: (auto) 10.255.255.33:123
Imposition Statistics:
Packet Count: 293221248408748033
Byte Count : 293180497827310268
Disposition Statistics:
Packet Count: 312030181667895045
Byte Count : 1187072129675406012
AS Number: 65000
MAC learning: enabled
MAC withdraw: enabled
MAC withdraw for Access PW: enabled
MAC withdraw sent on: bridge port up
MAC withdraw relaying (access to access): disabled
Flooding:
Broadcast & Multicast: enabled
Unknown unicast: enabled
MAC aging time: 300 s, Type: inactivity
MAC limit: 4000, Action: none, Notification: syslog
Bridge MTU: 1500
ACs: 0 (0 up), VFIs: 0, PWs: 0 (0 up), PBBs: 1 (1 up)
List of PBBs:
PBB Core, state is up
XC ID 0xff000001
MAC learning: enabled
Flooding:
Broadcast & Multicast: enabled
Unknown unicast: enabled
MAC aging time: 300 s, Type: inactivity
MAC limit: 4000, Action: none, Notification: syslog

Information about I-Component is shown but the same command, just change the bridge name:

RP/0/0/CPU0:XR3#show l2vpn bridge-domain bd-name EDGE_PART detail
Legend: pp = Partially Programmed.
Bridge group: PBB_EVPN, bridge-domain: EDGE_PART, id: 1, state: up, ShgId: 0, MSTi: 0
Coupled state: disabled
Type: pbb-edge, I-SID: 112233
Core-bridge: CORE_PART (State: Bridge Up)
MIRP-lite: not supported
Format: none
MAC learning: enabled
MAC withdraw: enabled
MAC withdraw for Access PW: enabled
MAC withdraw sent on: bridge port up
MAC withdraw relaying (access to access): disabled
Flooding:
Broadcast & Multicast: enabled
Unknown unicast: enabled
MAC aging time: 300 s, Type: inactivity
MAC limit: 4000, Action: none, Notification: syslog
Bridge MTU: 1500
ACs: 1 (1 up), VFIs: 0, PWs: 0 (0 up), PBBs: 1 (1 up)
List of PBBs:
PBB Edge, state is up
XC ID 0xff000002
MAC learning: enabled
Flooding:
Broadcast & Multicast: enabled
Unknown unicast: enabled
MAC aging time: 300 s, Type: inactivity
MAC limit: 4000, Action: none, Notification: syslog
List of ACs:
AC: GigabitEthernet0/0/0/0.333, state is up
Type VLAN; Num Ranges: 1
VLAN ranges: [333, 333]
MTU 1500; XC ID 0x3; interworking none
MAC learning: enabled
Flooding:
Broadcast & Multicast: enabled
Unknown unicast: enabled
MAC aging time: 300 s, Type: inactivity
MAC limit: 4000, Action: none, Notification: syslog

So far we have reviewed some fancy outputs of different “show” commands that informs us about status of configured service (mainly signaling). Now it’s time to make some kind of simulation to see what happens with packet in our network. We’ll use the same topology to what we have used previously in VPLS lab:

Don’t forget to disable direct link between SR1 and SR2. Now let’s configure our customers’ simulation:

Nokia (Alcatel-Lucent) VSR (SR 7750)	Cisco IOS XR (ASR 9000)
SR1	XR1
A:SR1>edit-cfg# candidate view ————————— configure service vprn 100 customer 2 create route-distinguisher 10.255.255.11:100 interface VPLS111_CUST create address 192.168.0.1/24 ipv6 address fc00::192:168:0:1/120 link-local-address fe80::51:111:1 exit sap 1/1/2:100 create exit no shutdown exit no shutdown exit exit exit —————————	RP/0/0/CPU0:XR3(config)#show conf vrf CUST100 address-family ipv4 unicast ! address-family ipv6 unicast ! ! interface GigabitEthernet0/0/0/0.100 vrf CUST100 ipv4 address 192.168.0.2 255.255.255.0 ipv6 address fe80::c5c0:111:2 link-local ipv6 address fc00::192:168:0:2/120 encapsulation dot1q 100 ! end
Linux – bridge A	Linux – bridge B
[root@localhost ~]# /sbin/vconfig add eth2 100 /sbin/vconfig add vnet2 111 ifconfig eth2.100 up ifconfig vnet2.111 up brctl addbr br101 brctl addif br101 eth2.100 brctl addif br101 vnet2.111 ifconfig br101 up	[root@localhost ~]# /sbin/vconfig add vnet6 222 /sbin/vconfig add vnet2 100 ifconfig vnet6.222 up ifconfig vnet2.100 up brctl addbr br102 brctl addif br102 vnet6.222 brctl addif br102 vnet2.100 ifconfig br102 up

Before we make ping, we check again the content of I-VPLS instances at SR1 and SR2:

Now it’s time to make some pings:

A:SR1# ping router 100 192.168.0.2
PING 192.168.0.2 56 data bytes
64 bytes from 192.168.0.2: icmp_seq=1 ttl=255 time=66.2ms.
64 bytes from 192.168.0.2: icmp_seq=2 ttl=255 time=11.5ms.
64 bytes from 192.168.0.2: icmp_seq=3 ttl=255 time=37.0ms.
64 bytes from 192.168.0.2: icmp_seq=4 ttl=255 time=25.0ms.
64 bytes from 192.168.0.2: icmp_seq=5 ttl=255 time=26.1ms.
—- 192.168.0.2 PING Statistics —-
5 packets transmitted, 5 packets received, 0.00% packet loss
round-trip min = 11.5ms, avg = 33.2ms, max = 66.2ms, stddev = 18.4ms

Our PBB-EVPN works! It’s wonderful, isn’t it? Let’s check what is going on the backstage. The first packet is ARP, because our customer need to resolve MAC address to construct proper ICMP packet. Wireshark shows it:

Unfortunately Wireshark doesn’t decode the packet properly, but still we see some information:

“20” – unlabeled ARP come to SR2 from VPRN 100 at SR1
“21” and “22” – SR2 add labels and send ARP (that is BUM traffic) to SR1 and XR3. You see it from MPLS labels.
“23” – XR3 swaps transport label and sends packet “21) further to SR1.
“24” – SR2 send unlabeled packet to VRF CUST100 at XR3
“25 – 28” back way of the packet from VRF CUST100 at XR3 to VPRN 100 at SR1. This traffic is sent as unicast one.

After ARP process is finished, ICMP itself comes:

During this process the MAC addresses are learned and filled in I-Component’s FDB, so we can check it’s operation:

A:SR1# show service id 22 fdb detail
============================================================================
Forwarding Database, Service 22
============================================================================
ServId MAC Source-Identifier Type Last Change
Age
—————————————————————————-
22 00:50:56:3a:83:ff sap:1/1/2:111 L/0 11/07/16 20:59:34
22 fa:ac:a6:14:01:03 b-eMpls: L/0 11/07/16 20:59:34
10.255.255.22:262142
—————————————————————————-
No. of MAC Entries: 2
—————————————————————————-
Legend: L=Learned O=Oam P=Protected-MAC C=Conditional S=Static
============================================================================
!
!
A:SR2# show service id 22 fdb detail
============================================================================
Forwarding Database, Service 22
============================================================================
ServId MAC Source-Identifier Type Last Change
Age
—————————————————————————-
22 00:50:56:3a:83:ff b-eMpls: L/0 11/07/16 20:59:20
10.255.255.11:262142
22 fa:ac:a6:14:01:03 sap:1/1/2:222 L/0 11/07/16 20:59:20
—————————————————————————-
No. of MAC Entries: 2
—————————————————————————-
Legend: L=Learned O=Oam P=Protected-MAC C=Conditional S=Static
============================================================================

In the same there is no changes in B-Components, as PE MAC addresses were initially discovered via BGP and all signaling was formed. That’s it with PBB-EVPN over MPLS. The final configs are here: linux_final sr1_final sr2_final xr3_final

Lessons learned

You see that unlabeled packets are displayed correctly, whereas packets send over PBB-EVPN aren’t clearly decoded. Let’s see, probably we can do it better. Here is the packet:

The problem is that Wireshark thinks that our labeled has control world, though it doesn’t have it.

This problem is well known and discussed in this thread at wireshark site.

Here what we have:

“00 a1 50 00 00 11” (marked with red) is PBB MAC address of the SR1 that is destination MAC address in this case. You can check it with what we have configured.
“00 a1 50 00 00 22” (marked with yellow) is PBB MAC address of SR2 that is source MAC address in this case.
“88 e7” (marked with green) is EtherType for PBB.
“00 01 b6 69” (marked with blue) is hexadecimal representation of I-SID, which is 112233.
“00 50 56 3a 83 ff” (marked with orange) is MAC address of the interface in VRF CUST100 at XR3 that is destination MAC address for customer traffic:

RP/0/0/CPU0:XR3#show int gig 0/0/0/0.100 | inc add
Hardware is VLAN sub-interface(s), address is 0050.563a.83ff
Internet address is 192.168.0.2/24

“fa ac a6 14 01 03” (marked with violet) is MAC address of the interface in VPRN 100 at SR1 that is source MAC address for customer traffic:

A:SR1# show router 100 interface “VPLS111_CUST” detail | match MAC
MAC Address : fa:ac:a6:14:01:03 Mac Accounting : Disabled

“08 00” (marked with brown) is EtherType for IPv4 that is encapsulated further.

Conclusion

EVPN-PBB, as I’ve already stated in the beginning, is probably the most scalable and powerful L2VPN technology that can be deployed into MPLS word of service providers. Pure EVPN with VXLAN data plane is perfect solution for data centers and their interconnections. Nokia (Alcatel-Lucent) SR OS supports both PBB-EVPN and pure EVPN, whereas Cisco IOS XR currently supports only PBB-EVPN (pure EVPN is supported at Cisco NX-OS).

So far we have reviewed all possible option of deploying L2VPN services across Service Provider network: VPWS, VPLS and EVPN. In the next article we’ll review L3VPN (BGP/MPLS VPN). Take care and good bye!

Support us

Support new interop and automation articles at karneliuk.com
EUR

I want to support with:

BR,

Anton Karneliuk