Hello my friend,

This article should have been much smaller than the recent one, because there is not as much configuration as it was there. But now I realise that it isn’t small. The reason for that: it’s impossible to overestimate the importance of this topic, because MTU mismatch is very painful and not-so-easy to troubleshoot problem in the networks. Let’s review what we should do to avoid problems with MTU.

What is MTU and how it influences our network’s life

MTU is abbreviation from Maximum Transmit Unit, which is actually the amount of information (including all protocol headers) that can be transferred across interface without being fragmented. So we can say that MTU is parameter of the interface. If we transmit something on one end, we should receive something on another end, right? So MTU has a less popular brother (or sister) that is called MRU, what stands for Maximum Receive Unit. For transmission to be successful MRU at receiving side must be equal or greater then MTU at transmitting side. As a rule at each interface MTU equals MRU, that’s why we usually used only term MTU, though it’s two different parameters.

034_net_01_frame
There are two most used types of MTU:

  • Layer 2 MTU (usually called just as MTU, port MTU, interface MTU) is a maximum size of frame including all Layer 2 headers. For example, for Ethernet default MTU size is 1514 bytes, where 1500 is data, and 14 byte is Ethernet header, which consists of 6 bytes destination MAC address, 6 bytes source MAC address and 2 bytes Ethertype. Point out that preamble and start frame delimeter as well as FCS isn’t included in this value. For 802.1Q Frame the default MTU is 1518 bytes, because we add 4-byte 802.1Q tag to our calculations.
  • Layer 3 MTU (usually called as IP MTU) is a maximum size of packet including Layer 3 header that can be transmitted without fragmentation. The default value for IP MTU is 1500 byte.

The interdependency between MTU and IP MTU is very straightforward. You can define IP MTU as:

IP MTU = MTU – L2 header (Ethernet or PPP) – other headers (MPLS, GRE and so on if they exist)

Basically you should align all L2 and L3 MTU in the network in order to avoid problems, caused by MTU mismatch. MTU mismatch arises when MTU of transmitter is bigger than MRU of receiver. In such situation such frame is silently dropped.

034_net_02_link_MTU

Another problem relates to the case when MTU is different at different links. In such situation the transmitter should send the frame that satisfy the lowest MTU following the whole path. If it’s bigger, then two variants are possible:

  • The first one is that IP traffic is being transmitted, so packets can be fragmented at the intermediate routers, which has interfaces with different MTU. This operation is done in control plane, so it uses capacity of CPU/Memory what is not desirable. Fragmentation means that original packet is divided into several smaller packets, which satisfy MTU/MRU at outgoing interface.
  • The second case is that the L2 traffic is being transmitted (switching environment or L2 VPN). There is no layer 2 protocol (or at least widely used protocol) that supports fragmentation. So these frames will be dropped.

034_net_03_path_MTU

The third issue with MTU is that MTU mismatch usually leads to the problems with routing protocol (mainly interior as OSPF or ISIS), because they have built it mechanisms to check MTU at both sides of link by sending this number in OSPF hello message or using padding of Hello packets in ISIS up to MTU size at interface. Though these checks can be disabled, it’s better not to do it, because they are your additional helpers to discover and fix MTU mismatch.

MTU in Nokia (Alcatel-Lucent) SR OS

There are a lot of conditions, which influences the MTU size at certain interface. For people, who works with Nokia (Alcatel-Lucent) SR 7750 for a long time, or who attends Nokia’s Service Architecture courses, it might looks easy, but for the rest of engineers this isn’t so obvious. Let’s dig onto details.
There are two different type of ports in Nokia (Alcatel-Lucent):

  • Network port is used for connectivity with core network (PE-PE or PE-P). If the interface has bandwidth of 1 Gbps or bigger, then its default MTU is 9212 bytes (and it is maximum possible value). Otherwise it will be 1514 bytes
  • Access port is used for customer’s connectivity. Access port is crucial component of all types of client services: L2 VPN (E-Line and E-LAN), L3 VPN (BGP MPLS VPN) and internet access (called IES – Internet Enhanced Service in Nokia (Alcatel-Lucent) Architecture), because only at access ports it’s possible to create SAP (Service Attach Point that is logical customer port). Default MTU is 1514, but it depends on interface encapsulation type.

As I’m using Nokia (Alcatel-Lucent) VSR, not physical SR 7750, I have another default MTU at Gigabit Ethernet port.

Let’s see what’s going on in the console:

*A:SR1>config>port# info
———————————————-
ethernet
exit
no shutdown
———————————————-
!
!
*A:SR1# show port 1/1/1
============================================================================
Ethernet Interface
============================================================================
Description     : 10/100/Gig Ethernet SFP
Interface       : 1/1/1                      Oper Speed : 1 Gbps
Physical Link   : Yes                        MTU        : 8936
Configured Mode : network                    Encap Type : null

As you remember (link to first article), we have to create a logical interface as it’s separated instance:

*A:SR1# configure router interface MTUTEST
*A:SR1>config>router>if$ info
———————————————-
address 10.0.0.0/31
port 1/1/1
no shutdown
———————————————-
!
!
*A:SR1# show router interface “MTUTEST” detail | match MTU
If Name       : MTUTEST
IP Oper MTU   : 8922                          ICMP Mask Reply : True

So IP MTU is 14 bytes lower as it should be according to the formula above.
Now let’s create a dot1q interface. To do it we need to delete the existing one, because we have to change encapsulation type of our port at Nokia (Alcatel-Lucent) SR OS router:

*A:SR1# configure router interface MTUTEST
*A:SR1>config>port# info
———————————————-
ethernet
encap-type dot1q
exit
no shutdown
———————————————-
!
!
*A:SR1>config# router interface MTUSUB
*A:SR1>config>router>if$ info
———————————————-
address 10.0.1.0/31
port 1/1/1:10
no shutdown
———————————————-
!
!
*A:SR1# show port 1/1/1
============================================================================
Ethernet Interface
============================================================================
Description     : 10/100/Gig Ethernet SFP
Interface       : 1/1/1                      Oper Speed : 1 Gbps
Physical Link   : Yes                        MTU        : 8936
Configured Mode : network                    Encap Type : 802.1q
!
!
*A:SR1# show router interface “MTUSUB” detail | match MTU
If Name         : MTUSUB
IP Oper MTU     : 8918                       ICMP Mask Reply : True

You see that that initial MTU hasn’t changed, meaning that if we have maximum size, but it’s an IP MTU that has changed. It’s 4 bytes lower than it was with encapsulation null. Point out that it is impossible to configure qinq encapsulation at network port.

Network port logic is clear: it has maximum MTU by default and then lowers IP MTU depending on encapsulation type.

Now let’s shift to access port a little bit. To see it I’ll configure something called IES (Internet Enhanced Services). I won’t explain in details this time, but technically it’s just a layer 3 interface places at access port. So we change configuration in the following way:

*A:SR1# configure port 1/1/1
*A:SR1>config>port# info
———————————————-
ethernet
mode access
exit
no shutdown
———————————————-
!
!
*A:SR1# configure service ies 1 customer 1 create
———————————————-
interface “MTUTEST” create
address 10.0.0.0/31
sap 1/1/1 create
exit
exit
no shutdown
———————————————-
!
!
*A:SR1# show port 1/1/1
============================================================================
Ethernet Interface
============================================================================
Description     : 10/100/Gig Ethernet SFP
Interface       : 1/1/1                      Oper Speed : 1 Gbps
Physical Link   : Yes                        MTU        : 1514
Configured Mode : access                     Encap Type : null
!
!
*A:SR1# show router interface “MTUTEST” detail | match MTU
If Name         : MTUTEST
IP Oper MTU     : 1500                       ICMP Mask Reply : True

As soon as you change interfaces type, MTU is set up to 1514 automatically. IP MTU at newly created interface follows standard rule to be 14 bytes less than MTU due to Ethernet header. Now let’s change type to dot1q:

*A:SR1# configure port 1/1/1
*A:SR1>config>port# info
———————————————-
ethernet
mode access
encap-type dot1q
exit
no shutdown
———————————————-
!
!
*A:SR1>config>service>ies# info
———————————————-
interface “MTUDOT” create
address 10.0.10.0/31
sap 1/1/1:10 create
exit
exit
no shutdown
———————————————-
!
!
*A:SR1# show port 1/1/1
============================================================================
Ethernet Interface
============================================================================
Description     : 10/100/Gig Ethernet SFP
Interface       : 1/1/1                      Oper Speed : 1 Gbps
Physical Link   : Yes                        MTU        : 1518
Configured Mode : access                     Encap Type : 802.1q
!
!
*A:SR1# show router interface “MTUDOT” detail | match MTU
If Name         : MTUDOT
IP Oper MTU     : 1500                      ICMP Mask Reply : True

Here you see that after change of encapsulation type to dot1q the MTU is changed to 1518 at port, whereas the IP MTU remains the same (1500 bytes).
The last case for Nokia (Alcatel-Lucent) SR OS will be configuring of encapsulation to qinq at SAP:

*A:SR1# configure port 1/1/1
*A:SR1>config>port# info
———————————————-
ethernet
mode access
encap-type qinq
exit
no shutdown
———————————————-
!
!
*A:SR1>config>service>ies# info
———————————————-
interface “MTUDOT” create
address 10.0.110.0/31
sap 1/1/1:10.110 create
exit
exit
no shutdown
———————————————-
!
!
*A:SR1# show port 1/1/1
============================================================================
Ethernet Interface
============================================================================
Description     : 10/100/Gig Ethernet SFP
Interface       : 1/1/1                      Oper Speed : 1 Gbps
Physical Link   : Yes                        MTU        : 1522
Configured Mode : access                     Encap Type : qinq
!
!
*A:SR1# show router interface “MTUDOT” detail | match MTU
If Name         : MTUDOT
IP Oper MTU     : 1500                       ICMP Mask Reply : True

Again, you see that IP MTU remains 1500 bytes, while port MTU has increased to accept the overall frame of 1522 bytes size (14 bytes Ethernet header + 4 byte 1st VLAN tag + 4 byte 2nd VLAN tag).

MTU in Cisco IOS XR

Now it’s turn of Cisco IOS XR to show how it behaves. All interfaces at Cisco routers (not only IOS XR as ASR 9000, but IOS / IOS XR routers as well) has MTU 1514 at physical interface. Let’s take a look at newly created interface:

RP/0/0/CPU0:XR1#sh run int gig 0/0/0/0
interface GigabitEthernet0/0/0/0
ipv4 address 10.0.0.1 255.255.255.254
!
!
RP/0/0/CPU0:XR1#sh int gig 0/0/0/0 | include MTU
Description: MTU test
MTU 1514 bytes, BW 1000000 Kbit (Max: 1000000 Kbit)
!
!
RP/0/0/CPU0:XR1#sh ipv4 interface gig 0/0/0/0 | include MTU
MTU is 1514 (1500 is available to IP)

So you see that MTU (Layer 2 MTU) is 1514 bytes, which corresponds to standard size Ethernet frame without tags. We haven’t configured any MTU parameters, so these values are defaults. Now we’ll see how Cisco IOS XR deals with 802.1Q frames.

In order to create dot1q encapsulation at port, you should just create sub-interface and map it no certain VLAN value. You don’t have to change encapsulation type, as you did it in Nokia (Alcatel-Lucent) SR OS:

RP/0/0/CPU0:XR1(config)#show conf
interface GigabitEthernet0/0/0/0.10
ipv4 address 10.0.10.1 255.255.255.254
encapsulation dot1q 10
!
!
RP/0/0/CPU0:XR1#show interfaces | include “0/0/0/0|MTU”
GigabitEthernet0/0/0/0 is up, line protocol is up
Description: MTU test
MTU 1514 bytes, BW 1000000 Kbit (Max: 1000000 Kbit)
GigabitEthernet0/0/0/0.10 is up, line protocol is up
MTU 1518 bytes, BW 1000000 Kbit (Max: 1000000 Kbit)
!
!
RP/0/0/CPU0:XR1#show ipv4 int gigabitEthernet 0/0/0/0.10 | include MTU
MTU is 1518 (1500 is available to IP)

For more references, read this article about configuration of trunks.

At a glance it might look quite strange, as you don’t change MTU parameter at interface (I mean interface as a physical instance), but it’s automatically increased at sub-interfaces. Let’s create QinQ sub-interface at Cisco IOS XR:

RP/0/0/CPU0:XR1(config)#show conf
interface GigabitEthernet0/0/0/0.110
ipv4 address 10.0.110.1 255.255.255.254
encapsulation dot1q 10 second-dot1q 110
!
!
RP/0/0/CPU0:XR1#show interfaces | include “Ethernet|MTU”
GigabitEthernet0/0/0/0 is up, line protocol is up
MTU 1514 bytes, BW 1000000 Kbit (Max: 1000000 Kbit)
GigabitEthernet0/0/0/0.10 is up, line protocol is up
MTU 1518 bytes, BW 1000000 Kbit (Max: 1000000 Kbit)
GigabitEthernet0/0/0/0.110 is up, line protocol is up
MTU 1522 bytes, BW 1000000 Kbit (Max: 1000000 Kbit)
!
!
RP/0/0/CPU0:XR1#show ipv4 interface gigabitEthernet 0/0/0/0.110 | inc MTU
MTU is 1522 (1500 is available to IP)

In general Cisco IOS XR use the following rule regarding MTU:

It adds the necessary amount of bytes to initial MTU of physical interfaces depending on sub-interface encapsulation (4 bytes for dot1q encapsulation, and 8 bytes for QinQ encapsulation).

Harmonization of MTU between Nokia (Alcatel-Lucent) SR OS and Cisco IOS XR

You see that default parameter of MTU in Nokia (Alcatel-Lucent) SR OS and Cisco IOS XR are different. Then the obvious question arises: “What to do further?” The answer is straightforward as well: “Configure MTU with the same value in Cisco IOS XR and Nokia (Alcatel-Lucent) SR OS”. But before doing it consider two key points, which come from the explanation above:

  • Cisco IOS XR will always add some bytes to configured MTU of the port (4 bytes for dot1q sub interface and 8 bytes for qinq sub interface)
  • Nokia (Alcatel-Lucent) SR OS uses configure MTU parameter as maximum value including all overheads.

This rule makes you to define, for which type of MTU you want to align MTU. It’s impossible to align null, dot1q and qinq together, just because Nokia (Alcatel-Lucent) SR OS and Cisco IOS XR calculates MTU in a different way. Let’s review very simple topology and make some configurations:

018_net_03_top_1

Let’s assume we want to send frames with MTU 2014 bytes (IP MTU 2000 bytes) in ordinary Ethernet frame without any additional encapsulation:

Nokia (Alcatel-Lucent) VSR (SR 7750) Cisco IOS XR (ASR 9000)
SR1 XR1

*A:SR1# configure port 1/1/1
*A:SR1>config>port# info
—————————–
ethernet
mtu 2014
exit
no shutdown
—————————–
*A:SR1# configure router interface toXR
*A:SR1>config>router>if$ info
—————————–
address 10.0.0.1/24
port 1/1/1
no shutdown
—————————–

RP/0/0/CPU0:XR1(config)#show conf
interface GigabitEthernet0/0/0/0
mtu 2014
ipv4 address 10.0.0.2 255.255.255.0
!
end

The configuration is really simple as you see above. Let’s compare the MTU and IP MTU parameters at interfaces. Nokia (Alcatel-Lucent) SR OS has the following:

A:SR1# show port 1/1/1 | match expression “MTU|Encap”
Physical Link   : Yes                MTU        : 2014
Configured Mode : network            Encap Type : null
!
!
A:SR1# show router interface “toXR” detail | match MTU
IP Oper MTU     : 2000               ICMP Mask Reply : True

Cisco IOS XR has the same parameters:

RP/0/0/CPU0:XR1#show int gig 0/0/0/0 | inc MTU
MTU 2014 bytes, BW 1000000 Kbit (Max: 1000000 Kbit)
!
!
RP/0/0/CPU0:XR1#show ipv4 int gig 0/0/0/0 | inc MTU
MTU is 2014 (2000 is available to IP)

And now I want to show you one difference in ping tool between Nokia (Alcatel-Lucent) SR OS and Cisco IOS XR. Nokia side:

A:SR1# ping 10.0.0.2 do-not-fragment size 1972 count 1
PING 10.0.0.2 1472 data bytes
1980 bytes from 10.0.0.2: icmp_seq=1 ttl=255 time=6.45ms.
—- 10.0.0.2 PING Statistics —-
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 6.45ms, avg = 6.45ms, max = 6.45ms, stddev = 0.000ms
!
!
A:SR1# ping 10.0.0.2 do-not-fragment size 1973 count 1
PING 10.0.0.2 1473 data bytes
—- 10.0.0.2 PING Statistics —-
1 packet transmitted, 0 packets received, 100% packet loss

Cisco side:

RP/0/0/CPU0:XR1#ping 10.0.0.1 df-bit size 2000
Type escape sequence to abort.
Sending 5, 1500-byte ICMP Echos to 10.0.0.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/9 ms
!
!
RP/0/0/CPU0:XR1#ping 10.0.0.1 df-bit size 2001
Type escape sequence to abort.
Sending 5, 1501-byte ICMP Echos to 10.0.0.1, timeout is 2 seconds:
.
Success rate is 0 percent (0/1)

You see that packet size for ICMP packet, which we configure in CLI, is different in Nokia (Alcatel-Lucent) SR OS and Cisco IOS XR:

  • In Nokia (Alcatel-Lucent) SR OS you set size of payload of ICMP packet for ping, which is then appended by IP header (20 bytes) and ICMP header (8 bytes).
  • In Cisco IOS XR you set size of the whole ICMP packet including IP and ICMP headers

Well, with this encapsulation null there is no hints. Let’s now configure the link between SR1 and XR1 in such a way to pass packets with IP MTU 2000, but the frames will have one 802.1q tag:

Nokia (Alcatel-Lucent) VSR (SR 7750) Cisco IOS XR (ASR 9000)
SR1 XR1

*A:SR1# configure port 1/1/1
*A:SR1>config>port# info
—————————–
ethernet
encap-type dot1q
mtu 2018
exit
no shutdown
—————————–
*A:SR1# configure router interface toXR
*A:SR1>config>router>if$ info
—————————–
address 10.0.10.1/24
port 1/1/1:10
no shutdown
—————————–

RP/0/0/CPU0:XR1(config)#show conf
interface GigabitEthernet0/0/0/0.10
ipv4 address 10.0.10.2 255.255.255.0
encapsulation dot1q 10
!
end

In Nokia (Alcatel-Lucent) SR OS we have to remove any mapping of physical port to logical instances (interfaces and SAP) before we can change encapsulation type of port.

As you see, we don’t change MTU at Cisco IOS XR. The reason for that is the rule, which we’ve described above. Let’s check that it’s so. Cisco IOS XR side:

RP/0/0/CPU0:XR1#show int gig 0/0/0/0.10 | inc MTU
MTU 2018 bytes, BW 1000000 Kbit (Max: 1000000 Kbit)
!
!
RP/0/0/CPU0:XR1#show ipv4 int gig 0/0/0/0.10 | inc MTU
MTU is 2018 (2000 is available to IP)

Now it’s time of the Nokia (Alcatel-Lucent) router:

A:SR1# show port 1/1/1 | match expression “MTU|Encap”
Physical Link   : Yes              MTU        : 2018
Configured Mode : network          Encap Type : 802.1q
!
!
A:SR1# show router interface “toXR” detail | match MTU
IP Oper MTU     : 2000             ICMP Mask Reply : True

Upon performing test ping, we see the same result as it has been above.

When you change MTU, don’t forget to reset port (shutdown / no shutdown).

Lessons learned

First I met this problem during my article writing in IPv4/OSPF article, when my routers failed to establish adjacency. Then I’ve seen this problem in the ISIS as well. In L2VPN topic MTU issue is even more important, so we’ll review it when we speak about them.

Conclusion

MTU is one of the critical parameter in the network, and frankly it’s one of the worst known thing by network engineers. Because it’s really simple thing and usually nobody cares about it, because it works by default. If engineers change it, usually they change it end-to-end in the network and also don’t care about it. But as you see, different vendors have different approach in calculation and configuration of L2/L3 MTU at ports, what may lead to traffic loss in the network, if it isn’t cared well. Though there is standardized mechanism to find the lowest MTU at link (path MTU discovery – PMTUD), it often fails due to security measures in the network. Take care and good bye!

Support us





BR,

Anton Karneliuk