Hello my friend,
This article should have been much smaller than the recent one, because there is not as much configuration as it was there. But now I realise that it isn’t small. The reason for that: it’s impossible to overestimate the importance of this topic, because MTU mismatch is very painful and not-so-easy to troubleshoot problem in the networks. Let’s review what we should do to avoid problems with MTU.
What is MTU and how it influences our network’s life
MTU is abbreviation from Maximum Transmit Unit, which is actually the amount of information (including all protocol headers) that can be transferred across interface without being fragmented. So we can say that MTU is parameter of the interface. If we transmit something on one end, we should receive something on another end, right? So MTU has a less popular brother (or sister) that is called MRU, what stands for Maximum Receive Unit. For transmission to be successful MRU at receiving side must be equal or greater then MTU at transmitting side. As a rule at each interface MTU equals MRU, that’s why we usually used only term MTU, though it’s two different parameters.
There are two most used types of MTU:
- Layer 2 MTU (usually called just as MTU, port MTU, interface MTU) is a maximum size of frame including all Layer 2 headers. For example, for Ethernet default MTU size is 1514 bytes, where 1500 is data, and 14 byte is Ethernet header, which consists of 6 bytes destination MAC address, 6 bytes source MAC address and 2 bytes Ethertype. Point out that preamble and start frame delimeter as well as FCS isn’t included in this value. For 802.1Q Frame the default MTU is 1518 bytes, because we add 4-byte 802.1Q tag to our calculations.
- Layer 3 MTU (usually called as IP MTU) is a maximum size of packet including Layer 3 header that can be transmitted without fragmentation. The default value for IP MTU is 1500 byte.
The interdependency between MTU and IP MTU is very straightforward. You can define IP MTU as:
IP MTU = MTU – L2 header (Ethernet or PPP) – other headers (MPLS, GRE and so on if they exist)
Basically you should align all L2 and L3 MTU in the network in order to avoid problems, caused by MTU mismatch. MTU mismatch arises when MTU of transmitter is bigger than MRU of receiver. In such situation such frame is silently dropped.
Another problem relates to the case when MTU is different at different links. In such situation the transmitter should send the frame that satisfy the lowest MTU following the whole path. If it’s bigger, then two variants are possible:
- The first one is that IP traffic is being transmitted, so packets can be fragmented at the intermediate routers, which has interfaces with different MTU. This operation is done in control plane, so it uses capacity of CPU/Memory what is not desirable. Fragmentation means that original packet is divided into several smaller packets, which satisfy MTU/MRU at outgoing interface.
- The second case is that the L2 traffic is being transmitted (switching environment or L2 VPN). There is no layer 2 protocol (or at least widely used protocol) that supports fragmentation. So these frames will be dropped.
The third issue with MTU is that MTU mismatch usually leads to the problems with routing protocol (mainly interior as OSPF or ISIS), because they have built it mechanisms to check MTU at both sides of link by sending this number in OSPF hello message or using padding of Hello packets in ISIS up to MTU size at interface. Though these checks can be disabled, it’s better not to do it, because they are your additional helpers to discover and fix MTU mismatch.
MTU in Nokia (Alcatel-Lucent) SR OS
There are a lot of conditions, which influences the MTU size at certain interface. For people, who works with Nokia (Alcatel-Lucent) SR 7750 for a long time, or who attends Nokia’s Service Architecture courses, it might looks easy, but for the rest of engineers this isn’t so obvious. Let’s dig onto details.
There are two different type of ports in Nokia (Alcatel-Lucent):
- Network port is used for connectivity with core network (PE-PE or PE-P). If the interface has bandwidth of 1 Gbps or bigger, then its default MTU is 9212 bytes (and it is maximum possible value). Otherwise it will be 1514 bytes
- Access port is used for customer’s connectivity. Access port is crucial component of all types of client services: L2 VPN (E-Line and E-LAN), L3 VPN (BGP MPLS VPN) and internet access (called IES – Internet Enhanced Service in Nokia (Alcatel-Lucent) Architecture), because only at access ports it’s possible to create SAP (Service Attach Point that is logical customer port). Default MTU is 1514, but it depends on interface encapsulation type.
As I’m using Nokia (Alcatel-Lucent) VSR, not physical SR 7750, I have another default MTU at Gigabit Ethernet port.
Let’s see what’s going on in the console:
|
*A:SR1>config>port# info |
As you remember (link to first article), we have to create a logical interface as it’s separated instance:
|
*A:SR1# configure router interface MTUTEST |
So IP MTU is 14 bytes lower as it should be according to the formula above.
Now let’s create a dot1q interface. To do it we need to delete the existing one, because we have to change encapsulation type of our port at Nokia (Alcatel-Lucent) SR OS router:
|
*A:SR1# configure router interface MTUTEST |
You see that that initial MTU hasn’t changed, meaning that if we have maximum size, but it’s an IP MTU that has changed. It’s 4 bytes lower than it was with encapsulation null. Point out that it is impossible to configure qinq encapsulation at network port.
Network port logic is clear: it has maximum MTU by default and then lowers IP MTU depending on encapsulation type.
Now let’s shift to access port a little bit. To see it I’ll configure something called IES (Internet Enhanced Services). I won’t explain in details this time, but technically it’s just a layer 3 interface places at access port. So we change configuration in the following way:
|
*A:SR1# configure port 1/1/1 |
As soon as you change interfaces type, MTU is set up to 1514 automatically. IP MTU at newly created interface follows standard rule to be 14 bytes less than MTU due to Ethernet header. Now let’s change type to dot1q:
|
*A:SR1# configure port 1/1/1 |
Here you see that after change of encapsulation type to dot1q the MTU is changed to 1518 at port, whereas the IP MTU remains the same (1500 bytes).
The last case for Nokia (Alcatel-Lucent) SR OS will be configuring of encapsulation to qinq at SAP:
|
*A:SR1# configure port 1/1/1 |
Again, you see that IP MTU remains 1500 bytes, while port MTU has increased to accept the overall frame of 1522 bytes size (14 bytes Ethernet header + 4 byte 1st VLAN tag + 4 byte 2nd VLAN tag).
MTU in Cisco IOS XR
Now it’s turn of Cisco IOS XR to show how it behaves. All interfaces at Cisco routers (not only IOS XR as ASR 9000, but IOS / IOS XR routers as well) has MTU 1514 at physical interface. Let’s take a look at newly created interface:
|
RP/0/0/CPU0:XR1#sh run int gig 0/0/0/0 |
So you see that MTU (Layer 2 MTU) is 1514 bytes, which corresponds to standard size Ethernet frame without tags. We haven’t configured any MTU parameters, so these values are defaults. Now we’ll see how Cisco IOS XR deals with 802.1Q frames.
In order to create dot1q encapsulation at port, you should just create sub-interface and map it no certain VLAN value. You don’t have to change encapsulation type, as you did it in Nokia (Alcatel-Lucent) SR OS:
|
RP/0/0/CPU0:XR1(config)#show conf |
For more references, read this article about configuration of trunks.
At a glance it might look quite strange, as you don’t change MTU parameter at interface (I mean interface as a physical instance), but it’s automatically increased at sub-interfaces. Let’s create QinQ sub-interface at Cisco IOS XR:
|
RP/0/0/CPU0:XR1(config)#show conf |
In general Cisco IOS XR use the following rule regarding MTU:
It adds the necessary amount of bytes to initial MTU of physical interfaces depending on sub-interface encapsulation (4 bytes for dot1q encapsulation, and 8 bytes for QinQ encapsulation).
Harmonization of MTU between Nokia (Alcatel-Lucent) SR OS and Cisco IOS XR
You see that default parameter of MTU in Nokia (Alcatel-Lucent) SR OS and Cisco IOS XR are different. Then the obvious question arises: “What to do further?” The answer is straightforward as well: “Configure MTU with the same value in Cisco IOS XR and Nokia (Alcatel-Lucent) SR OS”. But before doing it consider two key points, which come from the explanation above:
- Cisco IOS XR will always add some bytes to configured MTU of the port (4 bytes for dot1q sub interface and 8 bytes for qinq sub interface)
- Nokia (Alcatel-Lucent) SR OS uses configure MTU parameter as maximum value including all overheads.
This rule makes you to define, for which type of MTU you want to align MTU. It’s impossible to align null, dot1q and qinq together, just because Nokia (Alcatel-Lucent) SR OS and Cisco IOS XR calculates MTU in a different way. Let’s review very simple topology and make some configurations:
Let’s assume we want to send frames with MTU 2014 bytes (IP MTU 2000 bytes) in ordinary Ethernet frame without any additional encapsulation:
| Nokia (Alcatel-Lucent) VSR (SR 7750) | Cisco IOS XR (ASR 9000) |
| SR1 | XR1 |
|
*A:SR1# configure port 1/1/1 |
RP/0/0/CPU0:XR1(config)#show conf |
The configuration is really simple as you see above. Let’s compare the MTU and IP MTU parameters at interfaces. Nokia (Alcatel-Lucent) SR OS has the following:
|
A:SR1# show port 1/1/1 | match expression “MTU|Encap” |
Cisco IOS XR has the same parameters:
|
RP/0/0/CPU0:XR1#show int gig 0/0/0/0 | inc MTU |
And now I want to show you one difference in ping tool between Nokia (Alcatel-Lucent) SR OS and Cisco IOS XR. Nokia side:
|
A:SR1# ping 10.0.0.2 do-not-fragment size 1972 count 1 |
Cisco side:
|
RP/0/0/CPU0:XR1#ping 10.0.0.1 df-bit size 2000 |
You see that packet size for ICMP packet, which we configure in CLI, is different in Nokia (Alcatel-Lucent) SR OS and Cisco IOS XR:
- In Nokia (Alcatel-Lucent) SR OS you set size of payload of ICMP packet for ping, which is then appended by IP header (20 bytes) and ICMP header (8 bytes).
- In Cisco IOS XR you set size of the whole ICMP packet including IP and ICMP headers
Well, with this encapsulation null there is no hints. Let’s now configure the link between SR1 and XR1 in such a way to pass packets with IP MTU 2000, but the frames will have one 802.1q tag:
| Nokia (Alcatel-Lucent) VSR (SR 7750) | Cisco IOS XR (ASR 9000) |
| SR1 | XR1 |
|
*A:SR1# configure port 1/1/1 |
RP/0/0/CPU0:XR1(config)#show conf |
In Nokia (Alcatel-Lucent) SR OS we have to remove any mapping of physical port to logical instances (interfaces and SAP) before we can change encapsulation type of port.
As you see, we don’t change MTU at Cisco IOS XR. The reason for that is the rule, which we’ve described above. Let’s check that it’s so. Cisco IOS XR side:
|
RP/0/0/CPU0:XR1#show int gig 0/0/0/0.10 | inc MTU |
Now it’s time of the Nokia (Alcatel-Lucent) router:
|
A:SR1# show port 1/1/1 | match expression “MTU|Encap” |
Upon performing test ping, we see the same result as it has been above.
When you change MTU, don’t forget to reset port (shutdown / no shutdown).
Lessons learned
First I met this problem during my article writing in IPv4/OSPF article, when my routers failed to establish adjacency. Then I’ve seen this problem in the ISIS as well. In L2VPN topic MTU issue is even more important, so we’ll review it when we speak about them.
Conclusion
MTU is one of the critical parameter in the network, and frankly it’s one of the worst known thing by network engineers. Because it’s really simple thing and usually nobody cares about it, because it works by default. If engineers change it, usually they change it end-to-end in the network and also don’t care about it. But as you see, different vendors have different approach in calculation and configuration of L2/L3 MTU at ports, what may lead to traffic loss in the network, if it isn’t cared well. Though there is standardized mechanism to find the lowest MTU at link (path MTU discovery – PMTUD), it often fails due to security measures in the network. Take care and good bye!
Support us
BR,
Anton Karneliuk