Hello my friend,
after show pause we continue our blog series about the most popular network troubleshooting tools, which humankind has ever created. Today we’ll take a look at one of the most useful tool to obtain the information about the path between two endpoints and possible packet drops over that path. Ladies and gentlemen, please, welcome MTR.
2
3
4
5
retrieval system, or transmitted in any form or by any
means, electronic, mechanical or photocopying, recording,
or otherwise, for commercial purposes without the
prior permission of the author.
Can automation help with figuring what happened where?
In case of the infrastructure problems (networks, servers, VMs, containers), the time matters a lot. The quicker we can find the issue and fix that, the better it will be for our applications and our customers. Automation without doubts one of the key components, which allows you to quickly find and fix your issues.
In our trainings, the Live Network Automation Training (10 weeks) and Automation with Nornir (2 weeks), we explore a lot of real use cases, where the automation helps you to validate the state of you network and change it if necessary. You will learn the whole spectre of the automation approaches starting from the text-based automation used in the hyper scalers with the full configuration templated till the model-driven automation with NETCONF/RESTCONF/gNMI loaded with YANG data models for Cisco, Nokia, Arista and Cumulus. This knowledge comes with a lot of different exercises with direct console as well as Ansible/Python and Bash scripts. On top to that, you learn a lot of infrastructure skills such as building and managing Linux, KVM and Docker.
Moreover, we are running our trainings for 2 years already and constantly adopting them to the changes happening in the automation world. Master your automation skills with us.
Brief description
The changes in the networks, especially in such complex as internet, are happening constantly: routers and links are being built and decommissioned, being out of production for maintenance activity or due to outage, or new more powerful and highly-performant added, what leads to the congestions and partial loss or even complete disruption of forwarding. In the same time, if we don’t know where the traffic being dropped, it is impossible to perform the corresponding troubleshooting. This is where the MTR comes to the stage.
MTR stands for My Trace Route, which is a purpose-built tool, which significantly extend the capabilities of the built-in traceroute in Linux/MacOS or tracert in Windows. Namely, the MTR allows you collect the following information:
- In a matter of a few seconds mtr allows you to figure out the exact path between the source (host running MTR) and destination.
- It also shows you the packet loss on each hop and the latency figures.
- Various protocols can be used to measure the path (ICMP, UDP, TCP).
- Single tool supports both IPv4 and IPv6 addresses as well as FQDN.
Some online diagnostic tools, for example this one, incorporates results of MTR in their workflow.
In fact, almost any troubleshooting of WAN connections, if the WAN is provided by the 3rd party providers starts with the several MTR outputs. As such, it is difficult to overestimate the MTR’s value to the infrastructure (network, server, virtualisation) engineers.
Usage
Let’s take a look how you can benefit from using mtr.
#1. Installation
The mtr is available via package manager on all the *nix (Linux, Unix, Mac OS) platforms. Depending on the package manager available the installation the installation may vary.
For the Ubuntu/Debian (example from Raspberry PI):
For CentOS/RHEL 7 (replace yum to dnf for Fedora or CentOS 8):
For Mac OS it would depend on the installed package manager. For example, if you use brew, it would be:
However, the further process on Mac OS is a bit tricky. You need to copy the installed application (assuming you have version 0.94, or change the version) to a folder with other applications on your MAC:
2
$ cp /usr/local/Cellar/mtr/0.94/sbin/mtr-packet /usr/local/bin/
On Mac OS you need to run the mtr command with sudo permissions.
The installation is very quick and right after it is completed, you can start with your tests.
#2. Simple tests
In the simplest scenario, the execution of the mtr tests is very simple. You just need to type the mtr address or FQDN, where address is the IPv4 or IPv6 address of the host you are willing to test.
IPv4 test:
2
3
4
5
6
7
8
probe-0000 (192.168.1.236) 2021-03-31T00:00:27+0100
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. Communityzxxxxx.cust.communityfibre.co.uk 0.0% 7 2.1 2.7 2.1 3.5 0.5
2. 100.64.144.1 0.0% 7 5.8 4.8 4.2 5.8 0.7
3. one.one.one.one 0.0% 6 6.3 5.8 5.0 7.6 1.0
The output of the MTR consists of a semi formatted table having the following columns:
| Category | Sub-category | Meaning |
|---|---|---|
| Host | – | The ingress IP host along the path (ingress IP interfaces on router, server, firewall, etc). Can be expressed as IPv4 or IPv6 address or FQDN entry. |
| Packets | Loss% | Percentage of the packets lost at each particular hop |
| Packets | Sent | Amount of the packets sent (constantly increasing as the packets are being sent each 1 second) |
| Pings | Last | The latency of the last measurement to the particular hop |
| Pings | Avg | Average latency to the particular hop across all the results |
| Pings | Wrst | The worst latency result to the particular hop across all the results |
| Pings | StDev | The standard deviation between the average and best/worst results |
The rows inside the results of mtr are the hops themselves.
The same structure of the table is for other types of the addresses. For example, here is the structure for IPv6 trace route:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
probe-0000 (some_ipv6_address) 2021-03-31T00:06:46+0100
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. 2a02:6b6d:a02a:0:ea9f:80ff:fe4c:d90d 0.0% 22 2.1 4.4 1.9 11.0 2.6
2. 2a02:6b68:0:12::1 0.0% 22 15.7 7.4 3.9 16.9 4.0
3. 2a02:6b60:0:1:1::1a 0.0% 22 52.5 8.7 4.3 52.5 10.0
4. lag-3.agg-rr1.tab.lon.network.as201838.net 66.7% 22 4.6 7.1 4.6 11.8 2.6
5. ???
6. lag-1.edge1.thn.lon.network.as201838.net 95.0% 21 5.0 5.0 5.0 5.0 0.0
7. 2001:4860:1:1::814 0.0% 21 20.2 7.2 4.4 20.2 3.9
8. 2a00:1450:8135::1 0.0% 21 61.6 8.1 3.2 61.6 12.5
9. 2001:4860:0:1::54c6 0.0% 21 6.9 6.1 3.2 12.7 2.7
10. 2001:4860:0:1::1d87 0.0% 21 9.7 5.1 3.2 10.1 2.1
11. lhr25s12-in-x0e.1e100.net 0.0% 21 54.0 6.5 2.7 54.0 10.9
Provided IPv6 address is one of the public IPv6 addresses of Google.com.
FQDN test:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
probe-0000 (some_ipv6_address) 2021-03-31T00:08:34+0100
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. 2a02:6b6d:a02a:0:ea9f:80ff:fe4c:d90d 0.0% 36 3.9 3.1 1.7 8.1 1.4
2. 2a02:6b68:0:12::1 0.0% 35 12.3 6.9 3.8 15.9 2.7
3. 2a02:6b60:0:1:1::1a 0.0% 35 9.4 6.9 3.9 13.2 2.5
4. lag-3.agg-rr1.tab.lon.network.as201838.net 37.1% 35 4.7 6.3 3.8 12.0 2.1
5. xge-0-2-5.ar03.thn.lon.network.as201838.net 85.7% 35 7.5 6.8 4.4 12.3 3.3
6. lag-1.edge1.thn.lon.network.as201838.net 82.9% 35 12.1 7.0 3.9 12.1 3.2
7. 2001:4860:1:1::814 0.0% 35 5.3 7.4 4.0 16.3 3.4
8. 2a00:1450:8087::1 0.0% 35 4.5 6.2 4.1 13.7 2.6
9. 2001:4860:0:1::327c 0.0% 35 5.0 5.8 3.8 12.9 1.9
10. 2001:4860:0:135d::13 17.1% 35 5.5 6.4 4.4 12.1 2.0
11. 2001:4860::c:4002:1123 0.0% 35 8.8 6.3 5.2 9.0 1.0
12. 2001:4860::9:4001:45d3 5.7% 35 9.8 10.9 4.1 61.4 12.2
13. 2001:4860:0:1101::1 20.0% 35 5.9 7.0 5.2 14.8 2.2
14. 2001:4860:0:1::41dd 8.6% 35 5.9 7.2 4.8 14.5 2.7
15. lhr25s27-in-x0e.1e100.net 0.0% 35 4.5 5.2 4.0 10.7 1.2
Already at this stage, without the extra arguments to the mtr, you may see the benefits of using this tool.
#3. Advanced tests
Let’s take a look at some extra arguments, you can pass to the mtr tool. First of all, especially if the destination host has dual stack IP addresses (IPv4 and IPv6), you can specify, which IP address you want to reach by adding either key -4 or -6:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
My traceroute [v0.92]
probe-0000 (192.168.1.236) 2021-04-02T11:34:34+0100
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. Communityxxxxx.cust.communityfibre.co.uk 0.0% 5 1.7 2.4 1.7 4.9 1.4
2. 100.64.144.1 0.0% 5 4.4 4.6 3.5 6.4 1.1
3. lhr25s14-in-f14.1e100.net 0.0% 5 3.8 4.8 3.2 6.4 1.4
pi@probe-0000:~ $ mtr google.com -6
My traceroute [v0.92]
probe-0000 (some ipv6 address) 2021-04-02T11:35:37+0100
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. 2a02:6b6d:a02a:0:ea9f:80ff:fe4c:d90d 0.0% 19 3.0 3.1 1.7 8.2 1.7
2. 2a02:6b68:0:12::1 0.0% 18 4.1 6.1 4.1 11.9 2.2
3. 2a02:6b60:0:1:1::1a 0.0% 18 5.4 5.7 4.2 9.7 1.6
4. lag-3.agg-rr1.tab.lon.network.as201838.net 47.1% 18 3.9 5.7 3.9 8.8 1.6
5. xge-0-2-5.ar03.thn.lon.network.as201838.net 76.5% 18 4.1 7.0 3.6 15.5 5.7
6. lag-1.edge1.thn.lon.network.as201838.net 94.1% 18 5.3 5.3 5.3 5.3 0.0
7. 2001:4860:1:1::814 0.0% 18 6.1 5.2 3.7 9.8 1.5
8. 2a00:1450:8165::1 0.0% 18 3.5 5.0 3.1 7.8 1.7
9. 2001:4860:0:1::54d0 27.8% 18 5.9 5.0 4.0 7.2 0.9
10. 2001:4860:0:1::41b 0.0% 18 6.3 4.0 3.2 6.3 0.8
11. lhr25s11-in-x0e.1e100.net 0.0% 18 4.0 4.9 3.4 7.9 1.2
The next logical step would be see the exact IP addresses, as you can see, that the current approach is to resolve DNS names where possible and to use IP addresses, where not. To have the output systematic, you have possibility to add key -n to exclude the DNS resolution from the process at all or -b to show both IPv4/IPv6 addresses and DNS names:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
My traceroute [v0.92]
probe-0000 (some ipv6 address) 2021-04-02T11:44:49+0100
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. 2a02:6b6d:a02a:0:ea9f:80ff:fe4c:d90d 0.0% 30 2.3 3.4 1.7 10.0 2.0
2. 2a02:6b68:0:12::1 0.0% 30 4.8 5.2 3.9 10.2 1.3
3. 2a02:6b60:0:1:1::1a 0.0% 30 5.0 5.4 3.6 10.6 1.8
4. 2a02:6b60:0:1:1::a 36.7% 30 3.5 5.0 3.5 9.0 1.6
5. 2a02:6b60:0:1:1::34 89.7% 30 4.1 5.0 4.1 5.7 0.8
6. 2a02:6b60:0:1:2::53 66.7% 30 7.9 5.1 3.9 7.9 1.3
7. 2001:4860:1:1::814 0.0% 30 3.8 5.3 3.8 8.1 1.3
8. 2a00:1450:8165::1 0.0% 30 6.5 4.5 3.2 10.9 1.7
9. 2001:4860:0:1::54ce 0.0% 30 3.4 4.2 3.1 11.1 1.5
10. 2001:4860:0:1::41ab 20.0% 30 4.6 5.9 3.9 12.2 2.3
11. 2a00:1450:4009:80b::200e 0.0% 29 4.7 4.5 2.7 8.2 1.4
pi@probe-0000:~ $ mtr google.com -6 -b
My traceroute [v0.92]
probe-0000 (some ipv6 address) 2021-04-02T11:45:52+0100
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. 2a02:6b6d:a02a:0:ea9f:80ff:fe4c:d90d (2a02:6b6d:a02a:0:ea9f:80ff:f 0.0% 40 3.8 2.9 1.6 6.9 1.3
2. 2a02:6b68:0:12::1 (2a02:6b68:0:12::1) 0.0% 39 4.4 5.8 4.0 12.2 1.9
3. 2a02:6b60:0:1:1::1a (2a02:6b60:0:1:1::1a) 0.0% 39 4.0 6.0 4.0 15.1 2.3
4. lag-3.agg-rr1.tab.lon.network.as201838.net (2a02:6b60:0:1:1::a) 36.8% 39 4.1 4.5 3.5 7.0 0.9
5. ???
6. lag-1.edge1.thn.lon.network.as201838.net (2a02:6b60:0:1:2::53) 59.0% 39 3.6 4.4 3.4 7.1 1.1
7. 2001:4860:1:1::814 (2001:4860:1:1::814) 0.0% 39 6.1 5.1 3.6 10.0 1.2
8. 2a00:1450:8136::1 (2a00:1450:8136::1) 0.0% 39 3.4 4.3 3.0 9.6 1.5
9. 2001:4860:0:1101::1 (2001:4860:0:1101::1) 33.3% 39 4.9 6.2 4.3 34.4 5.8
10. 2001:4860:0:1101::f (2001:4860:0:1101::f) 23.1% 39 4.4 5.4 3.8 8.6 1.5
11. 2001:4860::c:4002:1123 (2001:4860::c:4002:1123) 0.0% 39 5.8 14.6 5.0 104.6 24.8
12. 2001:4860::9:4001:2717 (2001:4860::9:4001:2717) 30.8% 39 6.9 6.3 4.4 17.0 2.5
13. 2001:4860:0:1::3173 (2001:4860:0:1::3173) 0.0% 39 5.0 5.1 3.5 9.1 1.2
14. lhr48s11-in-x0e.1e100.net (2a00:1450:4009:81a::200e) 0.0% 39 4.3 5.0 3.6 10.5 1.5
The next aspect is to change the protocol used to trace the path. The default is ICMP, but it is possible to use UDP by adding argument -u or TCP by adding -T:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
probe-0000 (some ipv6 address) 2021-04-02T12:08:03+0100
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. 2a02:6b6d:a02a:0:ea9f:80ff:fe4c:d90d (2a02:6b6d:a02a:0:ea9f:80ff:f 0.0% 39 3.3 2.9 1.7 6.5 1.3
2. 2a02:6b68:0:12::1 (2a02:6b68:0:12::1) 0.0% 39 48.7 6.6 4.1 48.7 7.1
3. 2a02:6b60:0:1:1::1a (2a02:6b60:0:1:1::1a) 0.0% 38 5.1 5.1 3.4 7.4 0.8
4. lag-3.agg-rr1.tab.lon.network.as201838.net (2a02:6b60:0:1:1::a) 59.5% 38 4.8 4.9 3.8 7.5 1.0
5. xge-0-2-5.ar03.thn.lon.network.as201838.net (2a02:6b60:0:1:1::34) 97.3% 38 5.2 5.2 5.2 5.2 0.0
6. lag-1.edge1.thn.lon.network.as201838.net (2a02:6b60:0:1:2::53) 76.3% 38 28.4 7.3 4.0 28.4 7.9
7. 2001:4860:1:1::814 (2001:4860:1:1::814) 0.0% 38 70.9 7.2 4.2 70.9 10.7
8. 2a00:1450:8103::1 (2a00:1450:8103::1) 83.8% 38 4.8 4.2 2.9 5.6 0.9
2a00:1450:8165::1 (2a00:1450:8165::1)
2a00:1450:8088::1 (2a00:1450:8088::1)
2a00:1450:8135::1 (2a00:1450:8135::1)
2a00:1450:8087::1 (2a00:1450:8087::1)
9. 2001:4860:0:1::54d2 (2001:4860:0:1::54d2) 86.5% 38 4.2 4.0 3.5 4.4 0.3
2001:4860:0:1::41aa (2001:4860:0:1::41aa)
2001:4860:0:1::54ca (2001:4860:0:1::54ca)
2001:4860:0:1::418 (2001:4860:0:1::418)
2001:4860:0:1::1e02 (2001:4860:0:1::1e02)
10. 2001:4860:0:135e::12 (2001:4860:0:135e::12) 83.8% 38 4.1 4.3 3.8 4.7 0.3
2001:4860:0:1::54cf (2001:4860:0:1::54cf)
11. lhr25s33-in-x0e.1e100.net (2a00:1450:4009:81f::200e) 78.4% 38 3.9 4.6 3.7 6.0 0.9
2001:4860::c:4002:1123 (2001:4860::c:4002:1123)
pi@probe-0000:~ $ mtr google.com -6 -b -u
probe-0000 (some ipv6 address) 2021-04-02T12:08:50+0100
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. 2a02:6b6d:a02a:0:ea9f:80ff:fe4c:d90d (2a02:6b6d:a02a:0:ea9f:80ff:f 0.0% 28 3.0 3.0 1.7 8.1 1.4
2. 2a02:6b68:0:12::1 (2a02:6b68:0:12::1) 0.0% 28 4.9 5.5 4.3 8.5 1.0
3. 2a02:6b60:0:1:1::1a (2a02:6b60:0:1:1::1a) 0.0% 28 5.7 5.1 3.8 7.5 0.7
4. lag-3.agg-rr1.tab.lon.network.as201838.net (2a02:6b60:0:1:1::a) 50.0% 28 4.9 454.8 3.9 3160. 1142.
5. xge-0-2-5.ar03.thn.lon.network.as201838.net (2a02:6b60:0:1:1::34) 85.2% 28 7235. 2327. 4.8 7235. 3308.
6. lag-1.edge1.thn.lon.network.as201838.net (2a02:6b60:0:1:2::53) 57.1% 28 5.2 604.7 3.7 7205. 2078.
7. 2001:4860:1:1::814 (2001:4860:1:1::814) 0.0% 27 5.6 5.3 4.2 9.3 1.2
8. 2001:4860:0:1::781 (2001:4860:0:1::781) 0.0% 27 4.5 1033. 3.4 7195. 2013.
2001:4860:0:1::783 (2001:4860:0:1::783)
9. 2001:4860:0:1::54cf (2001:4860:0:1::54cf) 0.0% 27 4.7 5.1 3.3 8.2 1.2
2001:4860:0:1::54d1 (2001:4860:0:1::54d1)
10. lhr25s33-in-x0e.1e100.net (2a00:1450:4009:81f::200e) 0.0% 27 4.6 4.7 3.6 7.4 1.0
For the UDP/TCP you can also set the destination port using -P argument, whereas you can also specify the source port with -L:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
My traceroute [v0.92]
probe-0000 (some ipv6 address) 2021-04-02T12:33:18+0100
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. 2a02:6b6d:a02a:0:ea9f:80ff:fe4c:d90d (2a02:6b6d:a02a:0:ea9f:80ff:f 0.0% 14 2.7 6.2 1.8 55.6 14.2
2. 2a02:6b68:0:12::1 (2a02:6b68:0:12::1) 0.0% 14 4.6 20.5 4.5 127.5 36.1
3. 2a02:6b60:0:1:1::1a (2a02:6b60:0:1:1::1a) 0.0% 14 4.3 7.1 4.2 31.8 7.2
4. lag-3.agg-rr1.tab.lon.network.as201838.net (2a02:6b60:0:1:1::a) 69.2% 14 7219. 4635. 1010. 7219. 3088.
5. ???
6. lag-1.edge1.thn.lon.network.as201838.net (2a02:6b60:0:1:2::53) 92.9% 14 5.8 5.8 5.8 5.8 0.0
7. 2001:4860:1:1::814 (2001:4860:1:1::814) 0.0% 14 7.4 5.4 4.6 7.4 0.8
8. 2001:4860:0:1::781 (2001:4860:0:1::781) 7.7% 14 1050. 1044. 3.5 3143. 1329.
2001:4860:0:1::783 (2001:4860:0:1::783)
9. 2001:4860:0:1::54d3 (2001:4860:0:1::54d3) 0.0% 14 4.5 12.6 3.5 113.5 29.1
2001:4860:0:1::54d5 (2001:4860:0:1::54d5)
10. lhr25s34-in-x0e.1e100.net (2a00:1450:4009:820::200e) 0.0% 14 4.2 5.2 3.5 13.9 2.7
The last aspect, we would cover is the reporting. By default the mtr is an interactive tool; however, it is possible to make sure the report is produced in non-interactive way using the -r key:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
Start: 2021-04-02T12:35:52+0100
HOST: probe-0000 Loss% Snt Last Avg Best Wrst StDev
1.|-- 2a02:6b6d:a02a:0:ea9f:80f 0.0% 10 1.8 2.9 1.6 7.0 1.6
2.|-- 2a02:6b68:0:12::1 0.0% 10 5.6 5.5 4.2 7.2 1.0
3.|-- 2a02:6b60:0:1:1::1a 0.0% 10 4.9 5.7 3.6 9.9 1.9
4.|-- lag-3.agg-rr1.tab.lon.net 20.0% 10 4.8 5.0 3.8 6.2 0.8
5.|-- xge-0-2-5.ar03.thn.lon.ne 90.0% 10 6.1 6.1 6.1 6.1 0.0
6.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
7.|-- 2001:4860:1:1::814 0.0% 10 4.7 5.7 3.9 10.3 2.0
8.|-- 2a00:1450:8088::1 0.0% 10 4.3 4.3 3.3 8.9 1.7
9.|-- 2001:4860:0:1::1d86 0.0% 10 66.8 12.6 3.4 66.8 19.6
10.|-- 2001:4860:0:1100::f 40.0% 10 4.1 4.8 4.1 6.1 0.8
11.|-- 2001:4860::c:4000:dd7a 80.0% 10 5.4 11.5 5.4 17.7 8.7
12.|-- 2001:4860::9:4001:2716 20.0% 10 4.1 10.7 4.1 38.5 12.2
13.|-- 2001:4860:0:135d::1 0.0% 10 64.3 11.3 4.3 64.3 18.7
14.|-- 2001:4860:0:1::327d 0.0% 10 5.2 5.1 4.0 6.6 0.9
15.|-- lhr48s23-in-x0e.1e100.net 0.0% 10 62.7 10.6 4.0 62.7 18.3
As you can see, some information about IP address is truncated, as there is more text, than can be rendered in a standard terminal width. You can fix that using the -w:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
Start: 2021-04-02T12:38:45+0100
HOST: probe-0000 Loss% Snt Last Avg Best Wrst StDev
1.|-- 2a02:6b6d:a02a:0:ea9f:80ff:fe4c:d90d 0.0% 10 2.3 2.5 1.6 4.4 0.9
2.|-- 2a02:6b68:0:12::1 0.0% 10 4.4 4.4 3.9 4.7 0.3
3.|-- 2a02:6b60:0:1:1::1a 0.0% 10 4.0 4.6 3.9 5.7 0.6
4.|-- lag-3.agg-rr1.tab.lon.network.as201838.net (2a02:6b60:0:1:1::a) 80.0% 10 6.4 5.7 5.1 6.4 0.9
5.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
6.|-- lag-1.edge1.thn.lon.network.as201838.net (2a02:6b60:0:1:2::53) 90.0% 10 4.6 4.6 4.6 4.6 0.0
7.|-- 2001:4860:1:1::814 0.0% 10 5.3 4.5 3.7 5.7 0.6
8.|-- 2a00:1450:8088::1 0.0% 10 5.5 4.1 3.0 5.5 0.9
9.|-- 2001:4860:0:1::1d86 0.0% 10 3.8 4.5 3.5 7.0 1.1
10.|-- 2001:4860:0:1100::f 30.0% 10 4.3 4.6 3.7 5.7 0.8
11.|-- 2001:4860::c:4000:dd7a 70.0% 10 4.7 4.9 4.5 5.6 0.6
12.|-- 2001:4860::9:4001:2716 0.0% 10 15.1 6.0 4.1 15.1 3.3
13.|-- 2001:4860:0:135d::1 0.0% 10 4.2 5.1 4.1 8.6 1.4
14.|-- 2001:4860:0:1::327d 0.0% 10 4.5 4.5 3.8 5.9 0.6
15.|-- lhr48s23-in-x0e.1e100.net (2a00:1450:4009:808::200e) 0.0% 10 4.4 4.6 3.9 5.6 0.6
The information looks now easy understandable for us, humans. However, if you are building any automated pipeline, you need the format to be easy parsable by programs. You can achieve that by adding the key -C to produce the CSV table:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
Mtr_Version,Start_Time,Status,Host,Hop,Ip,Loss%,Snt, ,Last,Avg,Best,Wrst,StDev,
MTR.0.92,1617363965,OK,google.com,1,2a02:6b6d:a02a:0:ea9f:80ff:fe4c:d90d,0.00,10,0,1.88,2.38,1.48,4.83,0.94
MTR.0.92,1617363965,OK,google.com,2,2a02:6b68:0:12::1,0.00,10,0,4.58,4.42,3.80,5.10,0.42
MTR.0.92,1617363965,OK,google.com,3,2a02:6b60:0:1:1::1a,0.00,10,0,4.30,4.87,4.03,6.31,0.84
MTR.0.92,1617363965,OK,google.com,4,lag-3.agg-rr1.tab.lon.network.as201838.net (2a02:6b60:0:1:1::a),40.00,10,4,3.80,4.24,3.80,4.53,0.33
MTR.0.92,1617363965,OK,google.com,5,???,100.00,10,10,0.00,0.00,0.00,0.00,0.00
MTR.0.92,1617363965,OK,google.com,6,lag-1.edge1.thn.lon.network.as201838.net (2a02:6b60:0:1:2::53),10.00,10,1,4.03,3.99,3.45,4.43,0.36
MTR.0.92,1617363965,OK,google.com,7,2001:4860:1:1::814,0.00,10,0,4.45,5.51,3.73,10.43,2.41
MTR.0.92,1617363965,OK,google.com,8,2a00:1450:80ea::1,0.00,10,0,6.21,4.22,3.06,6.21,1.01
MTR.0.92,1617363965,OK,google.com,9,2001:4860:0:1101::1,0.00,10,0,4.31,4.81,4.31,5.41,0.38
MTR.0.92,1617363965,OK,google.com,10,2001:4860:0:1101::f,40.00,10,4,3.72,5.48,3.72,10.91,2.72
MTR.0.92,1617363965,OK,google.com,11,2001:4860::c:4000:dd7a,80.00,10,8,4.10,4.31,4.10,4.51,0.29
MTR.0.92,1617363965,OK,google.com,12,2001:4860::9:4001:2716,20.00,10,2,6.37,4.90,3.82,6.37,0.90
MTR.0.92,1617363965,OK,google.com,13,2001:4860:0:135e::1,20.00,10,2,5.74,5.91,5.10,8.90,1.29
MTR.0.92,1617363965,OK,google.com,14,2001:4860:0:1::2fd9,30.00,10,3,5.04,5.34,4.96,6.33,0.51
MTR.0.92,1617363965,OK,google.com,15,lhr48s08-in-x0e.1e100.net (2a00:1450:4009:818::200e),0.00,10,0,4.27,4.94,3.80,9.57,1.78
Or JSON format using -j argument:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
{
"report": {
"mtr": {
"src": "probe-0000",
"dst": "google.com",
"tos": "0x0",
"psize": "64",
"bitpattern": "0x00",
"tests": "10"
},
"hubs": [{
"count": "1",
"host": "2a02:6b6d:a02a:0:ea9f:80ff:fe4c:d90d",
"Loss%": 0.00,
"Snt": 10,
"Last": 1.83,
"Avg": 22.49,
"Best": 1.64,
"Wrst": 115.97,
"StDev": 40.11
},
{
"count": "2",
"host": "2a02:6b68:0:12::1",
"Loss%": 0.00,
"Snt": 10,
"Last": 4.08,
"Avg": 21.50,
"Best": 4.08,
"Wrst": 107.26,
"StDev": 33.06
},
!
! Further output is truncated for brevity
In our network automation training you will learn how to deal with CSV and JSON data formats in a programmable way.
#4. Ideas for automation
As mtr has capabilities both to give the view of the path view and also produce the structured output in a format easy consumable by Python or any other programming language, it is possible to run mtr test to a particular node, if the monitoring (e.g. smoke ping) is breaking or the latency sharply increases, triggering the mtr by the monitoring system as call out action.
Take me to the examplesLessons learned
During the MTR tests I figured out that our ISP hides the IPv4 path, which is very awkward (but was an interesting discovery).
Another useful lessons learned we did, is that the MTR has some limitations on the Mac OS, such as necessity to execute the command with sudo privileges and inability to produce the JSON output:
2
mtr: invalid option -- j
Conclusion
Looking over our blogposts, we have covered the most widely used tools for active network troubleshooting used worldwide in all sort of the networks from the smallest till the biggest. In the further blogposts within this series, we will focus on the tools used for the passive troubleshooting (e.g., packet capture and analysis). Take care and good bye.
Support us
P.S.
If you have further questions or you need help with your networks, our team is happy to assist you. Just book a free slot with us. Also don’t forget to share the article on your social media, if you like it.
BR,
Anton Karneliuk