Hello my friend,
some time ago we’ve been recently engaged in the troubleshooting of the performance issues. Namely, the speed of the communication between the application’s endpoints in two data centres was not persistent. Instead, it was deviating a lot having multiple TCP retransmissions for certain flows. The issues was successfully solved, and we’d like to share with you the tools we have used to identify and validate various aspects of traffic forwarding.
2
3
4
5
retrieval system, or transmitted in any form or by any
means, electronic, mechanical or photocopying, recording,
or otherwise, for commercial purposes without the
prior permission of the author.
Can automation help with performance troubleshooting?
Absolutely. During our network automation training we show how to utilise various Linux tools from configuration management tools (e.g. Ansible) and programming languages (Bash, Python). That gives you ready examples from our training, which you can use in your network immediately, and endless possibilities to create your own automated troubleshooting workflows.
Our network automation training has two faces: either live or self-paced. So you can choose yourself, what works better for you. On our side, we guide you from the foundation of the automation for the small networks till advanced automation use cases in big data centres, service providers, and clouds. You will lean how to structure the data using YANG modules, how to serialise it using JSON, XML, Protocol Buffers (Protobuf) depending on the application requirements and how to manage the fleet of your IT and network workloads relying gRPC, gNMI, NETCONF, REST API and many others with Bash, Ansible and Python. In addition to that you will master the solid foundation in Linux administration, Linux networking, network function virtualisation (NFV) with KVM and containerisation with Docker. That is all you need to be successful in network automation in one place.
Top 5 tools
The chosen tools covers the various aspects of the network performance troubleshooting and helps identifying the related problems. Here they are:
- speedtest
- iperf
- tcpdump
- tshark
- mtr
Each of these tools is unique in terms that each of them has its own applicability, which is only slightly overlaps with others (if any). How have we chosen tools? There were numerous criteria for that:
- They must be working from any Linux platform from CLI.
- They must be documented good enough so that we can use them.
- They must provide good and persistent results to be reliable.
- They must have functionality to undercover various aspects of the network performance at a deep level.
Let’s take a look into each tool.
#1. SpeedTest
Where does the troubleshooting typically start? It starts, when someone complains about the speed of the internet access or its stability. However, both of the characteristics are quite subjective. And here is where the speed test by Ookla could help you. It allows to measure the three key characteristics:
- Download speed from the Ookla test server
- Upload speed to the Ookla test server
- Ping response time towards
The Ookla test server is chosen automatically based on your IP address geo information: the closest server to you is chosen. However, there is a possibility to chose the server based on its ID, so that you can measure even the performance towards another country or part of the world what generally can show how the Internet infrastructure is working including both your ISP and all others ISPs along the path.
Availability: Needs to installed; can be installed on any Linux distribution
Where you can find it: https://www.speedtest.net/apps/cli
#2. iPerf
Say, you have tested the connectivity to an Ookla server, and it is OK. Or, you are not interested in a generic Internet speed, but rather want to check connectivity between your servers. The iPerf is definitely tool for you to go. It allows you to test:
- Speed of the connectivity from the client to the server.
- Speed of the connectivity from the server to the client.
- Speed of the TCP or UDP streams.
- Amount of retransmissions (for TCP) or packet drops (for UDP).
- Speed of the single or multiple streams using custom source port.
Both the server and the client are your servers running iPerf just in different modes, so you decide on your own what to test and how to test. Various tuning opportunities (random source port vs. defined source port, UDP vs. TCP, single stream vs. multiple streams) gives you possibility to simulate your application as close as possible in terms of the network behaviour, what is very important by troubleshooting.
Availability: Needs to installed; can be installed on any Linux distribution
Where you can find it: https://iperf.fr
#3. MTR
Once you figured out the performance between your endpoints, especially if that is not good enough, you might be interested in identifying, where the traffic might be dropped in the path. That’s exactly where the MTR can help. MTR stands for My Trace Route and, in fact, an enhanced version of the classical trace route tool. It has the following advantage over the original one:
- Shows the packet loss per each hop if any.
- Shows the path end to end much quicker than trace route.
- Shows the path in dynamic. If the drops per hop increase or decrease, it will be reflected in the output.
You can test the path using MTR towards any destination: it can be both target under your control or any arbitrary IP address.
Availability: Needs to installed; can be installed on any Linux distribution
Where you can find it: https://linux.die.net/man/8/mtr
#4. TCPDump
Another aspect of the performance (and not only) troubleshooting, is to see what are the packets being received or sent by your endpoint. That is useful both for the analysis in real time and for saving them in file, which can be viewed later on using a specific packets analysers. There is a tool, which is built-in in all the Linux distributions I’ve worked with (Ubuntu, Debian, CentOS, Cumulus Linux) and this tool is called tcpdump. It allows you:
- View in the real time all the packets coming or leaving the network element overall or through a particular interface.
- Filter the output so that only specific packets (e.g. based on specific protocol, or port ,or IP address) are visible.
- Save the packets into a file for further analysis instead of online viewing.
The tcpdump can get you immediate results: you can see whether your host receives/sends the traffic it is supposed to or not. If not, then you there might be something in the path or with the access-lists somewhere. If yes, the further analysis might be needed, such as analysis of flags/sequences for TCP stream and so on.
Availability: available by default in many Linux; can be installed on any Linux distribution
Where you can find it: https://www.tcpdump.org/manpages/tcpdump.1.html
#5. TShark
Everyone uses the Wireshark for the analysis of the packets, where the troubleshooting is getting more complex. What if you need to get the same functionality directly in the CLI? Terminal-based Wireshark (TShark) is built exactly for this purpose:
- It can read the packets directly from the wire (as the tcpdump) or from the file.
- It can apply complex filters, to figure the packet types or analyse the flows, as Wireshark is doing.
Being the Wireshark, the TShark has all the capabilities and the power of the parent product, just in the CLI format.
Availability: Needs to installed; can be installed on any Linux distribution
Where you can find it: https://www.wireshark.org/docs/man-pages/tshark.html
Take me to the examplesLessons learned
Despite you may see that some of the tools might be overlapping, all of them are very useful. In reality you would combine them in various mixes to troubleshoot the network related issues, such as performance, to find out the root cause. All of these tools have a tremendous amount of possible options how they can be applied. So spend some time on looking into the documentation.
Conclusion
Today we just scratched the surface of the performance troubleshooting. However, this topic is very important in all areas of the networking: 5G, clouds, internet infrastructure. We will cover the details of the usage of each tool in the upcoming articles. Take care and good bye.
Support us
P.S.
If you have further questions or you need help with your networks, I’m happy to assist you, just send me message. Also don’t forget to share the article on your social media, if you like it.
BR,
Anton Karneliuk
