Hello my friend,
Recently we were asked, what is the safest way to store the credentials for network devices to your automation tools (e.g., the one based on Python and gNMI). Building the network automation solutions for a while, we have a good answer to you.
2
3
4
5
retrieval system, or transmitted in any form or by any
means, electronic, mechanical or photocopying, recording,
or otherwise, for commercial purposes without the
prior permission of the author.
What is the most promising network automation protocol ?
gNMI was created by Google to manage their data centres and backbone network and is widely used by other biggest companies worldwide. However, it doesn’t mean that only the big guys can benefit from that. Every company and network can get the advantage of a single protocol for the configuration, operation, and streaming telemetry in their network provided your network devices support that.
At our trainings, advanced network automation and automation with Nornir (2nd step after advanced network automation), we give you detailed knowledge of all the technologies relevant:
- Data encoding (free-text, XML, JSON, YAML, Protobuf)
- Model-driven network automation with YANG, NETCONF, RESTCONF, GNMI.
- Full configuration templating with Jinja2 based on the source of truth (NetBox).
- Best automation programming languages (Python, Bash), configuration management tools (Ansible) and automation frameworks (Nornir).
- Network automation infrastructure (Linux, Linux networking, KVM, Docker).
This training showed me the holistic picture of network automation and I know now where I need to focus the most of my time and efforts.
Network Automation Student @ Karneliuk.com
Moreover, we put all mentions technologies in the context of the real use cases, which our team has solved and are solving in various projects in the service providers, enterprise and data centre networks and systems across the Europe and USA. That gives you opportunity to ask questions to understand the solutions in-depts and have discussions about your own projects. And on top of that, each technology is provided with online demos and you are doing the lab afterwards to master your skills. Such a mixture creates a unique learning environment, which all students value so much. Join us and unleash your potential.
Brief description
In the previous blogpost about the gNMI we have shared the approach to how use the self-signed certificate to connect to the network device. However, by the way how the gNMI is implemented in the network devices nowadays, it is needed to send the credentials (username and password) when you communicate with the network element for each operation. The reason for that is the following:
- Certificate (self-signed or PKI based) validates the endpoint and allows the encrypted gRPC channel to be established between the host running the GNMI client and the network device.
- Credentials authorise the user to perform certain operations (e.g., NOC technician can only gather the operational information, whereas the network operations are allowed to make a change).
As such, both credentials and certificates are equally important for the gNMI operation. The challenge, which arises is how we can store the credentials then?
- Storing the file with the clear text credentials is huge security vulnerability.
- Storing the file with the passwords in an encrypted format requires someone to provide another password to decrypt it, so the automation requires a human intervention or we again somewhere store that-another-passowrd in a clear text.
There is a solution to break that vicious circle, though, which my brothers in arms from system administrators shared with me, which is usage of a Linux environment variables in Linux.
Usage
Let’s take a closer look how it works.
#1. Setting environment variables
First of all, what is an environment variable in Linux? In a nutshell, this is a variable which is available for you in Linux for your current session. To validate all the variables, you need to type an env command in your Linux session:
2
3
4
5
6
7
8
9
TERM_PROGRAM=Apple_Terminal
SHELL=/bin/bash
TERM=xterm-256color
TMPDIR=/var/folders/g_/qzk59n7n7m12d0k39gt2q4z00000gp/T/
TERM_PROGRAM_VERSION=433
TERM_SESSION_ID=CA088303-0C09-4A18-894D-C17945BAD7AE
!
! Further output is truncated for brevity
This approach works for any Linux and Mac OS.
The interesting thing about the environment variables is that there are two types of them:
- Defined as part of you profile, which is persistent across all the sessions. This is not secure way.
- Created by you during the session and existing only within that particular session. This is quite a secure approach.
It is very easy to created the second version of environment variables, just type in CLI of your Linux or MAX this command:
Voila. The environment variable is created.
#2. Checking they are available only for you
We believe that you may have some questions at this stage, so we will try to answer all of them.
Is the variable created?
Yes, you have created the environment variable PYGNMI_USER and set its value to the string “pygnmi“.
How do I check that?
You can run the command env again and look visually for the variable, or add the filtering using grep:
2
PYGNMI_USER=pygnmi
How can I use it?
Later in this blogpost you will see how to use it in the Python script together with the pyGNMI. Directly from the CLI you can simple call that as using the echo tool with the argument ${variable_name}:
2
pygnmi
Is that available only for this shell session?
Yes, it is. It is very easy to check that. Just open another window inside your terminal client or connect to the destination host, if you have created that variable there using the SSH and try to run the command used to check check the value. You will see the empty string:
2
3
4
Password:
$ echo ${PYGNMI_USER}
The default behaviour of the Bash is to show the empty string for the value even if the variable does not exist.
In our network automation training you may learn this and a lot of user useful Linux life-hacks for your network automation.
#3. Using environment variables in your pyGNMI script
Finally we reach the point, where we can bring this knowledge into operation. First of all, let’s create the environment variables for our credentials, which are username and password:
2
$ export PYGNMI_PASS=pygnmi
Once completed you can create a script using pyGNMI and os, which is a standard Python library. The latter have a function evnironb, which collects the environment variables and make them available in your script:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
#!/usr/bin/env python
# Int is requred to create two environemnt variables:
# PYGNMI_USER, which will be used to store the username to connect to the network function
# PYGNMI_PASS, which will be used to store the password to connect to the network function
# Modules
from pygnmi.client import gNMIclient
import os
# Variables
hosts = [
{
"ip_address": "192.168.100.10",
"nos": "arista-eos",
"port": 57400
},
{
"ip_address": "192.168.100.11",
"nos": "nokia-sros",
"port": 57400
}
]
# Body
if __name__ == "__main__":
gnmi_user = "default" if "PYGNMI_USER" not in os.environ or not os.environ["PYGNMI_USER"] else os.environ["PYGNMI_USER"]
gnmi_pass = "default" if "PYGNMI_PASS" not in os.environ or not os.environ["PYGNMI_PASS"] else os.environ["PYGNMI_PASS"]
for host in hosts:
with gNMIclient(target=(host["ip_address"], host["port"]), username=gnmi_user,
password=gnmi_pass, insecure=True) as gc:
result = gc.capabilities()
print(f"{host['ip_address']}: {result}\n\n")
As a basis for this script we use the Capabilities RPC.
The class gNMIclient from our pyGNMI library requires the username and password to be provided at the time you create an object, which is used to communicate with the network devices. To provide them, we use the os.environ function, which dumps for us all the variables, which we put into conditional:
- If the environment variable PYGNMI_USER (or PYGNMI_PASS for password) exists and its value is defined, then use that value.
- If it doesn’t exist or value is an empty string, then use the default value predefined in the script.
#4. Validating the execution
First of all, let’s check that environment variables are defined:
2
3
PYGNMI_PASS=pygnmi
PYGNMI_USER=pygnmi
Now execute the Python script with pyGNMI:
2
3
4
5
6
7
8
192.168.100.10: {'supported_models': [{'name': 'openconfig-packet-match-types', 'organization': 'OpenConfig working group', 'version': '1.0.2'}, {'name': 'arista-qos-augments', 'organization': 'Arista Networks, Inc.', 'version': ''}, {'name': 'openconfig-platform-cpu', 'organization': 'OpenConfig working group', 'version': '0.1.1'}, {'name': 'arista-relay-agent-deviations', 'organization': 'Arista Networks <http://arista.com/>
!
! Some output is truncated for brevity
!
192.168.100.11: {'supported_models': [{'name': 'nokia-conf', 'organization': 'Nokia', 'version': '20.10.R3'},
!
! Furter output is truncated for brevity
You see that the connectivity to the devices is successful and you can grab the information. Now ssh to your host with pyGNMI again and check that there is no variables PYGNMI_USER and PYGNMI_PASS defined.
2
3
4
Password:
$ env | grep PYGNMI
Run the Python script again:
2
3
4
Host: 192.168.100.10:57400
Error: Authentication failed
CRITICAL:root:Host: 192.168.100.10:57400, Error: Authentication failed
The Python script used the default variables and, as they deviates from the one configured on the device, the authentication is not successful. In the same time, usage of the default variables allows you to make sure that the script ends with the meaningful error (Authentication failed), rather than having a traceback caused by the usage of undefined variables
GitHub and PyPI
You can see this and others examples of the pygnmi usage in our GitHub repository. Moreover, it is also available in PyPI, what makes the installation process very easy.
If you prefer video
You can watch this demo on our YouTube channel:
Lessons learned
To be honest, when the colleagues shown that to me and explain how that works, I was quite impressed. On the other hand, if not used properly, it may lead to the problems. For instance, if you put them as part of your environment file to make sure that they are persistent after the system reboot (or if you dockerize your application and therefore provide the password in there). In that case it is even less secure than having them in a plain text in a file with the tool itself. So don’t get into that trap.
Conclusion
Everything can be hacked, that is an important truth. So one of our goals, as network engineers and network automators, is to make sure the tools we are building and using are secure enough to reduce the possibility of sensitive information leak as much as possible. And usage of the environment variables helps us here a lot. Take care and good bye.
Support us
P.S.
If you have further questions or you need help with your networks, we are happy to assist you, just send us a message. Also don’t forget to share the article on your social media, if you like it.
BR,
Anton Karneliuk