Hello my friend,
Today we are going to talk about the last data type, which in generally exists in Python and Go (Golang), and which we need dearly for all meaningful applications including network and IT infrastructure automation. We are talking about structured, typed data, which is represented in Python in form of objects and classes and in Go (Golang) in form of structs. These structures are truly powerful and once you figure out how to use them, I’m quite confident you will be using it everywhere, where you can.
You Talk So Much About Go (Golang), But You Offer Python In Trainings. Why?
This question I’ve been asked rather frequently recently. Indeed, why do we in each blog post talk about Network Automation Trainings, which gravitate around Python/Ansible duet. The answer is straightforward: whilst Go (Golang) is very powerful as we showing it in these blogs, its usage in many cases is justified only in very high-scale environment. For majority of networks and IT systems, Python is great. It is suffice to say that entire OpenStack is built in Python. And ourselves we use it extensively in many customers’ systems. Go (Golang) is useful as extension of automation skills, not as replacement of Python in our opinion. Therefore…
We offer the following training programs in network automation for you:
- Zero-to-Hero Network Automation Training
- High-scale automation with Nornir
- Ansible Automation Orchestration with Ansble Tower / AWX
- Expert-level training: Closed-loop Automation and Next-generation Monitoring
During these trainings you will learn the following topics:
- Success and failure strategies to build the automation tools.
- Principles of software developments and the most useful and convenient tools.
- Data encoding (free-text, XML, JSON, YAML, Protobuf).
- Model-driven network automation with YANG, NETCONF, RESTCONF, GNMI.
- Full configuration templating with Jinja2 based on the source of truth (NetBox).
- Best programming languages (Python, Bash) for developing automation
- The most rock-solid and functional tools for configuration management (Ansible) and Python-based automation frameworks (Nornir).
- Network automation infrastructure (Linux, Linux networking, KVM, Docker).
- Orchestration of automation workflows with AWX and its integration with NetBox, GitHub, as well as custom execution environments for better scalability.
- Collection network data via SNMP and streaming telemetry with Prometheus
- Building API gateways with Python leveraging Fast API
- Integration of alerting with Slack and your own APIs
- … and many more
Moreover, we put all mentions technologies in the context of real use cases, which our team has solved and are solving in various projects in the service providers, enterprise and data centre networks and systems across the Europe and USA. That gives you opportunity to ask questions to understand the solutions in-depth and have discussions about your own projects. And on top of that, each technology is provided with online demos and labs to master your skills thoroughly. Such a mixture creates a unique learning environment, which all students value so much. Join us and unleash your potential.
What Are We Going To Talk Today About?
The real story from my perspective is that I’ve avoided using structured data in Python for literally years. Why to bother to think about classes (that is collection attributes and methods) and predefined attributes (that is the name of the variable within the class) if for the purpose of storing data I can use dictionaries. Putting aside the object-oriented programming (which is the topic for the next blog post), if I solely wanted to store data, I didn’t use classes. Things changed, when I started building API and web-apps using fastAPI and Django.
By the way, you can learn FastAPI and build blazing fast APIs for your network and infrastructure automation in our closed-loop automation training.
Things changed, because those frameworks forces you to use structured strictly-typed data. And they force you to use structured strictly-typed data on API layer, where you define what application shall receive from external interaction or what it shall it send back in a response. It is critical to have API create clear, as quite often developers of client- and server-side applications are very different people, often they even don’t know each other and all they have to ensure that applications can talk to each other is API documentation. After I’ve appreciated this concept truly,we’ve started using it actively internally, wherever we need to communicate data between functions, etc. Then, when I was on a C-journey, I’ve used structs a lot, as there are no objects in C. Likewise, there are no objects in Go (Golang), but there are structs like in C as well.
With that lengthy introduction, let’s get to the core of our discussion. We are going to overview:
- What are (data)classes in Python and structs in Go (Golang)?
- Which problems do they solve?
- How and where to use them?
Explanation
Before jumping to the core of today’s topic, let’s first understand the problem it solves. In previous blog post we’ve described dictionaries/maps in Python and Go (Golnag), and as a part of the discussion we showed how to check if a certain key exists in dictionary/map. You need to do that, because dictionary/map per-se doesn’t have any schema, meaning a definition of:
- Which keys it may contains.
- Which data types are associated with these keys.
- Which keys are mandatory (you must ensure they present in your request) and which are optional (you may omit them and then default value will be used)
If you try to call non-existing key in dictionary/map, you will get an error during the execution of your application.
All these questions are answered, when we use structured strictly-typed data types. In Python this is achieved using classes and objects.
Classes and objects are far more than just structured data, they are at heart of object-oriented programming (OOP) in Python. That’s why we will touch base on OOP in the next blog post.
It works as such:
- You specify a class
- Within this class you define which attributes (think about them as variables created within the class) of which data type you want to have
- Then you create object out of this class (called class instance).
- During this process you may (or may have to) provide values for some or all the attributes
- Then you can pass this object anywhere (e.g., to the input of the function) and all its variables are passed as well
Python is not strictly typed, so to achieve the true type-strictness, external libraries needs to be used.
Go (Golang) isn’t an object-oriented, programming language, so it doesn’t have concept of objects and classes. Instead, much like C, it is a procedural programming language. So it doesn’t have classes. Instead, it has structs, which is serving the exact purpose of having structured strict typed data.
At a pseudo code level, it would look like as follows:
2
3
4
5
6
7
8
9
10
key1 : data_type_1
key2 : data_type_2
key3 : data_type_3
key4 : data_type_4
item = SomeData(key1="abc", key2=123, ...)
print(item.key1)
some_other_func(item)
Where:
- “datacontainer” called “SomeData” is a schema defining possible keys and associated data types
- “key1 : data_type_1” is a representation of key with the name “key1” and data type “data_type_1“, which can exist in this object/struct. multiple keys are listed
- “item” is instantiated using the data container “SomeData” and the relevant keys are passed during this process
- You can access then every individual key without need to check if it exists, as it is defined by schema strictly.
- You can also pass the entire container to input of the function
There is no such term as data container neither in Python or Go (Golang), we’ve completely made up this term for the purpose of this discussion
Let’s see how it works on the code examples.
Examples
We are going to implement the following scenario:
- You are going create three structured data:
- one to store credentials, which shall have 2 keys: username and password.
- one to store information about devices, that is hostname, ip address, port to connect, and network operating system
- one to store multiple devices to act as an inventory
- These data structures shall be populated from environment variables.
- Print the content of the variables
Python
We propose the following code in Python to complete this scenario:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
# Import os
import os
from typing import List
# Data models
class User:
"""Class to store user credentials"""
def __init__(self, username, password):
self.username: str = username
self.password: str = password
class Device:
"""Class to store device information"""
def __init__(self, name: str, port: int, nos: str = None, ip: str = None):
self.name: str = name
self.port: int = port
self.nos = nos
self.ip = ip
class Inventory:
"""Class to store inventory information"""
def __init__(self):
self.devices: List[Device] = []
# Functions
def get_credentials() -> User:
"""Function to retrieve credentials from the environment"""
return User(*os.getenv("AUTOMATION_CREDS").split(","))
def get_inventory() -> Inventory:
"""Function to retrieve inventory from the environment"""
# Create an empty list to store devices
result = Inventory()
# Loop through the environment variables
for key, value in os.environ.items():
# Check if the key starts with AUTOMATION_DEVICE_
if key.startswith('AUTOMATION_DEVICE_'):
# Split the value by comma and create a new device object
split_value = value.split(',')
result.devices.append(
Device(
name=split_value[0],
port=int(split_value[1]),
nos=split_value[3],
ip=split_value[2],
)
)
# Return the result
return result
# Execution
if __name__ == "__main__":
# Get the credentials
user = get_credentials()
# Print the credentials
print(f"Username: {user.username}")
print(f"Password: {user.password}")
# Get the inventory
inventory = get_inventory()
# Print inventory memory address
print(f"Memory address of inventory: {id(inventory):#x}")
# Print the inventory
print("Inventory:")
for device in inventory.devices:
print(f"Device: {device.name}")
print(f"Port: {device.port}")
print(f"IP: {device.ip}")
print(f"NOS: {device.nos}")
print("\n")
Many concepts we have covered in previous blog posts in this series about Python and Go (Golang). We recommend to look in previous to get more information about for-loops, module imports, and read from Linux/MAC environment.
Let’s review the key new components:
- We have classes instances, which start with the keyword “class” followed by the name of the class.
- Each class has a mandatory constructor method “__init__()“. Method is a function associated with class (much like attribute is a variable associated with class). This function may take external arguments.
- These external arguments we convert to attributes (e.g., self.username, self.password are attributes).
- Self is a representation of the object from the its own inside. All methods must have self as the first argument. This though we will discuss in the next blog post, for now we use just a single built-in method “__init__()“
- You can provide a default value for specific variable. E.g., “nos : str = None” means that argument nos shall be of type string with default value set to None, which is not a string. That highlights that Python is not strict-typed by default.
- Function “get_credentails()” returns instance of “User” class. Within this function we create a new object, which requires two arguments: username and password. We can pass them in a key-worded format, meaning specifying pairs of “key=value” or using positional format, where arguments must be in the same order as defined in “__init__()” function. The second approach is taken, which resulted in line “*os.getenv(“AUTOMATION_CREDS”).split(“,”)”. Let’s decode it:
- Function “os.getenv(“AUTOMATION_CREDS”)” looks in environment variable “AUTOMATION_CREDS” and return its value.
- This value is then split over comma using “.split(“,”)” method into a list
- and then using “*” in the beginning of this statement we turn list into a sequence individual positional arguments for the function.
- Function “get_inventory()” return instance of “Inventory” class. This function iterates over dictionary keys, and if the key starts with sub-string “AUTOMATION_DEVICE_“, then its value split into a list with “,” as separator. Then the instance of “Device()” class is created and is added to the inventory. The instance is populated this time using key-worded arguments, with port argument being also passed via type conversion.
- The main code is executed. Cryptic command “f”{id(inventory):#x}”” returns in hexadecimal format the memory address of the variable, which contains this variable.
Here is the example of the code execution:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Username: karneliuk
Password: lab
Memory address of inventory: 0x7f93376eb880
Inventory:
Device: leaf-1
Port: 830
IP: 192.168.1.1
NOS: arista-eos
Device: leaf-2
Port: 22
IP: 192.168.1.1
NOS: cisco-nxos
As mentioned a few times, Python by default isn’t strict typed language. For it to be strictly typed, you need to add quite a bit of validation for your input data, as nothing prevents you to put boolean instead of string and this code will accept it just fine. Rather than you write all the validators yourself, you can rely on 3rd party packages. The one we are using a lot ourselves is called “pydantic“, which is a major driving power behind FastAPI. In the code examples for this post in GitHub you can find example written with Pydantic.
Go (Golang)
The very same flow we’ve just created in Python using classes and objects we now re-create in Go (Golang):
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
package main
// Imports
import (
"fmt"
"os"
"strconv"
"strings"
)
// Data types
type User struct {
// Class to store user credentials
username string
password string
}
type Device struct {
// Class to store device information
hostname string
port uint64
ip string
nos string
}
// Class to store inventory information
type Inventory []Device
// Aux functions
func getCredentials() User {
// Function to retrieve credentials from the environment
blocks := strings.Split(os.Getenv("AUTOMATION_CREDS"), ",")
return User{
blocks[0],
blocks[1],
}
}
func getInventory() *Inventory {
// Create an empty list to store devices
result := &Inventory{}
// Loop through the environment variables
for _, kv := range os.Environ() {
// Check if the key starts with AUTOMATION_DEVICE_
if strings.Contains(kv, "AUTOMATION_DEVICE_") {
// Split the value by comma and create a new device object
blocks := strings.Split(strings.Split(kv, "=")[1], ",")
devicePort, err := strconv.ParseUint(blocks[1], 10, 64)
if err != nil {
fmt.Printf("Got error when converting string to uint: %v\n", err)
os.Exit(1)
}
*result = append(*result, Device{
blocks[0],
devicePort,
blocks[3],
blocks[2],
})
}
}
// Result
return result
}
// Main function
func main() {
/* Main business logic */
// Get the credentials
user := getCredentials()
// Print credentails
fmt.Printf("%+v\n", user)
// Get inventory
inventory := getInventory()
// Print inventory memory address
fmt.Printf("Memory address of inventory: %v\n", &inventory)
// Print inventory content
fmt.Printf("%+v\n", *inventory)
}
As we mentioned in Python, many concepts we’ve covered before; please, refer to previous blog posts if you have questions.
Here is what relevant new here:
- The structs are defined using the instruction “type“, which stands for type definition. You need to provide name and the type, which is “struct” in two cases and slice of defined struct “[]Device” in one case. Which each type you specify keys and associated data types. In contrast to Python, Go is by default strict-typed, so you have to ensure your data will match.
- We create a helper function “getCredentials()” which returns struct “User“. The struct is populated in Go using positional arguments, so we read from environment the variable of “AUTOMATION_CREDS” using “os.Getenv()” function and then split it into a list, which is fed to “User” struct.
- The next helper function “getInventory()” returns pointer to a struct “Inventory“, what allows to avoid copy data between functions, which is a significant performance improvement if you have a few hundreds of devices in your inventory. The pointer is created using “result := &Inventory{}” statement, which equals to “new()” function covered earlier in this series. Within this function we iterate through all environment variables and when we find a match for our pattern, this variable is split using “,” as separator and then populate struct Device(). There is a bit of complexity in comparison to Python, as we need to convert string to unsigned integer. This is achieved in Golang using function “ParseUint()” from “strconv” library, which returns two values: converted value and error. The latter shall be checked to ensure that conversion was successful.
- In the main block we call the functions and print their results. When we print content of structs, it is useful to use “%+v” pattern instead of “%v“. The former will print both key and value, whilst the latter prints only value.
Here is the result of execution:
2
3
4
{username:karneliuk password:lab}
Memory address of inventory: 0xc00005a048
[{hostname:leaf-1 port:830 ip:arista-eos nos:192.168.1.1} {hostname:leaf-2 port:22 ip:cisco-nxos nos:192.168.1.1}]
It is worth mentioning that in Go (Golang) you can create custom data types not only for structs or list of structs, but for any other data type, including standard ones. In the next blog post you will learn why you may want to do it.
Lessons in GitHub
You can find the final working versions of the files from this blog at out GitHub page.
Conclusion
This blog post concludes our overview of all data types existing in Python and Go (Golang). If you completed all the labs, which is a prerequisite for proper understanding of this topic, you shall have by now a good grasp of how to store data in these programming languages and how to use it for computations. In the next blog we’ll touch, as mentioned a few times, on the topic of object-oriented programming. Take care and good bye.
Support us
P.S.
If you have further questions or you need help with your networks, I’m happy to assist you, just send me message. Also don’t forget to share the article on your social media, if you like it.
BR,
Anton Karneliuk
