Hello my friend,
So far the only way to provide user input to your Python and Go (Golang) applications we’ve shared with you in these blog series was the environment. Whilst it is a powerful way, which is heavily used especially in cloud native world, where we utilize Kubernetes, it is not the only way to provide user input. Today we’ll review another mechanism, which is text files.
Is Software Development Not Valuable Job Anymore?
Lately I’ve seen more and more posts on LinkedIn that AI is taking software development jobs away and/or making them less profitable. I’m myself use various AIs as code assistants, so I can see massive massive boost in productivity. At the same time, often AI generates code, which simply doesn’t work regardless the amount of iterations you try it with different prompts. Or it does generates working code, which is far less performance optimized that it can be. Therefore, I’m convinced that software engineers are here to stay for quite a bit. Moreover, network and IT infrastructure automation is a specific domain, which knowledge is even less acquirable by AI now due to lack of structured data for models training. Which means, you shall start studying network automation today with worries that it won’t be relevant.
We offer the following training programs in network automation for you:
- Zero-to-Hero Network Automation Training
- High-scale automation with Nornir
- Ansible Automation Orchestration with Ansble Tower / AWX
- Expert-level training: Closed-loop Automation and Next-generation Monitoring
During these trainings you will learn the following topics:
- Success and failure strategies to build the automation tools.
- Principles of software developments and the most useful and convenient tools.
- Data encoding (free-text, XML, JSON, YAML, Protobuf).
- Model-driven network automation with YANG, NETCONF, RESTCONF, GNMI.
- Full configuration templating with Jinja2 based on the source of truth (NetBox).
- Best programming languages (Python, Bash) for developing automation
- The most rock-solid and functional tools for configuration management (Ansible) and Python-based automation frameworks (Nornir).
- Network automation infrastructure (Linux, Linux networking, KVM, Docker).
- Orchestration of automation workflows with AWX and its integration with NetBox, GitHub, as well as custom execution environments for better scalability.
- Collection network data via SNMP and streaming telemetry with Prometheus
- Building API gateways with Python leveraging Fast API
- Integration of alerting with Slack and your own APIs
- … and many more
Moreover, we put all mentions technologies in the context of real use cases, which our team has solved and are solving in various projects in the service providers, enterprise and data centre networks and systems across the Europe and USA. That gives you opportunity to ask questions to understand the solutions in-depth and have discussions about your own projects. And on top of that, each technology is provided with online demos and labs to master your skills thoroughly. Such a mixture creates a unique learning environment, which all students value so much. Join us and unleash your potential.
Start your automation training today.
What Are We Going To Talk Today?
Files are everywhere, when we talk about IT world. Text files, images, data bases, binaries, and many more. That’s why there is no surprise that dealing with files is major part of any programming language. Today we are going to start topic of dealing with files by discussing the following questions:
- What are text files?
- How to read text files?
- How to write to text files?
Explanation
In a nutshell, text files, which content is typically simply UTF-8 or ASCII (or some other) encoded characters. You can see their content in Linux/MAC by simply using “cat” or “more” tools. Here is an example of the text file, which will use later in our coding section:
2
3
4
5
6
7
8
9
10
[credentials]
username: karneliuk
password: lab
[inventory]
hostname: leaf-1
ip_address: 192.168.1.1
port: 830
nos: arista-eos
Text files can be more complicated if they include metadata used to modify its content (e.g, fonts, styles, colors, etc), which you can find in advanced text editors such as Microsoft Word or Google Docs. The major difference between text files and binary files (e.g., images) is that binary files you typically cannot open without a specific programs to deal with them, so you cannot easily modify them.
In network and IT infrastructure automation we often deal with text files (sometimes they are also called plaintext files). We have CSV spreadsheets, INI/XML/JSON/YAML serialized files, we have code of our applications; all these are example of text files. As such, it is crucial to know how to deal with these files: how to read from them and how to write to them.
From the data structure perspective, when you read text file in a programming language, you get a string. That’s correct, just one string, despite you can see multiple lines when you open this file in terminal or using any text editor. Such a string called multi-line string as it contains multiple new line characters represented by “\n” symbol. If you want to deal with individual lines rather than with one multiple string, you shall use corresponding functions to split string in a list/slice of strings using “\n” as a separator.
This fact, that text is a string when read from file, also imposes additional requirements when you write to text file, that is what you write to a file shall be string as well. As such, regardless of data structures you are using in your programming language internally, whether those literals, lists/slices, dictionaries/maps, classes/structs, you need to convert them to one string before you would be able to write a file. Let’s show with code how all these operations work.
Examples
In the practical section today we’ll read a simple text file with Python and Go (Golang), so that you can provide your input to your applications. Once the file is read, you will see how it can be split in a list/slice of strings for further analysis/processing. Afterwards, we’ll do a light touch on regular expressions to do a lookup of content in text. Finally, the modified content of the original file will be stored in a new file.
Python
The following snippet of the Python code implements the aforementioned logic:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
import re
# Body
if __name__ == "__main__":
# Get paths
file_to_open = "../data/file.txt"
file_to_save = "../data/output.txt"
# Read file
with open(file_to_open, "rt", encoding="utf-8") as f:
data = f.read()
# Print the raw text file
print(data)
# File is a multiline string, so split it to lines
new_data = []
for ind, line in enumerate(data.splitlines()):
print(f"line {ind:>03}: {line}")
# Make FQDN
if re.match("^hostname:\s+.*$", line) and "network.karneliuk.com" not in line:
line += ".network.karneliuk.com"
# Copy line to new output
new_data.append(line)
# Save result file
with open(file_to_save, "wt", encoding="utf-8") as f:
f.write("\n".join(new_data))
Let’s break it down:
- We import only one library “re“, what stands for Regular Expressions, which will be used later.
- Using “with … as …” context manager applied to open() function we open the file descriptor for the file. The benefit of “with … as …” is that it automatically closes the file once it exists the context, so that you don’t need to think about it yourself.
- open() has 2 mandatory arguments and we’ve added one more optional:
- The first positional argument provides path to file (content of the variable file_to_open)
- The second positional argument specified the mode file is being opened (“rt” stands for read text).
- The optional key-worded argument encoding specified the encoding/decoding table for the content of the file.
- The content of the opened file using method read() applied to object with file f is read from file and assigned to variable data, which is a string.
- The content of the string is printed.
- Then using method splitlines() applied to the string, we get list of strings. This method uses “\n” as a separator. Additionally, we pass it via function enumerate(), which returns original values in addition to sequenced index of the element
- Content of each line is printed using formatted string. TO make the output looking better, we use padding of number with leading spaces ensuring that the number uses equal number of characters is each line.
- Using regexp “^hostname:\s+.*$” we look for a specific string using match() function from re module applied to the content of the line.
- We also use another technique to lookup for a sub-string in a string. Both methods are viable, we show both of them so that you can pick up the best one for you out of content.
- In case there is a match for both conditionals, we modify the content of the string.
- Finally, we save the content of the file. This time we use mode “wt” meaning “write text” when we open the file and method .write() applied to the file, which inputs require a string. So we create string out of list of strings using join() function and new line character as a joiner.
Let’s execute our Python program:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
[credentials]
username: karneliuk
password: lab
[inventory]
hostname: leaf-1
ip_address: 192.168.1.1
port: 830
nos: arista-eos
line 000: [credentials]
line 001: username: karneliuk
line 002: password: lab
line 003:
line 004: [inventory]
line 005: hostname: leaf-1
line 006: ip_address: 192.168.1.1
line 007: port: 830
line 008: nos: arista-eos
You see two outputs of the file, first one with a single multiline string printed and another is per-line iteration.
Additionally, when we execute the file, it creates new (or rewrites existing) file:
2
3
4
5
6
7
8
9
10
[credentials]
username: karneliuk
password: lab
[inventory]
hostname: leaf-1.network.karneliuk.com
ip_address: 192.168.1.1
port: 830
nos: arista-eos
You can see that the content of the line starting with hostname is modified compared to the original file, meaning that both conditionals were true upon execution.
Go (Golang)
Now let’s implement the very same scenario in Go (Golang):
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
package main
// Import
import (
"fmt"
"os"
"regexp"
"strings"
)
// Aux functions
func loadFile(f string) string {
/* Helper function to read file */
// Read file
bs, err := os.ReadFile(f)
if err != nil {
fmt.Printf("error: %v\n", err)
}
// Return data
return string(bs)
}
func saveFile(f string, c []string) bool {
/* Helper function to write to file */
// Prepare data for write
data := []byte(strings.Join(c, "\n"))
// Write to file
err := os.WriteFile(f, data, 0644)
if err != nil {
fmt.Printf("error: %v\n", err)
return false
}
// Return result
return true
}
// Main
func main() {
// Get paths
fileToOpen := "../data/file.txt"
fileToSave := "../data/output.txt"
// Read file
data := loadFile(fileToOpen)
// Print the raw text file
fmt.Println(data)
// File is a multiline string, so split it to lines
newData := []string{}
for ind, line := range strings.Split(data, "\n") {
fmt.Printf("line %03d: %v\n", ind, line)
// Make FQDN
re := regexp.MustCompile(`^hostname:\s+.*$`)
if re.MatchString(line) && !strings.Contains(line, "network.karneliuk.com") {
line += ".network.karneliuk.com"
}
// Copy line to new output
newData = append(newData, line)
}
// Save result to file
r := saveFile(fileToSave, newData)
if r {
fmt.Println("File is saved successfully.")
} else {
fmt.Println("File is NOT saved successfully.")
}
}
This time Go code is significantly different to what we showed in Python; mainly, because Go (Golang) is a procedural programming language and we found it easier / more logic to add some helpers:
- Helper function loadFile() reads the content of the file from a provided path. To actually read the file, the function ReadFile() from os package is used. In contrast to Python, this function returns 2 variables: slice of bytes and error. We evaluate if there is an error, and if not, we return converted to a string slice of bytes. In Python we’ve read string directly, without manually decoding bytes to characters.
- Helper function saveFile() writes the slice of strings to a file. Pretty much like it is in Python, input to os.WriteFile() function requires string; therefore, using Join() function from strings packages the slice of strings is transformed into a string. Additionally, in Go (Golang) it is required to provide permissions for the created/updated file: we pass “0644“, what stands for read-write permissions for file owners and read-only permissions for everyone else. We added a simple return boolean variable, which is used to signal if writing to file was successful or not.
- With the main body we create a variable data using the result of loadFile() helper.
- Then we create a new list of slices, which will be populated once we go through the original file line by line. The latter is achieved using Split() function from strings package as well as for-loop.
- For the purpose of the lookup within the string, we are using regular expression module regexp. Using function MustCompile() we are creating an object (struct), which contains the regular expression, which is executed using method MatchString(). Additionally, to lookup for sub-string the function Contains() from strings module is used.
- In case both conditionals are validated, the original line is amended.
- The newData variable is populated with every line we loop through.
- Using the helper function saveFIle() the content of the list of strings is written to disk.
Here is the execution of the Go (Golang) code:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
[credentials]
username: karneliuk
password: lab
[inventory]
hostname: leaf-1
ip_address: 192.168.1.1
port: 830
nos: arista-eos
line 000: [credentials]
line 001: username: karneliuk
line 002: password: lab
line 003:
line 004: [inventory]
line 005: hostname: leaf-1
line 006: ip_address: 192.168.1.1
line 007: port: 830
line 008: nos: arista-eos
line 009:
File is saved successfully.
After the execution, the new file is created or content of the existing file is entirely re-written.
Lessons in GitHub
You can find the final working versions of the files from this blog at out GitHub page.
Conclusion
Dealing with files is an important part of any programming language, as it is one of two main ways to get into your application. The second one is APIs, which will be covered later in our blog series. Whilst in this blogpost we haven’t reveled much how further process imported data apart from basic text manipulation, it is a stepping stone in JSON/XML/YAML processing, which is the topic for the next blog post. Stay tuned. Take care and good bye.
Support us
P.S.
If you have further questions or you need help with your networks, I’m happy to assist you, just send me message. Also don’t forget to share the article on your social media, if you like it.
BR,
Anton Karneliuk
