Site icon Karneliuk

From Python to Go 010. Dealing With Text Files. And Tiny Bit On Regexp.

Anton Karneliuk

Hello my friend,

So far the only way to provide user input to your Python and Go (Golang) applications we’ve shared with you in these blog series was the environment. Whilst it is a powerful way, which is heavily used especially in cloud native world, where we utilize Kubernetes, it is not the only way to provide user input. Today we’ll review another mechanism, which is text files.

Is Software Development Not Valuable Job Anymore?

Lately I’ve seen more and more posts on LinkedIn that AI is taking software development jobs away and/or making them less profitable. I’m myself use various AIs as code assistants, so I can see massive massive boost in productivity. At the same time, often AI generates code, which simply doesn’t work regardless the amount of iterations you try it with different prompts. Or it does generates working code, which is far less performance optimized that it can be. Therefore, I’m convinced that software engineers are here to stay for quite a bit. Moreover, network and IT infrastructure automation is a specific domain, which knowledge is even less acquirable by AI now due to lack of structured data for models training. Which means, you shall start studying network automation today with worries that it won’t be relevant.

We offer the following training programs in network automation for you:

During these trainings you will learn the following topics:

Moreover, we put all mentions technologies in the context of real use cases, which our team has solved and are solving in various projects in the service providers, enterprise and data centre networks and systems across the Europe and USA. That gives you opportunity to ask questions to understand the solutions in-depth and have discussions about your own projects. And on top of that, each technology is provided with online demos and labs to master your skills thoroughly. Such a mixture creates a unique learning environment, which all students value so much. Join us and unleash your potential.

Start your automation training today.

What Are We Going To Talk Today?

Files are everywhere, when we talk about IT world. Text files, images, data bases, binaries, and many more. That’s why there is no surprise that dealing with files is major part of any programming language. Today we are going to start topic of dealing with files by discussing the following questions:

  1. What are text files?
  2. How to read text files?
  3. How to write to text files?

Explanation

In a nutshell, text files, which content is typically simply UTF-8 or ASCII (or some other) encoded characters. You can see their content in Linux/MAC by simply using “cat” or “more” tools. Here is an example of the text file, which will use later in our coding section:



1
2
3
4
5
6
7
8
9
10
$ cat ../data/file.txt 

[credentials]

username: karneliuk

password: lab



[inventory]

hostname: leaf-1

ip_address: 192.168.1.1

port: 830

nos: arista-eos

Text files can be more complicated if they include metadata used to modify its content (e.g, fonts, styles, colors, etc), which you can find in advanced text editors such as Microsoft Word or Google Docs. The major difference between text files and binary files (e.g., images) is that binary files you typically cannot open without a specific programs to deal with them, so you cannot easily modify them.

In network and IT infrastructure automation we often deal with text files (sometimes they are also called plaintext files). We have CSV spreadsheets, INI/XML/JSON/YAML serialized files, we have code of our applications; all these are example of text files. As such, it is crucial to know how to deal with these files: how to read from them and how to write to them.

From the data structure perspective, when you read text file in a programming language, you get a string. That’s correct, just one string, despite you can see multiple lines when you open this file in terminal or using any text editor. Such a string called multi-line string as it contains multiple new line characters represented by “\n” symbol. If you want to deal with individual lines rather than with one multiple string, you shall use corresponding functions to split string in a list/slice of strings using “\n” as a separator.

This fact, that text is a string when read from file, also imposes additional requirements when you write to text file, that is what you write to a file shall be string as well. As such, regardless of data structures you are using in your programming language internally, whether those literals, lists/slices, dictionaries/maps, classes/structs, you need to convert them to one string before you would be able to write a file. Let’s show with code how all these operations work.

Examples

In the practical section today we’ll read a simple text file with Python and Go (Golang), so that you can provide your input to your applications. Once the file is read, you will see how it can be split in a list/slice of strings for further analysis/processing. Afterwards, we’ll do a light touch on regular expressions to do a lookup of content in text. Finally, the modified content of the original file will be stored in a new file.

Python

The following snippet of the Python code implements the aforementioned logic:



1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
"""From Python to Go: Python: 010 - Text files"""



import re





# Body

if __name__ == "__main__":

    # Get paths

    file_to_open = "../data/file.txt"

    file_to_save = "../data/output.txt"



    # Read file

    with open(file_to_open, "rt", encoding="utf-8") as f:

        data = f.read()



    # Print the raw text file

    print(data)



    # File is a multiline string, so split it to lines

    new_data = []

    for ind, line in enumerate(data.splitlines()):

        print(f"line {ind:>03}: {line}")



        # Make FQDN

        if re.match("^hostname:\s+.*$", line) and "network.karneliuk.com" not in line:

            line += ".network.karneliuk.com"



        # Copy line to new output

        new_data.append(line)



    # Save result file

    with open(file_to_save, "wt", encoding="utf-8") as f:

        f.write("\n".join(new_data))

Let’s break it down:

  1. We import only one library “re“, what stands for Regular Expressions, which will be used later.
  2. Using “with … as …” context manager applied to open() function we open the file descriptor for the file. The benefit of “with … as …” is that it automatically closes the file once it exists the context, so that you don’t need to think about it yourself.
  3. open() has 2 mandatory arguments and we’ve added one more optional:
    • The first positional argument provides path to file (content of the variable file_to_open)
    • The second positional argument specified the mode file is being opened (“rt” stands for read text).
    • The optional key-worded argument encoding specified the encoding/decoding table for the content of the file.
  4. The content of the opened file using method read() applied to object with file f is read from file and assigned to variable data, which is a string.
  5. The content of the string is printed.
  6. Then using method splitlines() applied to the string, we get list of strings. This method uses “\n” as a separator. Additionally, we pass it via function enumerate(), which returns original values in addition to sequenced index of the element
  7. Content of each line is printed using formatted string. TO make the output looking better, we use padding of number with leading spaces ensuring that the number uses equal number of characters is each line.
  8. Using regexp “^hostname:\s+.*$” we look for a specific string using match() function from re module applied to the content of the line.
  9. We also use another technique to lookup for a sub-string in a string. Both methods are viable, we show both of them so that you can pick up the best one for you out of content.
  10. In case there is a match for both conditionals, we modify the content of the string.
  11. Finally, we save the content of the file. This time we use mode “wt” meaning “write text” when we open the file and method .write() applied to the file, which inputs require a string. So we create string out of list of strings using join() function and new line character as a joiner.

Let’s execute our Python program:



1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
$ python3 main.py 

[credentials]

username: karneliuk

password: lab



[inventory]

hostname: leaf-1

ip_address: 192.168.1.1

port: 830

nos: arista-eos



line 000: [credentials]

line 001: username: karneliuk

line 002: password: lab

line 003: 

line 004: [inventory]

line 005: hostname: leaf-1

line 006: ip_address: 192.168.1.1

line 007: port: 830

line 008: nos: arista-eos

You see two outputs of the file, first one with a single multiline string printed and another is per-line iteration.

Additionally, when we execute the file, it creates new (or rewrites existing) file:



1
2
3
4
5
6
7
8
9
10
$ cat ../data/output.txt 

[credentials]

username: karneliuk

password: lab



[inventory]

hostname: leaf-1.network.karneliuk.com

ip_address: 192.168.1.1

port: 830

nos: arista-eos

You can see that the content of the line starting with hostname is modified compared to the original file, meaning that both conditionals were true upon execution.

Go (Golang)

Now let’s implement the very same scenario in Go (Golang):



1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
/* From Python to Go: Go (Golang): 010 - Text files */



package main



// Import

import (

        "fmt"

        "os"

        "regexp"

        "strings"

)



// Aux functions

func loadFile(f string) string {

        /* Helper function to read file */



        // Read file

        bs, err := os.ReadFile(f)

        if err != nil {

                fmt.Printf("error: %v\n", err)

        }



        // Return data

        return string(bs)

}

func saveFile(f string, c []string) bool {

        /* Helper function to write to file */



        // Prepare data for write

        data := []byte(strings.Join(c, "\n"))



        // Write to file

        err := os.WriteFile(f, data, 0644)

        if err != nil {

                fmt.Printf("error: %v\n", err)

                return false

        }



        // Return result

        return true

}



// Main

func main() {

        // Get paths

        fileToOpen := "../data/file.txt"

        fileToSave := "../data/output.txt"



        // Read file

        data := loadFile(fileToOpen)



        // Print the raw text file

        fmt.Println(data)



        // File is a multiline string, so split it to lines

        newData := []string{}

        for ind, line := range strings.Split(data, "\n") {

                fmt.Printf("line %03d: %v\n", ind, line)



                // Make FQDN

                re := regexp.MustCompile(`^hostname:\s+.*$`)

                if re.MatchString(line) && !strings.Contains(line, "network.karneliuk.com") {

                        line += ".network.karneliuk.com"

                }



                // Copy line to new output

                newData = append(newData, line)

        }



        // Save result to file

        r := saveFile(fileToSave, newData)

        if r {

                fmt.Println("File is saved successfully.")

        } else {

                fmt.Println("File is NOT saved successfully.")

        }

}

This time Go code is significantly different to what we showed in Python; mainly, because Go (Golang) is a procedural programming language and we found it easier / more logic to add some helpers:

  1. Helper function loadFile() reads the content of the file from a provided path. To actually read the file, the function ReadFile() from os package is used. In contrast to Python, this function returns 2 variables: slice of bytes and error. We evaluate if there is an error, and if not, we return converted to a string slice of bytes. In Python we’ve read string directly, without manually decoding bytes to characters.
  2. Helper function saveFile() writes the slice of strings to a file. Pretty much like it is in Python, input to os.WriteFile() function requires string; therefore, using Join() function from strings packages the slice of strings is transformed into a string. Additionally, in Go (Golang) it is required to provide permissions for the created/updated file: we pass “0644“, what stands for read-write permissions for file owners and read-only permissions for everyone else. We added a simple return boolean variable, which is used to signal if writing to file was successful or not.
  3. With the main body we create a variable data using the result of loadFile() helper.
  4. Then we create a new list of slices, which will be populated once we go through the original file line by line. The latter is achieved using Split() function from strings package as well as for-loop.
  5. For the purpose of the lookup within the string, we are using regular expression module regexp. Using function MustCompile() we are creating an object (struct), which contains the regular expression, which is executed using method MatchString(). Additionally, to lookup for sub-string the function Contains() from strings module is used.
  6. In case both conditionals are validated, the original line is amended.
  7. The newData variable is populated with every line we loop through.
  8. Using the helper function saveFIle() the content of the list of strings is written to disk.

Here is the execution of the Go (Golang) code:



1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
$ go run .

[credentials]

username: karneliuk

password: lab



[inventory]

hostname: leaf-1

ip_address: 192.168.1.1

port: 830

nos: arista-eos



line 000: [credentials]

line 001: username: karneliuk

line 002: password: lab

line 003: 

line 004: [inventory]

line 005: hostname: leaf-1

line 006: ip_address: 192.168.1.1

line 007: port: 830

line 008: nos: arista-eos

line 009: 

File is saved successfully.

After the execution, the new file is created or content of the existing file is entirely re-written.

Lessons in GitHub

You can find the final working versions of the files from this blog at out GitHub page.

Conclusion


Dealing with files is an important part of any programming language, as it is one of two main ways to get into your application. The second one is APIs, which will be covered later in our blog series. Whilst in this blogpost we haven’t reveled much how further process imported data apart from basic text manipulation, it is a stepping stone in JSON/XML/YAML processing, which is the topic for the next blog post. Stay tuned. Take care and good bye.

Support us






P.S.

If you have further questions or you need help with your networks, I’m happy to assist you, just send me message. Also don’t forget to share the article on your social media, if you like it.

BR,

Anton Karneliuk 

Exit mobile version