Hello my friend,

Congratulations if you reach this blog post starting from the beginning of our series. I look in the list of topic I’m going to cover with respect Python to Go (Golang) transition and I see that I’ve already written more than 50% of blog posts; so we are over the half! Secondly, from today we start talking solely about practical network and IT infrastructure automation topics, as all the foundational things are already covered, such as data types, code flow control, reading and parsing files, using CLI arguments, exceptions handling to name a few. Today we’ll talk about how we can template text files. With respect to network and IT infrastructure automation that means we look how to prepare configuration of our network devices, Linux servers or further appliances.

Isn’t AI Going to Replace All Software Developers Anyway?

Almost daily I see in LinkedIn posts about how cool AI is in generating code. It is cool, I agree with that. And I myself use Co-pilot for my private projects for already more than a year and ChatGPT for certain things. However, my experience is that at this stage they don’t yet replace engineers (neither they would be able in near future in my opinion). Network and IT Automation Infrastructure Engineers are not just code monkeys, AI is indeed good there; Network and IT Infrastructure Automation Engineers are about understanding requirements, developing and deploying solutions per these requirements and technical constrains, maintaining and further improving solutions, and troubleshooting where needed. So if you just generate code with AI and put it straight to production without rigorous testing and deep understanding what it really does, you are already doomed.

That’s why we encourage you to join our Network Automation Training programs, where we put software development in context of Network and IT infrastructure problems, and where we show the whole lifecycle from idea to packaged containerized product. Build solid foundations and then advance with AI with us:

We offer the following training programs in network automation for you:

• Zero-to-Hero Network Automation Training
• High-scale automation with Nornir
• Ansible Automation Orchestration with Ansble Tower / AWX
• Expert-level training: Closed-loop Automation and Next-generation Monitoring

During these trainings you will learn the following topics:

• Success and failure strategies to build the automation tools.
• Principles of software developments and the most useful and convenient tools.
• Data encoding (free-text, XML, JSON, YAML, Protobuf).
• Model-driven network automation with YANG, NETCONF, RESTCONF, GNMI.
• Full configuration templating with Jinja2 based on the source of truth (NetBox).
• Best programming languages (Python, Bash) for developing automation
• The most rock-solid and functional tools for configuration management (Ansible) and Python-based automation frameworks (Nornir).
• Network automation infrastructure (Linux, Linux networking, KVM, Docker).
• Orchestration of automation workflows with AWX and its integration with NetBox, GitHub, as well as custom execution environments for better scalability.
• Collection network data via SNMP and streaming telemetry with Prometheus
• Building API gateways with Python leveraging Fast API
• Integration of alerting with Slack and your own APIs
• … and many more

Moreover, we put all mentions technologies in the context of real use cases, which our team has solved and are solving in various projects in the service providers, enterprise and data center networks and systems across the Europe and USA. That gives you opportunity to ask questions to understand the solutions in-depth and have discussions about your own projects. And on top of that, each technology is provided with online demos and labs to master your skills thoroughly. Such a mixture creates a unique learning environment, which all students value so much. Join us and unleash your potential.

Start your automation training today.

What Are We Going To Talk Today?

Templating configuration files is one of the classical tasks essential for Network and IT Infrastructure automation. It is also a foundation of Infrastructure as Code (IaC) approach. Therefore we are going to discuss it today with specific attention to these questions:

1. How to create scalable templates for configuration files in Python and Go (Golang), which takes into account various issues with data?
2. How to use templates and produce configuration files?

Explanation

General

In a nutshell, templating is a process of taking one object called template, and using some mechanics applying it a set of your variables in order to obtain new object with template filled with data. As aforementioned, this process is heavily used in Infrastructure as Code; in fact, IaC cannot exist without templating. But even before that, in software development templating has been heavily used for ages dynamically generating the webpages based on defined schemas using input from users or local databases.

Python And Jinja2

As such, it is not a surprise that libraries for templating files we use in Network and IT Automation are those ones, which we originally created for web site. In Python such library is called Jinja (or Jinja2). It is actively used in such Website and API gateways frameworks as Flask and Fast API.

Jinja is also driving power behind Ansible. Enroll at Network Automation Training to master both in-depth.

Despite being used closely with Python, Jinja2 has its own syntax. Take a look at the following snippet:

1
2
3
4
5
6
7
8
9
{%- if device.hostname -%}

hostname {{ device.hostname }}

!

{%- endif %}

{%- if device.interfaces -%}

{% for interface in device.interfaces %}

interface {{ interface.name }}

{%- endfor %}

{%- endif %}

Here is what’s going in this snippet:

• This template expects to receive from Python dictionary or data class “device“.
• Inserting value happens within double curly braces “{{ value_to_insert }}“. So “{{ device.hostname }}” will insert key “hostname” from dictionary “device” or value of attribute “hostname” from data class “device“.
• Code flow control elements are provided within curly braces followed/appended by percent “{% … %}“:
  • Conditional “if-elif-else” is supported. For example, expression “{% if device.interfaces %}…{% endif %}” validates if key/attribute “interface” of “device” is value:
    • True for boolean
    • Non-empty string
    • no-zero for numbers
    • non-empty list or dictionary
  • for-loops are supported as well. For example, expression “{% for interface in device.interfaces %}…{% endfor %}” iterates over each element of list “interface” of “device“.
• You can see dash “-” right after “{%” or right before “%}“. This controls if the new line character shall be subtracted before and/or after the line with code flow control in the resulting text file.

Check our Zero-to-Hero Network Automation Training for more details.

We mentioned earlier that template is an object. In Python you can create multi-line string with template directly in your code or you can put template in a fully separate file, which you read in your code. Typically the latter is more preferable as you can have many templates for different use cases, which you develop separately without touching your underlying Python code.

By the way, Jinja2 is an external package, which you need to install if you want to use it.

Here is a simple code, how to template code with Jinja2 in Python (it is assumed that your template is in separate file “template.j2“):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# Import jinja2 package once it is installed

import jinja2



# Variable matching template

device = {"hostname": "leaf1", "interfaces": [{"name": Ethernet1}]}



# Read template from file and create corresponding object

with open("template.j2", "r", encoding="utf-8") as file:

    templ = jinja2.Template(file.read())



# Fill in template

result = templ.render(device=device)



# Display result

print(result)

Inline comments in this snippet explains what the code does. As you can see, the process has essentially just 4 steps:

1. Get your template.
2. Get your input variables.
3. Render template with data into result data.
4. Do something with the created output.

Go (Golang) And Go-templates

Go (Golang), being widely used for web services and network/IT infrastructure automation these days, have templating functionality as well. In contrast with Python this functionality is available in the built-in library “text/template“. Let’s create the very same example using Go-template, as we have created previously in Jinja:

1
2
3
4
5
6
7
8
9
{{- if .Hostname -}}

hostname {{ .Hostname }}

!

{{- end }}

{{- if .Interfaces }}

{{- range .Interfaces }}

interface {{ .Name }}

{{- end -}}

{{- end }}

At a glance, they are very similar. Let’s unpack what do we have here:

• We use something called cursor, which is represented by “.” symbol. Essentially, this is the data structure we pass inside: it can be struct, slice, or anything else. That’s what in Python/Jinja2 we have “device“, named class. The important note is that cursor will change if you go over nested slices. For example, “{{ range .Interfaces }}” will set cursor as every nested struct of Interfaces slice.
• Interpolation if values follows identical to Python/Jinja2 pattern, that is wrapping the name of the variable in double curly braces “{{ .Hostname }}“.
• Code flow control is enclosed in double curly braces as well “{{ … }}“; however, in contrast to Python/Jinja2 the ending tag is always just “{{ end }}“, which makes it a little be more complicated to troubleshoot Go-templates :
  • Conditional “if-else if-else” is supported. For example, expression “{{ if .Interfaces }}…{{ end }}” validates if there is a non-default value of key “Interfaces” from the current data structure pointed by cursor.
  • For-loops are supported. For example, expression “{{ range .Interfaces }}…{{ end }}” iterates over each element of list “Interfaces” from the current data structure pointed by cursor.
• Dashes “–” have exactly the same meaning and impact in Go (Golang) as they have in Python/Jinja2.

In the same way as in Python, you can have Go-templates defined in your code, or you can store them outside of the code in separate files, which you then read and render. Here is how the sample process would look like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
// Import package for templating

import (

    "text/template"

    "os"

    "fmt"

)



// Variable matching template

type Device struct {

    Hostname   string

    Interfaces []struct {

        name string

    }

}



func main() {

    // Read template from file and create corresponding object

    templ, err := template.New(p).ParseFiles(p)

    if err != nil {

       fmt.Println("Get error ", err)

        os.Exit(1)

    }



    // Fill in template and display result

    templ.Execute(os.Stdout, device)

}

The workflow of working with Go-templates in Go (Golang) is fairly identical to Python/Jinja2:

1. Create template
2. Create variables
3. Fill in template with variables
4. Do something with the result

Working with templates are the same 2 commands: create and populate. The difference in Go (Golang) is that you save output in struct supporting io.Write interface, which in this case is represented by os.Stdout and to write to file you need something different (continue reading for that).

Examples

As we claim that we talk about practical Python and Go (Golang) for network and IT infrastructure automation, we will deploy the real practical scenario, which encompasses many topics we’ve covered so far:

1. Read and parse into structured data YAML file from the path provided as CLI argument.
2. Read and load template from the path provided as CLI argument.
3. Populate template with the data
4. Save the result into files

Python

For this application, we need to install a number external packages:

• Jinja2 for templating itself
• Pyyaml for parsing YAML files, as we already used before
• Pydantic for data schemas and strict typing in Python

Here is how you do it:

1
$ pip install jinja2 pyyaml pydantic

Now let’s write the application to match our requirements:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
"""From Python to Go: Python: 014 - Templating configuration."""



# Modules

import argparse

from typing import List, Union

import sys

import os

import yaml

from pydantic import BaseModel

import jinja2





# Classes

class IPAddress(BaseModel):

    """Class to store IP address data."""

    address: str

    prefix: int





class Interface(BaseModel):

    """Class to store interface data."""

    name: str

    description: Union[str, None] = None

    ip4: Union[IPAddress, None] = None

    enabled: bool = False





class Device(BaseModel):

    """Class to store credentials."""

    hostname: str

    interfaces: List[Interface]





# Functions

def read_args() -> argparse.Namespace:

    """Helper function to read CLI arguments."""

    parser = argparse.ArgumentParser(description="User input.")

    parser.add_argument("-d", "--data", type=str, help="Path to the input file.")

    parser.add_argument("-t", "--template", type=str, help="Path to the template.")

    return parser.parse_args()





def load_inventory(filename: str) -> List[Device]:

    """Function to load inventory data."""

    # Open file

    try:

        with open(filename, "r", encoding="utf-8") as f:

            data = yaml.safe_load(f)



    except FileNotFoundError as e:

        print(e)

        sys.exit(1)



    # Populate list of devices

    devices = []

    for device in data:

        devices.append(Device(**device))



    return devices





def load_template(filename: str) -> jinja2.Template:

    """Function to load Jinja2 template."""

    # Open file

    try:

        with open(filename, "r", encoding="utf-8") as file:

            return jinja2.Template(file.read())



    except FileNotFoundError as e:

        print(e)

        sys.exit(1)





def create_configuration(devices: List[Device], t: jinja2.Template) -> bool:

    """Function to create configuration files."""

    # Render template

    os.makedirs("output", exist_ok=True)

    try:

        for device in devices:

            with open(f"output/{device.hostname}.txt", "w", encoding="utf-8") as f:

                f.write(t.render(device=device))



        return True



    except Exception as e:

        print(e)

        return False





# Main

if __name__ == "__main__":

    # Get arguments

    args = read_args()



    # Load inventory

    try:

        inventory = load_inventory(args.data)

    except FileNotFoundError as e:

        print(e)

        sys.exit(1)



    # Load template

    template = load_template(args.template)



    # Create configuration

    if create_configuration(inventory, template):

        print("Configuration files created.")

    else:

        print("Something went wrong.")

        sys.exit(1)

If you have questions to parts of this code, we encourage you to list the whole blog series “From Python to Go“, as all the concepts are already explained apart from the templates, which were explained earlier in this blog.

A few important remarks:

1. We use here pydantic instead of dataclasses. The main reason is that is easier to populated nested objects from dictionary using pydantic as it does all the heavy-lifting for you. We also add default value for “None” for many keys, where we expect the value may be missing in the input. This is a good practice to ensure key consistency and will simply templates. it is also inline with Go (Golang) structs, which have default values for all the keys, if they aren’t provided.
2. Reading CLI arguments, Parsing YAML files, and catching exceptions are already known to you.
3. To create template object, you read content of the file into “Jinja2.Template()” class.
4. To create directory for output file, the function “mkdirs()” from the built-in “os” package is used.
5. Templating is done using “render()” method, which takes as input the variable, which shall be matched to one in the template and produce as the output the string, which is saved then in the file.
6. Output files are named after the host names of the devices.

And now Jinja2 code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
{%- if device.hostname -%}

hostname {{ device.hostname }}

!

{%- endif %}

{%- if device.interfaces -%}

{% for interface in device.interfaces %}

interface {{ interface.name }}

{%- if interface.description %}

  description "{{ interface.description }}"

{%- endif %}

{%- if interface.enabled %}

  no shutdown

{%- else %}

  shutdown

{%- endif %}

{%- if interface.ip4 %}

  no switchport

  ip address {{ interface.ip4.address }}/{{ interface.ip4.prefix }}

{%- else %}

  switchport

{%- endif %}

!

{%- endfor %}

{%- endif %}

As this is configuration file for network device, we have a number of checks, we need to do before templating parts of configuration. For example, if there is no description is provided for interface we shall not type just “description” word without value as it will cause an issue, when we are to apply configuration to the device.

Let’s execute this code:

1
2
$ python main.py -d ../data/devices.yaml -t devices.j2

Configuration files created.

As a result of this code execution, the following files we created:

1
2
3
4
5
6
7
$ tree

.

├── devices.j2

├── main.py

└── output

    ├── leaf1.txt

    └── leaf2.txt

Let’s take a look inside a file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ cat output/leaf1.txt 

hostname leaf1

!

interface Ethernet1

  description "link to leaf2 Ethernet1"

  no shutdown

  no switchport

  ip address 10.0.0.0/31

!

interface Ethernet2

  description "link to spine1 Ethernet1"

  no shutdown

  switchport

!

Code examples and input files are provided in the link in the bottom of this blog post.

Go (Golang)

From the Go (Golang) perspective, the only external package we need is the one to parse YAML files:

1
$ go get gopkg.in/yaml.v3

Once that is available, you can develop the application in Go (Golang) to meet our business requirements:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
/* From Python to Go: Go: 014 - Templating configuration. */



package main



// Imports

import (

    "flag"

    "fmt"

    "os"

    "text/template"



    "gopkg.in/yaml.v3"

)



// Types

type IPAddress struct {

    /* Class to store IP address data. */

    Address string `yaml:"address"`

    Prefix  int    `yaml:"prefix"`

}

type Interface struct {

    /* Class to store interface data. */

    Name        string    `yaml:"name"`

    Description string    `yaml:"description"`

    IP4         IPAddress `yaml:"ip4"`

    Enabled     bool      `yaml:"enabled"`

}

type Device struct {

    /* Class to store credentials. */

    Hostname   string      `yaml:"hostname"`

    Interfaces []Interface `yaml:"interfaces"`

}

type Arguments struct {

    /* Class to starte CLI arguments */

    Data     string

    Template string

}



// Functions

func readArgs() Arguments {

    /* Helper function to read CLI arguments */

    result := Arguments{}



    flag.StringVar(&result.Data, "d", "", "Path to the input file")

    flag.StringVar(&result.Template, "t", "", "Path to the template.")



    flag.Parse()



    return result

}



func loadInventory(p string) *[]Device {

    /* Function to load inventory data. */



    // Open file

    bs, err := os.ReadFile(p)

    if err != nil {

        fmt.Println("Get error ", err)

        os.Exit(1)

    }



    // Load inventory

    result := &[]Device{}



    err = yaml.Unmarshal(bs, result)

    if err != nil {

        fmt.Println("Get error ", err)

        os.Exit(1)

    }



    // Return result

    return result

}



func loadTemplate(p string) *template.Template {

    /* Helper function to load template. */



    // Load template

    templ, err := template.New(p).ParseFiles(p)

    if err != nil {

        fmt.Println("Get error ", err)

        os.Exit(1)

    }



    // Return result

    return templ

}



func createConfiguration(d *[]Device, t *template.Template) bool {

    /* Function to create configuration files. */



    // Create output directory

    err := os.MkdirAll("output", 0777)

    if err != nil {

        fmt.Println("Get error ", err)

        return false

    }



    // Render template

    for i := 0; i < len(*d); i++ {

        // Create file

        f, err := os.Create("output/" + (*d)[i].Hostname + ".txt")

        if err != nil {

            fmt.Println("Get error ", err)

            f.Close()

            return false

        }



        // Render template

        t.Execute(f, (*d)[i])



        // Close file

        f.Close()

    }



    return true

}



// Main

func main() {

    // Get arguments

    args := readArgs()



    // Load inventory

    inventory := loadInventory(args.Data)



    // Load template

    templ := loadTemplate(args.Template)



    // Create configuration

    if createConfiguration(inventory, templ) {

        fmt.Println("Configuration files created.")

    } else {

        fmt.Println("Something went wrong.")

    }

}

Same as we mentioned after the Python’s snippet, If you have questions to parts of this code, we encourage you to list the whole blog series “From Python to Go“.

Breakdown:

1. We put majority of functionality in separated functions to make the “main()” functions clean and easily understandable.
2. Reading CLI arguments, Parsing YAML files, and catching exceptions are already known to you.
3. To create template object, you provide path towards your template just as sting to “New()” function from “text/template” module AND to “ParseFiles()” receiver function of created struct as result of “New()“. This can be done in two steps, but can be also in a single.
4. The new file is created using “os.Create()” function, which either creates a new file or truncates content of the existing one, if there is something. The opened file shall be closed using receiver function, as “f.Close()“.
5. The templating is done using “Execute()” receiver function, which takes the created file as the first argument and user data, which will be associated with cursor inside the template as the second one.
6. For each device the new file is created.

And here is Go-template:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
{{- if .Hostname -}}

hostname {{ .Hostname }}

!

{{- end }}

{{- if .Interfaces }}

{{- range .Interfaces }}

interface {{ .Name }}

{{- if .Description }}

  description "{{ .Description }}"

{{- end }}

{{- if .Enabled }}

  no shutdown

{{- else }}

  shutdown

{{- end }}

{{- if .IP4.Address }}

  no switchport

  ip address {{ .IP4.Address }}/{{ .IP4.Prefix }}

{{- else }}

  switchport

{{- end }}

!

{{- end -}}

{{- end }}

This templates perform absolutely the same job as the one in Python/Jinja2 above. There is one more important difference though between this template and what we have in Jinja2:

• In Jinja2 we use pydantic BaseModel objects. So, for IP address we either have None value, if there is no IP address value provided, which is also a default one, or the specific IP addresses. As such, check “{% if interface.ip4 %}” is vaild.
• In Go-template we consume structs, which have all the keys all the time. So check “{{ if .IP4 }}” won’t work, because there is a nested struct further, even if that is empty. in Go (Golang) itself you can use reflection, which I haven’t found a way how to use in template yet. So we used check “{{ if .IP4.Address }}” to ensure that nested IP address is not provided.

Let’s execute this code:

1
2
$ go run . -d ../data/devices.yaml -t devices.tmpl 

Configuration files created.

The resulting file structure:

1
2
3
4
5
6
7
8
9
10
11
$ tree

.

├── devices.tmpl

├── go.mod

├── go.sum

├── main.go

└── output

    ├── leaf1.txt

    └── leaf2.txt



1 directory, 6 files

And one of the resulting files:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ cat output/leaf1.txt 

hostname leaf1

!

interface Ethernet1

  description "link to leaf2 Ethernet1"

  no shutdown

  no switchport

  ip address 10.0.0.0/31

!

interface Ethernet2

  description "link to spine1 Ethernet1"

  no shutdown

  switchport

!

Which is identical to what you got as a result of Python/Jinja2.

Lessons in GitHub

You can find the final working versions of the files from this blog at out GitHub page.

Conclusion

Templating is the first essential steps in network /IT infrastructure automation. Once you master it, the next step will be to apply it to network devices / servers. Take care and good bye.

Support us

Support new interop and automation articles at karneliuk.com
EUR

I want to support with:
9.99 EUR4.99 EUR24.99 EUR49.99 EUR99.99 EUR199.99 EUR

P.S.

If you have further questions or you need help with your networks, I’m happy to assist you, just send me message. Also don’t forget to share the article on your social media, if you like it.

BR,

Anton Karneliuk