Hello my friend,
This blog continues discussion of how to manage devices (network switches and routers, servers, virtual machines, etc) using SSH, which we started in previous blog. In this discussion we’ll cover advanced interaction with devices, which include multiple commands, change of contexts and validations. Let’s dive in.
If You Say “From Python to Go”, Shall I Learn Python?
Each programming language has its strong and weak capabilities. Golang, by virtue being a low level (or at least much lower level compared to Python) is very good, when performance and efficiency are paramount. However, you don’t need it for all applications. Python give quicker time to market, possibility to iteratively develop your code with Jupyter and vast ecosystem of existing libraries. Both programming languages are important and both of them play crucial role in IT and network infrastructure management. So if you are good with Python, learn Go (Golang) using our blog series.
And if you are not, or you want to have good intro to IT and network automation holistically, enroll to our training programms:
We offer the following training programs in network automation for you:
- Zero-to-Hero Network Automation Training
- High-scale automation with Nornir
- Ansible Automation Orchestration with Ansble Tower / AWX
- Expert-level training: Closed-loop Automation and Next-generation Monitoring
During these trainings you will learn the following topics:
- Success and failure strategies to build the automation tools.
- Principles of software developments and the most useful and convenient tools.
- Data encoding (free-text, XML, JSON, YAML, Protobuf).
- Model-driven network automation with YANG, NETCONF, RESTCONF, GNMI.
- Full configuration templating with Jinja2 based on the source of truth (NetBox).
- Best programming languages (Python, Bash) for developing automation
- The most rock-solid and functional tools for configuration management (Ansible) and Python-based automation frameworks (Nornir).
- Network automation infrastructure (Linux, Linux networking, KVM, Docker).
- Orchestration of automation workflows with AWX and its integration with NetBox, GitHub, as well as custom execution environments for better scalability.
- Collection network data via SNMP and streaming telemetry with Prometheus
- Building API gateways with Python leveraging Fast API
- Integration of alerting with Slack and your own APIs
- … and many more
Moreover, we put all mentions technologies in the context of real use cases, which our team has solved and are solving in various projects in the service providers, enterprise and data center networks and systems across the Europe and USA. That gives you opportunity to ask questions to understand the solutions in-depth and have discussions about your own projects. And on top of that, each technology is provided with online demos and labs to master your skills thoroughly. Such a mixture creates a unique learning environment, which all students value so much. Join us and unleash your potential.
Start your automation training today.
What Are We Going To Talk Today?
As aforementioned, we continue discussion how to interact with IT and network devices (routers, switches, firewalls, servers, virtual machines, containers, etc) using SSH. In previous blog we showed a simple scenario, where you send one command and receive one output. Whilst it was possible to extend it easily for Python, it was more problematic for Go (Golang). In today’s blog we will show how to significantly improve the capability:
- How is that possible to send multiple commands to the SSH remote host?
- What are the associated problems?
- What is general approach to do changes on IT/network equipment with SSH?
- What are the purpose-built libraries out there to improve experience?
Explanation
When we do real production-grade interaction with remote devices, for example for the purpose of making change, we normally send a lot of commands as well as receiving a lot of output. Lets take a look at few examples:
Configuration of new BGP neighbor
In this change you would need to:
- Create a route policy, which filters incoming/outgoing network reachability information (NLRI) updates (simply, route updates) towards the new neighbor.
- Possibly these route policies will require prefix lists and/or community lists to be created/modified as well.
- Then to enter the BGP routing configuration context, create a neighbor with all relevant details including created policies
This configuration alone will easily be a few dozens of configuration lines long. As we talk though about production-grade changes, you would also want to capture operational state of BGP before and after the change to ensure you achieved the relevant result.
Installation of New Packages in Linux
In this change you would need to:
- Upload the relevant packages on Linux servers
- Unpack them and make relevant permission adoptions
- Perform installation
- Make adjustment to existing services or create a new one.
Same as above for network example, you may need quite a few commands here, as well as logic to capture the state of the Linux before and after the change if you do in production systems.
Ansible is very good for server management, and you can learn it in-dept at our network automation trainings.
Challenges with SSH
We briefly mentioned in the previous blog post, but it is worth reiterating it here as well. The nature of SSH communication requires client to send requests to server, wait for response and then to process it. There could be multiple ways how client can identify if the response is received in full; however, the most robust is to parse the received text from server until the specific CLI pattern is found. Whilst it sounds easy in straightforward, it is a little bit more challenging when dealing with network equipment, as CLI pattern changes during the configuration, when you traverse the configuration contexts, e.g.:
2
3
4
karneliuk-router-01# configure terminal
karneliuk-router-01(config)# interface Loopback 10
karneliuk-router-01(config-if)#
The snippet above is the one you will see in Cisco IOS / IOS XE, Cisco IOS XR and Cisco NX-OS operating system as well as in Arista EOS. Nokia SR OS and Junipers are slightly different, but they have the same idea. This means that your application, which interacts with network devices via SSH be able to dynamically adapt to changing prompt.
Another challenge is, let’s admit, SSH communication happen over network, which is not always stable. The remote host you communicating with also may experience issues, so your application need some timeout handling to detect that communication is not happening anymore; otherwise, there is a to get stuck on a dead session.
All this mean that you need to have good tooling for SSH to handle all these cases.
Purpose-built SSH Libraries
Good news is that such libraries to exist. There is a number of such libraries in Python, less in Go (Golang). There is though one, which is from my experience is quite good and it exists both in Python and Go (Golang). Imagine, same-ish syntax and capabilities in both Python and Go (Golang), isn’t it beautiful?
The library we are going to use in today’s blog is called Scrapli:
The reason we are going to use this library is because it addresses all the aforementioned challenges, has good documentation and extensive support, both from maintainer and from community.
Example
Picture costs thousand words; so we will demonstrate you how to change the script from the previous blog post to be production-grade leveraging scrapli in both Python and Go (Golang). Namely, we would:
- Capture state of device using specific show command before and after the change.
- Perform multiline configuration change.
- Compare two captured state to generate a difference to simplify engineering job analyzing the impact.
Python
There is number of 3rd party libraries, which shall be installed for this code to work:
Once libraries are installed, let’s progress with developing our application with Python:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
# Modules
import argparse
import datetime
import os
import sys
import difflib
from dataclasses import dataclass
from typing import List
import yaml
from scrapli import Scrapli
# Classes
@dataclass
class Credentials:
"""Class to store credentials."""
username: str
password: str
@dataclass
class Instruction:
"""Class to store instructions."""
command: str
config: List[str]
@dataclass
class Result:
"""Class to store command execution data."""
instruction: Instruction
diff: str
timestamp: datetime.datetime
class Device:
"""Class to interact with netowrk device."""
def __init__(self, hostname: str, ip_address: str, platform: str, credentials: Credentials):
self.hostname = hostname
self.ip_address = ip_address
self.platform = platform
self.credentials = credentials
self.results: List[Result] = []
def execute_change(self, instruction: Instruction) -> None:
"""Method to execute change."""
# Connect to device
with Scrapli(
host=self.ip_address,
auth_username=self.credentials.username,
auth_password=self.credentials.password,
platform=self.platform,
auth_strict_key=False,
) as conn:
# Get state before change
before = conn.send_command(instruction.command)
# Apply change
conn.send_configs(instruction.config)
# Get state after change
after = conn.send_command(instruction.command)
# Diff
diff = "\n".join(
difflib.context_diff(
before.result.splitlines(),
after.result.splitlines(),
lineterm="",
)
)
self.results.append(
Result(
instruction=instruction,
diff=diff,
timestamp=datetime.datetime.now(),
)
)
# Functions
def read_args() -> argparse.Namespace:
"""Helper function to read CLI arguments."""
parser = argparse.ArgumentParser(description="User input.")
parser.add_argument("-i", "--inventory", type=str, help="Path to inventory file.")
return parser.parse_args()
def load_inventory(filename: str, credentials: Credentials) -> List[Device]:
"""Function to load inventory data."""
# Open file
try:
with open(filename, "r", encoding="utf-8") as f:
data = yaml.safe_load(f)
except FileNotFoundError as e:
print(e)
sys.exit(1)
# Populate list of devices
result = []
for device in data:
result.append(Device(credentials=credentials, **device))
return result
def get_credentials() -> Credentials:
"""Function to get credentials."""
username = os.getenv("AUTOMATION_USER")
password = os.getenv("AUTOMATION_PASS")
return Credentials(username, password)
# Main code
if __name__ == "__main__":
# Read CLI arguments
args = read_args()
# Get credentials
credentials = get_credentials()
# Load inventory
devices = load_inventory(args.inventory, credentials=credentials)
# Config
instruction = Instruction(
command="show interfaces description",
config=["interface Loopback 23", "description Test"],
)
# Execute command
for device in devices:
device.execute_change(instruction)
# Print results
for device in devices:
print(f"Device: {device.hostname}")
for result in device.results:
print(f"Config: {result.instruction.config}", f"Impact: {result.diff}", f"Timestamp: {result.timestamp}", sep="\n")
It is strongly recommended to read previous blog post about SSH, as there is a number of concepts and structures explained there, which are omitted in this blog post for brevity.
The key new concepts are the following:
- Two new libraries are imported:
- scrapli for advanced SSH interaction with remote network devices or servers
- difflib to compare two text files (output of captured states) and outline differences
- The new data class “Instruction” is introduced to store both the validation command as well as configuration, which is to be applied to network devices. In this example this class is populated with:
- command “show interfaces description” to get the description of all the interfaces configured on the target network device.
- config commands, which is a list of strings, which contain the configuration , which shall be pushed to network devices.
- From the previous example, method “execute_command()” is renamed to “execute_change()” and it expects now the object of Instruction class as an argument.
- Within this function we do the interaction with network device:
- Using class “Scrapli()” we create object for SSH connectivity with the device. In addition to IP address/FQDN and credentials, it requires to provide a platform value, which shall be from the core supported ones.
- Object is created using “with … as …” context manager, which means the SSH channel will be “open()” and “close()” automatically.
- To send non-configuration command, method “send_command()” is used. If there are multiple commands are to be sent, then “send_commands()” can be used instead.
- To send configuration, method “send_config()” is used. If there are multiple commands are to be sent, then “send_configs()” can be used instead.
- Those four commands return Response (or MultiResponse) objects from “scrapli” library, which has field “result“, storing the string output from network device.
- Finally, using function “context_diff()” from “difflib” library, the difference between two outputs is identified.
- Afterwards the result are stored on in the modified Result, which then is printed back in the main body
To execute this script, much like in the previous blog, you need to set environment variables with credentials (username and password), so that your application can read it:
2
$ export AUTOMATION_PASS='***'
Then execute the script:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
UserWarning:
**************************************************************************************************************************** Authentication Warning! *****************************************************************************************************************************
scrapli will try to escalate privilege without entering a password but may fail.
Set an 'auth_secondary' password if your device requires a password to increase privilege, otherwise ignore this message.
**********************************************************************************************************************************************************************************************************************************************************************************
warn(warning_message)
Device: dev-pygnmi-eos-001
Command: ['interface Loopback 23', 'description Test']
Impact: ***
---
***************
*** 2,6 ****
--- 2,7 ----
Et1 up up
Et2 up up
Lo0 up up
+ Lo23 up up Test
Lo51 admin down down pytest-update-test-33
Ma1 up up
Timestamp: 2025-03-07 21:24:11.267377
Execution of change went successfully and you see diff, which shows that new interface was added (identified by “+” symbol).
Now let’s implement the same use case in Go (Golang).
Go (Golang)
The following packages are to be installed:
2
3
$ go get gopkg.in/yaml.v3
$ go get github.com/scrapli/scrapligo
In addition scrapli and yaml, we need to installed 3rd go-cmp party for comparing texts. It is possible to complete use case without it; however, it will take more time to write logic for comparing texts.
Now the code for this application, which performs advanced SSH management:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
package main
import (
"flag"
"fmt"
"log"
"os"
"strings"
"time"
"github.com/google/go-cmp/cmp"
"github.com/scrapli/scrapligo/driver/options"
"github.com/scrapli/scrapligo/platform"
"gopkg.in/yaml.v3"
)
// Imports
// Types and Receivers
type Arguments struct {
/* Class to starte CLI arguments */
Inventory string
}
type Crendetials struct {
/* Struct to store credentials. */
Username string
Password string
}
type Instruction struct {
Command string
Config []string
}
type Result struct {
/* Struct to store command execution result. */
Instruction Instruction
Diff string
Timestamp time.Time
}
type Device struct {
/* Struct to interact with netowrk device. */
Hostname string `yaml:"hostname"`
IpAddress string `yaml:"ip_address"`
Platform string `yaml:"platform"`
Crendetials Crendetials
Result []Result
}
func (d *Device) executeChange(i Instruction) {
/* Method to execute command */
// Get platform
p, err := platform.NewPlatform(
(*d).Platform,
(*d).IpAddress,
options.WithAuthNoStrictKey(),
options.WithAuthUsername(d.Crendetials.Username),
options.WithAuthPassword(d.Crendetials.Password),
)
if err != nil {
log.Fatalln("failed to create platform; error: ", err)
}
// Get netowrk driver
dr, err := p.GetNetworkDriver()
if err != nil {
log.Fatalln("failed to fetch network driver from the platform; error ", err)
}
// Open session
err = dr.Open()
if err != nil {
log.Fatalln("failed to open driver; error: ", err)
}
defer dr.Close()
// Get change before start
before, err := dr.SendCommand(i.Command)
if err != nil {
log.Fatalln("failed to send command; error: ", err)
}
// Apply change
_, err = dr.SendConfigs(i.Config)
if err != nil {
log.Fatalln("failed to send config; error: ", err)
}
// Get state after change
after, err := dr.SendCommand(i.Command)
if err != nil {
log.Fatalln("failed to send command; error: ", err)
}
// Diff
diff := cmp.Diff(strings.Split(before.Result, "\n"), strings.Split(after.Result, "\n"))
// Update the result
(*d).Result = append((*d).Result, Result{
Instruction: i,
Diff: diff,
Timestamp: time.Now(),
})
}
// Functions
func readArgs() Arguments {
/* Helper function to read CLI arguments */
result := Arguments{}
flag.StringVar(&result.Inventory, "i", "", "Path to the inventory file")
flag.Parse()
return result
}
func loadInventory(p string) *[]Device {
/* Function to load inventory data. */
// Open file
bs, err := os.ReadFile(p)
if err != nil {
fmt.Println("Get error ", err)
os.Exit(1)
}
// Load inventory
result := &[]Device{}
err = yaml.Unmarshal(bs, result)
if err != nil {
fmt.Println("Get error ", err)
os.Exit(1)
}
// Return result
return result
}
func getCredentials() Crendetials {
/* Function to get credentials. */
return Crendetials{
Username: os.Getenv("AUTOMATION_USER"),
Password: os.Getenv("AUTOMATION_PASS"),
}
}
// Main
func main() {
/* Core logic */
// Read CLI arguments
cliArgs := readArgs()
// Get credentials
sshCreds := getCredentials()
// Load inventory
inventory := loadInventory(cliArgs.Inventory)
// Config
instruction := Instruction{
Command: "show interfaces description",
Config: []string{
"interface Loopback 23",
"description Go_Test_2",
},
}
// Execute commands
for i := 0; i < len(*inventory); i++ {
(*inventory)[i].Crendetials = sshCreds
(*inventory)[i].executeChange(instruction)
}
// Print results
for i := 0; i < len(*inventory); i++ {
for j := 0; j < len((*inventory)[i].Result); j++ {
fmt.Printf(
"Config: %v\nImpact: %v\nTimestamp: %v\n",
(*inventory)[i].Result[j].Instruction.Config,
(*inventory)[i].Result[j].Diff,
(*inventory)[i].Result[j].Timestamp,
)
}
}
}
It is strongly encouraged to read the previous blog about basic SSH interaction with network devices and servers with Python and Go (Golang) to get grasp on the code before reading explanation below.
The details of what’s new:
- The relevant third party libraries (ones mentioned above) are imported.
- Struct Instruction is added with the same keys and data types as Instruction data class in Python.
- Receiver function “func (d *Device) executeChange(i Instruction)” replaced “func (d *Device) executeCommand(i string)“, following the same logic as explained in Python section.
- Within that function interaction with network device is done:
- First of all, using function NewPlatform(), the pointer to struct for platform is created. Same as with Scrapli() class in Python, NewPlatform requires actual platform name, which is supported by Scrapli package.
- From the Platform pointer the new device driver is obtained using receiver function (method) GetNetworkDriver(), which is also pointer.
- Using Open() function the SSH session is established and using the “defer Close()” the closure is scheduled right before the exit out of the function.
- Much as in Python, there are four main methods to interact with the remote host:
- “func (d *network.Driver) sendCommand(command string)” to execute single non-configuration command.
- “func (d *network.Driver) sendCommands(commands []string)” to execute multiple non-configuration command.
- “func (d *network.Driver) sendConfig(config string)” to execute single configuration command.
- “func (d *network.Driver) sendConfigs(config []string)” to execute multiple configuration command.
- All these methods returns pointer to Response struct, which has key “Result” storing output from network device.
- Using “Diff()” function from “go-cmp” package the two slices of strings are compared and it is identified what is different.
- Then results are printed.
You shall have credentials set in your environment by now. See Python part for details.
Execution of this newly developed Go (Golang) application:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Config: [interface Loopback 23 description Go_Test_23]
Impact: []string{
... // 2 identical elements
"Et2 up up",
"Lo0 up up",
strings.Join({
"Lo23 up up ",
" Go_Test_2",
+ "3",
}, ""),
"Lo51 admin down down "...,
"Ma1 up up",
}
Timestamp: 2025-03-08 17:36:35.170185362 +0000 GMT m=+2.091754052
The output is slightly more cryptic here compared to Python:
- In Python the entire line was highlighted, which was new or different
- In Go (Golang) only the new characters are shown
Lessons in GitHub
You can find the final working versions of the files from this blog at out GitHub page.
Conclusion
In this blog post you have learned two things, critical to IT and network automation: what is the correct process to implement change including pre and post checks as well how to interact with remote hosts via SSH at advanced level. Coupling this knowledge with templating, and in general foundational skills you learn before in this blog series, you are in a good position to start automating. Take care and good bye.
Support us
P.S.
If you have further questions or you need help with your networks, I’m happy to assist you, just send me message. Also don’t forget to share the article on your social media, if you like it.
BR,
Anton Karneliuk
