Hello my friend,

We have spent some time on configuration of the network functions using NETCONF/YANG using native vendor models and OpenConfig. Now the time is come to talk about information, which we can extract from the router in YANG data models, what is also called model-driven telemetry.

Brief description

Nowadays telemetry is a hot topic for discussion on various network events, conferences or just vendors’ presentation to customers. As usual, without knowing the details about what it is, how it could be used and which benefit it can bring you, such discussions aren’t more than just a hype. So in this article we’ll try to discovery what telemetry means, for sure, with examples in multivendor environment with Cisco IOS XR and Nokia (Alcatel-Lucent) SR OS.

For a long time, SNMP was the primary source of the information about state of the network. It has evolved a lot since its first release and current version 3 has even built-in security using encryption and hashing. Nevertheless, not all the information about network devices are available through SNMP, that’s why still some information is gathered through show commands and proper parsing, which isn’t flexible task. On the other hand, new era of networking, where SDN comes at the scene, requires much more information from the network about its actual state; this is information must be available to SDN controller or network analytics instantly on anytime basis.

And that’s where model-driven telemetry could help a lot. Model-driven means that telemetry is based on the YANG modules, which are complementary to configuration modules we have used a lot previously. It means that virtually all information about network states should be available through operational YANG modules

As you’ve already learned, there are different types of YANG modules available: vendor-native and vendor-neutral (like OpenConfig). In this article we’ll focus on the vendor-native modules for Cisco IOS XR and Nokia (Alcatel-Lucent) SR OS, as it’s the first natural step in telemetry. In the following articles we’ll review telemetry available in OpenConfig YANG modules as well.

In terms of transport protocol, which is used for model-driven telemetry, there are different options. The most popular deployment is based on gRPC (or recently gNMI), which uses HTTP and perfectly fits for transmission of the structured data. There is another option, which is to use TCP directly, but if we speak about cross-vendor implementation, gRPC/gNMI looks more favorable.

Nevertheless, in our examples you will see some NETCONF, though NETCONF itself isn’t supposed to be transport for telemetry. The reason for that is that we don’t build any telemetry collector yet, but rather focusing on the data collecting itself. Probably we’ll review the streaming of telemetry data in the upcoming articles.

What are we going to test?

We are going to collect information from the network functions in their native YANG modules for operational data for:

• Nokia (Alcatel-Lucent) SR OS 16.0.R3
• Cisco IOS XR 6.1.2

We’ll use “netconf_get” Ansible module for that.

Software version

The following software components are used in this lab:

• CentOS 7 with python 2.7.
• Ansible 2.6.4
• Nokia SR OS 16.0.R3 [guest VNF]
• Cisco IOS XR 6.1.2 [guest VNF]

See the previous article to get details how to build the lab

Topology

We are using our standard topology:

The logical topology is quite simple so we have just back to back connectivity between Nokia SR OS based VNF SR1 and Cisco IOS XR based VNF XR3, it’s just expressed in ASCII:

The initial configuration you can see in the attached files: 136_config_initial_XR3 136_config_initial_SR1

Brief topology check

From the configuration prospective we have configured just 3 topics:

• Interfaces;
• OSPF including Segment Routing;
• BGP for VPNv4/VPNv6 unicast address families;

This set of the configuration is very basic on the one hand; on the other hand, it represents basis for modern service provider network capable to provide all kind of IP/MPLS services for customers. We won’t go into explanation of the details, as each of them were covered into details earlier in multivendor setup with Nokia (Alcatel-Lucent) SR OS and Cisco IOS XR: interfaces, OSPF, Segment Routing and BGP IP VPNs. Here we’ll just review some operational commands.

We start with the status of the ports and interfaces on Nokia (Alcatel-Lucent) SR OS based network function SR1:

===============================================================================
Ports on Slot A
===============================================================================
Port Admin Link Port Cfg Oper LAG/ Port Port Port C/QS/S/XFP/
Id State State MTU MTU Bndl Mode Encp Type MDIMDX
——————————————————————————-
A/1 Up Yes Up 1514 1514 – netw null faste MDI
A/4 Up No Ghost 1514 1514 – netw null faste
===============================================================================

[]
A:admin@SR1# show router interface

Then we check the adjacency of the OSPF:

Followed by the routing table:

And we finish with the status of BGP peering:

For Cisco IOS XR based network function XR3 we take the same sequence of the verification activities. Here are we have status of the interfaces at XR3:

Then we check status of OSPF adjacency:

Neighbors for OSPF 0

Neighbor ID Pri State Dead Time Address Interface
10.0.0.11 1 FULL/ – 00:00:35 10.11.33.11 GigabitEthernet0/0/0/0
Neighbor is up for 00:02:07

Then routing table:

Gateway of last resort is not set

And we are finishing with BGP peering for XR3:

Process RcvTblVer bRIB/RIB LabelVer ImportVer SendTblVer StandbyVer
Speaker 1 1 1 1 1 0

Neighbor Spk AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down St/PfxRcd
10.0.0.11 0 65000 6 7 1 0 0 00:01:35 0

As you see, adjacency is established both for OSPF and BGP. Actually, we could have omitted check of the routing table as BGP check implicitly includes it: if IPv4 addresses of loopbacks aren’t propagated properly, BGP session won’t come up.

After we have verified state of the network, we will check the same parameter using operational YANG models, hence we’ll go for telemetry data.

Cisco IOS // Operational YANG modules

In one of the OpenConfig articles we have collected all the YANG modules available on the Cisco IOS XR devices. Besides OpenConfig YANG modules, what we have used, there are also a lot of native modules, which are either configuration modules or operational ones. The operational has word “oper” in its name like on the example below:

To understand what is inside, we can take an example of some module and render it using “pyang”:

As you might spot, all the nodes in this YANG module have “ro” attribute, meaning they are read only and we only can read data from them, not to configure. For each configuration YANG module there is typically operational module available.

Cisco IOS XR // Model-driven telemetry algorithm

As we’ve said in the beginning, we’ll try to collect information about network state in YANG operational modules using NETCONF. To do that we’ll use “netconf_get” Ansible module, which we have covered earlier. The algorithm is quite straightforward:

• We define some so called “configuration profiles” we’d like to collect telemetry data about. In our case they are: interfaces, OSPF routing and BGP routing
• Per each profile we have dedicated NETCONF request as Cisco IOS XR uses different YANG modules for different operational data
• Fetch telemetry data using NETCONF/YANG
• Parse data to extract some information

The last point is a bit synthetic in this article, as in real telemetry scenario you will use some messaging bus as Kafka for example, which will store those data, so you won’t need to parse it in that way. But about Kafka will talk in the separate article.

Cisco IOS XR // Ansible playbook for telemetry collection

To implement the algorithm above, our playbook using roles has the following structure:

The master Ansible playbook has very easy structure:

Same structure we have used more or less with all of our Ansible playbook with roles.

The “group_vars” for Cisco IOS XR contains authentication data and some other general parameters:

Before we go deeper into details of playbooks, let’s take a look into file with “vars” for Cisco IOS XR based network function XR3:

As we have mentioned above, you can see here different configuration profiles, which are used for checking telemetry.

In reality, initial configuration was also done using these configuration profiles based configuration YANG modules, but this is out of scope for this article.

Now, it’s time to take a look into working horses of this automation using Ansible for telemetry, that is actual playbooks with tasks. Let’ start with “main.yml”:

– name: VERIFICATION // {{ inventory_hostname }} // CREATING TEST REPORT
file:
dest: /tmp/{{ inventory_hostname }}_test_report.txt
state: touch

– name: VERIFICATION // {{ inventory_hostname }} // COLLECTING TELEMETRY AND SEARCHING DATA
include_tasks: comparing_loop.yml
loop: “{{ PROFILE.node.configuration_profiles }}”

– name: VERIFICATION // {{ inventory_hostname }} // COMPILING REPORT
shell: “cat /tmp/temp_report_136_{{ inventory_hostname }}_* > /tmp/{{ inventory_hostname }}_test_report.txt”

Take a look at article about Ansible roles, if you have questions on this structure.

In the playbook above we do the following actions:

• We import the file with variables, which contains configuration profiles to check
• We try to delete previous file with report. Even if it doesn.t exist, we go further
• We collect telemetry for each profile (looping per profile name) using external playbook with tasks called “comparing_loop.yml”
• We combine all the files with reports in a single report
• We send message that everything is done

You might have spot that the crucial part is concentrated in the 3^rd task, where we collect telemetry using another playbook, hence the next obvious step is to review that playbook:

– name: VERIFICATION // {{ inventory_hostname }} // COLLECTING TELEMETRY AND SEARCHING DATA // MODIFIICATION OF COLLECTED TELEMETRY FOR PYTHON PROCESSING
replace:
path: /tmp/{{ inventory_hostname }}/{{ inventory_hostname }}_{{ item.profile }}_yang_telemetry.json
regexp: ‘-‘
replace: ‘_’

– name: VERIFICATION // {{ inventory_hostname }} // COLLECTING TELEMETRY AND SEARCHING DATA // IMPORTING COLLECTED TELEMETRY DATA
include_vars:
file: /tmp/{{ inventory_hostname }}/{{ inventory_hostname }}_{{ item.profile }}_yang_telemetry.json
name: COLLECTED

In this playbook we have the same number of tasks, as in the previous one, so it’s quite big. I guess, you want to know, what exactly we are doing here. Here we go:

• We send the RPC message over NETCONF to collect live data for particular operational YANG modules. The proper request is chosen from the template “netconf_request.j2” based on the name of the configuration profile
• We save output of the previous command to “/tmp” folder so that we can review details later on.
• We replace all “-“ with “_” in saved telemetry output, as it’s impossible in Python for variable to contain “-“ in its name.
• We import back saved telemetry data as variables.
• We look for some interesting for us parameters in telemetry based on some predefined (by ourselves) template.

As a lot of real action is dependent on files with templates, we need to understand them. The first template called “netconf_request.j2” is used for creating proper NETCONF request:

{% elif item.profile == ‘routing_ospf’ %}

{% elif item.profile == ‘routing_bgp’ %}

Depending on the name of the configuration profile, we create proper request. The namespace and the name of the block we check within file with the operational YANG modules. The following example for interfaces:

For XML schema we take the parameter of the “namespace” from the corresponding YANG module. For the request itself we take the name of inter interesting high-level container. In the example above we have 2 high-level containers: “interface-dampening” and “interface-properties”. We are interested only in the second one, so we put its name.

The rest of the templates are used to extract some information from the telemetry. As an example, for interfaces we check the status of the ports:

{% endfor %}

For OSPF routing we check the status of the neighbors:

{% else %}
After software update:
There is no OSPF neighbors detected

{% endif %}

And for the BGP routing, we check the status of the BGP peers:

{% else %}
Problem with BGP process on local node ({{ inventory_hostname }})

{% endif %}

When the preparation is done meaning all the playbooks, variables and templates are defined, we can test them.

Cisco IOS // Collecting telemetry via NETCONF/YANG with Ansible

For now the names of the variable from telemetry don’t say you anything, so let’s execute our Ansible playbook to collect telemetry:

TASK [Gathering Facts] *************************************************************************************************************************************************
ok: [XR3]

TASK [cisco/136_lab : VERIFICATION // XR3 // IMPORTING INFRASTRUCTURE PROFILE] *****************************************************************************************
ok: [XR3]

TASK [cisco/136_lab : VERIFICATION // XR3 // DELETE PREVIOUS TEST REPORT] **********************************************************************************************
changed: [XR3]

TASK [cisco/136_lab : VERIFICATION // XR3 // CREATING TEST REPORT] *****************************************************************************************************
changed: [XR3]

TASK [cisco/136_lab : VERIFICATION // XR3 // COLLECTING TELEMETRY AND SEARCHING DATA] **********************************************************************************
included: /home/aaa/ansible/roles/cisco/136_lab/tasks/comparing_loop.yml for XR3
included: /home/aaa/ansible/roles/cisco/136_lab/tasks/comparing_loop.yml for XR3
included: /home/aaa/ansible/roles/cisco/136_lab/tasks/comparing_loop.yml for XR3

TASK [cisco/136_lab : VERIFICATION // XR3 // COLLECTING TELEMETRY AND SEARCHING DATA // FETCHING TELEMETRY DATA] *******************************************************
ok: [XR3]

TASK [cisco/136_lab : VERIFICATION // XR3 // COLLECTING TELEMETRY AND SEARCHING DATA // SAVING TELEMETRY DATA] *********************************************************
changed: [XR3]

TASK [cisco/136_lab : VERIFICATION // XR3 // COLLECTING TELEMETRY AND SEARCHING DATA // MODIFIICATION OF COLLECTED TELEMETRY FOR PYTHON PROCESSING] ********************
changed: [XR3]

TASK [cisco/136_lab : VERIFICATION // XR3 // COLLECTING TELEMETRY AND SEARCHING DATA // IMPORTING COLLECTED TELEMETRY DATA] ********************************************
ok: [XR3]

TASK [cisco/136_lab : VERIFICATION // XR3 // COLLECTING TELEMETRY AND SEARCHING DATA // COMPILING interfaces] **********************************************************
ok: [XR3]

TASK [cisco/136_lab : VERIFICATION // XR3 // COLLECTING TELEMETRY AND SEARCHING DATA // FETCHING TELEMETRY DATA] *******************************************************
ok: [XR3]

TASK [cisco/136_lab : VERIFICATION // XR3 // COLLECTING TELEMETRY AND SEARCHING DATA // SAVING TELEMETRY DATA] *********************************************************
changed: [XR3]

TASK [cisco/136_lab : VERIFICATION // XR3 // COLLECTING TELEMETRY AND SEARCHING DATA // MODIFIICATION OF COLLECTED TELEMETRY FOR PYTHON PROCESSING] ********************
changed: [XR3]

TASK [cisco/136_lab : VERIFICATION // XR3 // COLLECTING TELEMETRY AND SEARCHING DATA // IMPORTING COLLECTED TELEMETRY DATA] ********************************************
ok: [XR3]

TASK [cisco/136_lab : VERIFICATION // XR3 // COLLECTING TELEMETRY AND SEARCHING DATA // COMPILING routing_ospf] ********************************************************
changed: [XR3]

TASK [cisco/136_lab : VERIFICATION // XR3 // COLLECTING TELEMETRY AND SEARCHING DATA // FETCHING TELEMETRY DATA] *******************************************************
ok: [XR3]

TASK [cisco/136_lab : VERIFICATION // XR3 // COLLECTING TELEMETRY AND SEARCHING DATA // SAVING TELEMETRY DATA] *********************************************************
changed: [XR3]

TASK [cisco/136_lab : VERIFICATION // XR3 // COLLECTING TELEMETRY AND SEARCHING DATA // MODIFIICATION OF COLLECTED TELEMETRY FOR PYTHON PROCESSING] ********************
changed: [XR3]

TASK [cisco/136_lab : VERIFICATION // XR3 // COLLECTING TELEMETRY AND SEARCHING DATA // IMPORTING COLLECTED TELEMETRY DATA] ********************************************
ok: [XR3]

TASK [cisco/136_lab : VERIFICATION // XR3 // COLLECTING TELEMETRY AND SEARCHING DATA // COMPILING routing_bgp] *********************************************************
changed: [XR3]

TASK [cisco/136_lab : VERIFICATION // XR3 // COMPILING REPORT] *********************************************************************************************************
changed: [XR3]

TASK [cisco/136_lab : VERIFICATION // XR3 // REPORTING READINESS] ******************************************************************************************************
ok: [XR3] => {
“msg”: “Collection of telemetry data from XR3 is done.”
}

PLAY [nokia] ***********************************************************************************************************************************************************
skipping: no hosts matched

After the execution of the playbook is finished, we can check the results:

These 3 files contain telemetry data, what we have requested: interfaces, OSPF and BGP. To be honest, they are very huge, especially BGP’s telemetry. So I provide you some snippet from interface level telemetry:

Actually all the information you can collect through various show commands are available in JSON format in telemetry. For OSPF and BGP there is MUCH more information, what you can review. Like for OSPF you will have even all LSAs from LSDB in telemetry output.

The last point in verification for Cisco will be to take a look onto test report, where we check:

• State of the interfaces
• Status of the OSPF neighbors
• Status of the BGP peering

As we have explained earlier, we create the test report per configuration profile and then merge together:

Port: GigabitEthernet0/0/0/0
Status: im_state_up

Port: GigabitEthernet0/0/0/1
Status: im_state_admin_down

Port: Loopback0
Status: im_state_up

Port: MgmtEth0/0/CPU0/0
Status: im_state_up

Port: Null0
Status: im_state_up

Port: nV_Loopback0
Status: im_state_up

Port: nV_Loopback1
Status: im_state_up

===========================================
Verification of interfaces is done
===========================================
+——————————————+
| Checking of the BGP status |
+——————————————+

Problem with BGP process on local node (XR3)

===========================================
Verification of BGP neighbors is done
===========================================

+——————————————+
| Checking of the OSPF status |
+——————————————+

Neighbor:
RID: 10.0.0.11
IP: 10.11.33.11
Connected to: GigabitEthernet0/0/0/0
Status: mgmt_nbr_full

I think, you have spotted that BGP has the problems in the output. The reason for that is that 2 GB RAM in my virtual lab is not sufficient for telemetry, and NETCONF/YANG request to collect BGP telemetry data pushes the memory state on the virtual network function XR3 to critical state and BGP process is broken afterwards. Nevertheless, if we increase the amount of available memory, then it’s OK. So, that’s how model-driven telemetry looks like on Cisco IOS XR. Here you can download the ansible playbooks for this article: 136_lab.tar

Lessons learned

As we have learned that collection of the telemetry data via NETCONF is quite resource intensive task, so it shouldn’t be used in production. We advise you to go for gRPC/gNMI to stream telemetry data.

Conclusion

For the sake of brevity I have reduced the output of the provided commands, but the amount of the information contained in the telemetry is just overwhelming. It’s really new oil in the networking, as this telemetry information allows you to build any kind of business logic. Later on we’ll review Kafka, so you will get much more understanding how the whole solution is working, but even now you can assess the amount of information in model-driven telemetry. In the nxt article we’ll take a look on model-driven telemetry in Nokia SR OS. Take care and good bye!

Support us

P.S.

If you have further questions or you need help with your networks, I’m happy to assist you, just send me message (https://karneliuk.com/contact/). Also don’t forget to share the article on your social media, if you like it.

BR,

Anton Karneliuk