Hello my friend,

Logs are one of the most valuable (and therefore important) things, which we have in troubleshooting of our network devices and services. The brief overview of logs gives us understanding of what and when has happened. This paradigm is applicable both for production and lab environment, that’s why we proceed further with configuration of logging at Nokia (Alcatel-Lucent) SR OS and Cisco IOS XR.

Where can we find logs?

I will start the article with the explanation, what I won’t do. I won’t describe the configuration of remote logging (to syslog server) and will focus only on local logging.

Though local logging isn’t academic term, it describes the core idea of that process. The outputs of logging process (actually messages from different protocols and other parts of operation system) are stored locally to our router, regardless speaking about Nokia (Alcatel-Lucent) VSR (or SR 7750, for example) or Cisco IOS XRv. There are four main destinations, where we can send logs locally. Let’s review them on both platforms:

Nokia (ALU) SR OS Cisco IOS XR Explanation
console console Logs are sent to CLI (command line interface), where you can observe them in real-time. In this case console line itself (serial or USB console).
session monitor Logs are sent to CLI, which now relates to ssh/telnet connections.
memory buffered Logs are stored in router’s RAM. They are lost after reboot of Cisco router.
file-id archive Logs are stored in file on flash/HDD. They are resistant to reboot of the device.

For all other Cisco OS (IOS, IOS XE, NX-OS, ASA OS) there are the same types of log’s destinations.

Let’s review how we can configure all these types of local logging, as well as how we can see their content. But first of all let’s see the default configuration. Probably we don’t have to configure anything. How do you think?

Default configuration of logging in Nokia (Alcatel-Lucent) SR OS

The good news is that logging is configured by default both at Cisco IOS XR and Nokia (Alcatel-Lucent) SR OS. Another question that the default logs configuration usually doesn’t satisfy the needs of troubleshooting or learning. For example at Nokia (Alcatel-Lucent) we have the following loggers configured by default:

A:SR3# show log log-id
=============================================================================
Event Logs
=============================================================================
Log Source    Filter Admin Oper       Logged     Dropped Dest       Dest Size
Id            Id     State State                         Type       Id
—————————————————————————–
99 M          none   up    up             36           0 memory           500
100 M         1001   up    up             14          21 memory           500
=============================================================================

There are two logs configured by default. The 99th log shows all events that have taken place in the router since the last reboot (you can figure it out from “dest type” that is memory). The 100th log shows only major ones. At this pane you see the summary information of all configured log. The detailed content you see be adding the log number to the command above:

A:SR3# show log log-id 99
=============================================================================
Event Log 99
=============================================================================
Description : Default System Log
Memory Log contents  [size=500   next event=37  (not wrapped)]36 2016/07/13 14:44:58.99 UTC MAJOR: SYSTEM #2002 Base Configuration Save Succeeds
“Configuration file saved to: cf3:\config.cfg”35 2016/07/13 14:09:18.89 UTC MAJOR: SYSTEM #2002 Base Configuration Save Succeeds
“Configuration file saved to: cf3:\config.cfg”34 2016/07/13 14:08:05.04 UTC MAJOR: SYSTEM #2002 Base Configuration Save Succeeds
“Configuration file saved to: cf3:\config.cfg”33 2016/07/13 14:06:03.65 UTC MAJOR: SYSTEM #2002 Base Configuration Save Succeeds
“Configuration file saved to: cf3:\config.cfg”Press any key to continue (Q to quit)

In Nokia (Alcatel-Lucent) SR OS the latest logs are shown first, whereas in Cisco IOS XR the oldest logs are shown first. You will see that later.

I’ve said previously that 100th log has only information about major events. How have I found it out? In Nokia (Alcatel-Lucent) SR OS (VSR or SR 7750) we have concept of log filters, which gives you possibility to selectively filter only interesting information. You can see that “Filter ID” for 100th log has value 1001. Let’s take a look on that filter:

A:SR3# show log filter-id 1001
=============================================================================
Log Filter
=============================================================================
Filter-id     : 1001     Applied       : yes      Default Action: drop
Description   : Collect events for Serious Errors Log
—————————————————————————–
Log Filter Match Criteria
—————————————————————————–
Entry-id      : 10                      Action        : forward
Application   :                         Operator      : off
Event Number  : 0                       Operator      : off
Severity      : major                   Operator      : greaterThanOrEqual
Subject       :                         Operator      : off
Match Type    : exact string                          :
Message       :
Match Type    : sub string              Operator      : off
Router        :                         Operator      : off
Match Type    : exact string            Operator      : off
Description   : Collect only events of major severity or higher
—————————————————————————–
=============================================================================

Log-filters have very rich set of parameters, which can be used for log filtering. In this filter (which is built-in by default) the parameter “Severity” together with operator “greaterThanOrEqual” and action “forward, what together mean that all the logs with severity major or higher will be forwarded to the destination (memory for log 100). All other logs will be dropped, which you can figure out from field “default action” with value “drop”.

There is one more important thing, before we’ll switch to Cisco. In Nokia (Alcatel-Lucent) SR OS there also four types of logs sources in addition to different destinations. These inputs are specific for SR OS, there is no such analog in Cisco, that’s why I want to spend some time describing it. There four types are as follows:

  • Main” relates to all usual events that happens in Nokia (Alcatel-Lucent) SR operation (i.e. establishing IGP adjacency, booting of card, flapping of ports, etc).
  • Change” means the changes done in the configuration of router.
  • Security” relates to security breaching events.
  • Debug-trace” covers all the low level information (BGP update packets, ARP request/reply, etc)

The details of their configuration we’ll provide further in configuration sections.

Default configuration of logging in Cisco IOS XR

The logging structure in Cisco IOS XR (actually all types of IOS) is much simpler. You have to enter just one command:

RP/0/0/CPU0:XR1#show logging
Fri Jun 17 00:55:50.143 UTC
Syslog logging: enabled (0 messages dropped, 0 flushes, 0 overruns)
Console logging: level warnings, 3093 messages logged
Monitor logging: level debugging, 4643 messages logged
Trap logging: level informational, 0 messages logged
Buffer logging: level debugging, 5496 messages loggedLog Buffer (2097152 bytes):RP/0/0/CPU0:Jun 16 23:27:29.097 : config[65695]: %MGBL-CONFIG-6-DB_COMMIT : Configuration committed by user ‘cisco’. Use ‘show configuration commit changes 1000000147’ to view the changes.
RP/0/0/CPU0:Jun 16 23:30:13.155 : config[65695]: %MGBL-SYS-5-CONFIG_I : Configured from console by cisco
RP/0/0/CPU0:Jun 16 23:34:27.438 : config[65695]: %MGBL-CONFIG-6-DB_COMMIT : Configuration committed by user ‘cisco’. Use ‘show configuration commit changes 1000000148’ to view the changes.

In the beginning there are configuration parameters for logging. Here provided the default levels, which you have at the just new booted router.

For those of you, who are new to Cisco IOS XR, by default debugging isn’t logged to console comparing to Cisco IOS, where it was done.

Let’s speak about differences between Nokia (Alcatel-Lucent) SR OS and Cisco IOS XR:

  • As I’ve already mentioned, the information is provided in such a way that at the beginning you see the oldest logs.
  • Cisco IOS XR has no option for choosing source of the logs, so all the information from the system including debug is collected together
  • Cisco IOS XR has no option for filtering what is logged besides severity. Nokia (Alcatel-Lucent) SR OS provides possibility to filter what is logged based on protocol, text string and so on.
  • Cisco IOS XR has no concept of separation of log streams. The command “show logging” shows you all the logs saved
  • For filtering saved logs Cisco IOS XR provides possibility to filter them only based on regexp. Nokia (Alcatel-Lucent) SR OS has such possibility as well. On the other hand, if you have applied the necessary filter at the beginning, it’s either not needed or provide additional level of filtering granularity.

Probably after that comparison you can think that I like Nokia (Alcatel-Lucent) and hate Cisco. No, I don’t. On the one hand I’m CCIE, so I can’t hate Cisco 🙂 On the other hand I’m Nokia’s employee and have possibility to lab extensively our IP/MPLS portfolio and I find it really awesome. I respect greatly both vendors and by that comparison I just outline the different approaches, which both vendors implement. Nokia (Alcatel-Lucent) SR OS provides greater control on what, how and where to log, whereas Cisco provides very easy configuration of logging and access to all the logged information simultaneously.

Logging to the console

The first thing that I want to configure with is logging to the console. Whereas in Cisco IOS XR it’s configured by default for messages with severity level 5 and higher, in Nokia (Alcatel-Lucent) SR OS such logging isn’t configured at all. So, let’s start with the latter then:

*A:SR3>config>log# info
———————————————-
log-id 1
description “DEBUG_TO_CONSOLE”
time-format local
from debug-trace
to console
no shutdown
exit
———————————————-

This configuration of log stream will send all low level information (and only it) to the console, when we’ll activate debug at our Nokia (Alcatel-Lucent) VSR (or SR 7750). By default all debugging is turned off, just in the same way as it done at Cisco. What is also interesting is that you can switch off this stream without deconfiguring it just by typing:

*A:SR3# configure log log-id 1 shutdown

This is very convenient, because you can have some sort of preconfigured debugging streams with certain filters, which are activated only in emergency cases.

Well, the provided configuration sends only debug information to console. If you want to send also information about other events (“main”, “change” or “security”), you have to configure separate log streams. Let’s say I want to send to the console also information about ports flapping (going up or down). I need the following configuration to accomplish this goal:

*A:SR3>config>log# info
———————————————-
filter 1
default-action drop
entry 1
action forward
match
message eq pattern “Interface”
exit
exit
exit
log-id 2
filter 1
from main
to session
no shutdown
exit
———————————————-

Actually you can add many different entries to the log’s filter. Let’s test, whether this configuration works:

A:SR3# configure router
A:SR3>config>router# interface “toXR1”
A:SR3>config>router>if# shutdown
1 2016/07/13 21:35:31.39 UTC WARNING: SNMP #2004 Base toXR1
“Interface toXR1 is not operational”*A:SR3>config>router>if# no shutdown
2 2016/07/13 21:36:07.60 UTC WARNING: SNMP #2005 Base toXR1
“Interface toXR1 is operational”

Overall logic for Nokia (Alcatel-Lucent) SR OS log stream is: one source, one destination and one filter per stream.

Now it’s time to configure the same logging level in Cisco IOS XR. It’s just one command:

RP/0/0/CPU0:XR1(config)#show conf
logging console debugging
end

After the numerous lines of configuration in Nokia (Alcatel-Lucent) SR OS, such simplification looks incredible. But in SR OS, I’ve configured exactly what I want: all active debug traces and interface flapping. All other logs aren’t displayed in my CLI. In Cisco’s configuration I’ll see everything, what is going on in the router (all information of all severity levels + active debug traces).

RP/0/0/CPU0:XR1(config)#int gig 0/0/0/0
RP/0/0/CPU0:XR1(config-if)#shut
RP/0/0/CPU0:XR1(config-if)#commit
Fri Jun 17 04:08:52.010 UTC
RP/0/0/CPU0:Jun 17 04:08:52.040 : ifmgr[227]: %PKT_INFRA-LINK-5-CHANGED : Interface GigabitEthernet0/0/0/0, changed state to Administratively Down
RP/0/0/CPU0:Jun 17 04:08:52.140 : config[65737]: %MGBL-CONFIG-6-DB_COMMIT : Configuration committed by user ‘cisco’. Use ‘show configuration commit changes 1000000154’ to view the changes.
RP/0/0/CPU0:XR1(config-if)#no shut
RP/0/0/CPU0:XR1(config-if)#commit
Fri Jun 17 04:08:59.400 UTC
RP/0/0/CPU0:Jun 17 04:08:59.410 : ifmgr[227]: %PKT_INFRA-LINK-3-UPDOWN : Interface GigabitEthernet0/0/0/0, changed state to Down
RP/0/0/CPU0:Jun 17 04:08:59.410 : ifmgr[227]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface GigabitEthernet0/0/0/0, changed state to Down
RP/0/0/CPU0:XR1(config-if)#RP/0/0/CPU0:Jun 17 04:08:59.440 : ifmgr[227]: %PKT_INFRA-LINK-3-UPDOWN : Interface GigabitEthernet0/0/0/0, changed state to Up
RP/0/0/CPU0:Jun 17 04:08:59.440 : ifmgr[227]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface GigabitEthernet0/0/0/0, changed state to Up
RP/0/0/CPU0:Jun 17 04:08:59.510 : config[65737]: %MGBL-CONFIG-6-DB_COMMIT : Configuration committed by user ‘cisco’. Use ‘show configuration commit changes 1000000155’ to view the changes.

It worth to say that in Cisco IOS XR it’s possible to active/deactivate logging for certain protocol under this protocol configuration or for certain interface under its configuration. For instance “log adjacency changes disable” under “router ospf xyz” configuration will effectively disable all information about OSPF logged to all destinations (console, buffer and all others).

Logging to ssh/telnet CLI

The previously described logging destination (console) is applicable to you only, if you are physically attached to the console port on the device. In real life such case is usual for installation of new device or lab activities, where you have direct access to the devices. In all other cases, like day-to-day operation or troubleshooting you will be connected to the devices remotely via ssh or telnet, what results in another CLI. This makes us to create separate configuration for logging.

In general you can be connected remotely to the physical console by KVM device, which you access via ssh/telnet. In that case to you connection will be treated by router (both Cisco and Nokia (Alcatel-Lucent)) as serial, so logging is configured according to previous chapter.

Comparing to the configuration of logging to console, you just have to change the destination. Let’s create the rule at Nokia (Alcatel-Lucent) VSR (SR 7750), which will send to telnet console information about OSPF process:

*A:SR3>config>log# info
———————————————-
filter 11
default-action drop
description “OSPF”
entry 1
action forward
match
application eq “ospf”
exit
exit
exit
log-id 3
description “OSPF_TO_TERMINAL”
filter 11
from main
to session
no shutdown
exit
———————————————-

We can’t test it right now as we don’t have OSPF configured. But you can apply it to the configuration from my second article, where I’ve briefly configured OSPF at Nokia (Alcatel-Lucent) VSR (SR 7750) and Cisco IOS XR.

As you may guess, in Cisco IOS XR the configuration of such logging is impossible, so we’ll just configure the output of logs to ssh/telnet session, which is called in Cisco vty lines, with severity level of 6 and above. As you can see above, by default Cisco IOS XR sends even debug to console, what can effectively overload you line and prevent any commands to be entered, if debug is improper activated.

RP/0/0/CPU0:XR1(config)#show conf
logging monitor notifications
end

To see the output of logs in the vty line, when you are connected to Cisco IOS XR router, you should activate such function:

RP/0/0/CPU0:XR1#terminal monitor

Logging to buffer

Obviously it’s impossible to be connected to the device 24/7. On the other hand very often different issues happen, when we are away from console, so we need to have a possibility to read about them (from associated logs) afterwards. Besides logging to syslog server, which isn’t covered in this article, the most common way is to store logs locally in RAM. These logs are stored until the reboot and completely deleted after it. Usually it isn’t problem, because for long-term purposes the logs are stored either at local disc or to syslog server.

At Nokia (Alcatel-Lucent) VSR (SR 7750) the logging to the memory (RAM) is activated by default for main issues stream, so we don’t have to configure it. Let’s configure the same for the debugging.

*A:SR3>config>log# info
———————————————-
log-id 4
description “DEBUG_TO_MEMORY”
from debug-trace
to memory
no shutdown
exit
———————————————-

Now you can to see the logs from main stream using rule configured by default “show log log-id 99” and to see debugs you will use “show log log-id 4”.

Cisco IOS XR has configured by default all logs of all severity levels to be logged to the console. You can change it by:

RP/0/0/CPU0:XR1(config)#show conf
logging buffered notifications
end

Logging to local storage

The last but not least topic regarding local storage of logs is writing them to the file, which is stored at local HDD/flash. You will want to do it, if you don’t have central syslog server, but you want the logs to be saved after reboot.

At Nokia (Alcatel-Lucent) VSR (SR 7750) you can do it as follows:

*A:SR3>config>log# info
———————————————-
file-id 1
location cf3:
exit
log-id 10
from main
to file 1
no shutdown
exit
———————————————-

Now you can check that this log stream is operational:

A:SR3# show log log-id
=============================================================================
Event Logs
=============================================================================
Log Source    Filter Admin Oper       Logged     Dropped Dest       Dest Size
Id            Id     State State                         Type       Id
—————————————————————————–
10 M         none   up    up              1           0 file       1     N/A
=============================================================================

To see the content of this log, you don’t need to do anything special. Just look it through as any other log stream.

A:SR3# show log log-id 10
=============================================================================
Event Log 10
=============================================================================
Description : (Not Specified)
26 2016/07/15 10:58:06.96 UTC MAJOR: SYSTEM #2002 Base Configuration Save Succeeds
“Configuration file saved to: cf3:\config.cfg”

Now let’s take a look at Cisco IOS XR. The configuration of such type of logging is a much longer comparing to what we have already done there:

RP/0/0/CPU0:XR1(config)#show conf
logging archive
device harddisk
severity warnings
file-size 10
frequency daily
!
end

Though it saves logs to the HDD, you should be careful with it. The main idea here that it doesn’t save logs to the HDD in real time, but rather it archives them once a day or week depending on the configuration. If your Cisco IOS XR router was rebooted before such replication of logs to HDD, you will lose them.

Debugging

The last topic for this article is debugging. It goes hand in hand with logging, because its output fully depends on configuration of logging. At this point I assume that all the previous configuration is implemented, including sending of debug to the console. Actually the activation and deactivation of debugging is the same in Nokia (Alcatel-Lucent) VSR (SR 7750) and Cisco IOS XR:

Nokia (Alcatel-Lucent) SR OS Cisco IOS XR
A:SR3# debug router ip icmp RP/0/0/CPU0:XR1#debug icmp ipv4

Let’s test how it works by issuing ping from Cisco IOS XR to Nokia (Alcatel-Lucent) VSR (SR 7750). Cisco side:

RP/0/0/CPU0:XR1#ping 10.0.0.1 count 1
Fri Jul 15 12:25:58.561 UTC
Type escape sequence to abort.
Sending 1, 100-byte ICMP Echos to 10.0.0.1, timeout is 2 seconds:
RP/0/0/CPU0:Jul 15 12:25:58.791 : ipv4_io[266]: IPv4 ICMP: Sending icmp echo request to 10.0.0.1
!
Success rate is 100 percent (1/1), round-trip min/avg/max = 1/1/1 ms
RP/0/0/CPU0:XR1#RP/0/0/CPU0:Jul 15 12:25:58.791 : ipv4_io[266]: ip_icmp_lib_ipv4_receive: Receiving pak(0xb0c08113) tid: 5
RP/0/0/CPU0:Jul 15 12:25:58.791 : ipv4_io[266]: IPV4 ICMP: GigabitEthernet0/0/0/0: Received echo reply from 10.0.0.1
RP/0/0/CPU0:Jul 15 12:25:58.791 : ipv4_io[266]: ip_icmp_lib_ipv4_receive: sending pak(0xb0c08113) to transport: 1, tid: 5

Nokia’s side:

A:SR3#
21 2016/07/15 11:58:58.47 UTC MINOR: DEBUG #2001 Base PIP
“PIP: ICMP
instance 1 (Base), interface index 4 (toXR1),
ICMP  ingressing on toXR1:
10.0.0.2 -> 10.0.0.1
type: Echo (8)  code: No Code (0)

22 2016/07/15 11:58:58.47 UTC MINOR: DEBUG #2001 Base PIP
“PIP: ICMP
instance 1 (Base), interface index 4 (toXR1),
ICMP  egressing on toXR1:
10.0.0.1 -> 10.0.0.2
type: Echo Reply (0)  code: No Code (0)

My personal opinion is that in Nokia (Alcatel-Lucent) SR OS the output is much more readable and informative than in Cisco, though it uses more space for its output. In Nokia (Alcatel-Lucent) SR OS output you see full information regarding source/destination IP addresses in ICMP packets, as well as type/code of ICMP messages. From that point of view Cisco IOS XR show only destination of ping (source for reply) and in general in text form, what type of the ICMP packet is sent or received.

If you want to disable debugs, you can do it either selectively for certain technology or in overall all logging:

Nokia (Alcatel-Lucent) SR OS Cisco IOS XR

A:SR3# debug router ip no icmp
!
A:SR3# no debug

RP/0/0/CPU0:XR1#no debug icmp ipv4
!
RP/0/0/CPU0:XR1#undeb all

Lessons learned

Probably I’ve spend too much time on such topic, because my aim is not to provide command to command parity between Nokia (Alcatel-Lucent) SR OS and Cisco IOS XR. But what I’ve faced during my labs is that absence of default logging to the console of VSR (SR 7750) confuses me a lot, because after Cisco I’ve expectation to see everything what is going on in real time.

Conclusion

In general the configuration of logging in Nokia (Alctel-Lucent) SR OS can be more complicated, because it provides you much more options for selectively logs filtering than Cisco IOS XR. In reality such filtering significantly eases the troubleshooting, because you can in advance predefine all the necessary filters (for BGP, ISIS, LDP and so on) and during troubleshooting you will just per technology check its status. Moreover the structure, where each log stream is stored separately provides possibility to have older logs per used filter, than you could have, if all logs are stored together. On the other hand you have possibility to configure almost the same simple “all-inclusive” log, as we have in Cisco. So it’s totally up to you how you will use this possibility.

Support us





BR,
Anton Karneliuk