Hello my friend,
This is the third article where we use the Mellanox SN 2010 running Cumulus Linux. And today we cover enormously important topic: network security. More precisely, we will speak about the data plane and the control plane protection. Cisco IOS XR and Nokia SR OS accompany us in this journey.
2
3
4
5
retrieval system, or transmitted in any form or by any
means, electronic, mechanical or photocopying, recording,
or otherwise, for commercial purposes without the
prior permission of the author.
Thanks
Special thanks for Avi Alkobi from Mellanox and Pete Crocker and Attilla de Groot from Cumulus Networks for providing me the Mellanox switch and Cumulus license for the tests.
Disclaimer
This blogpost is the continuation of the previous one, where we have brought the Mellanox SN 2010 to the operational with Cumulus Linux 3.7.9 on board. If you want to learn the details about this process, you are welcomed to read that article.
Brief description
Each week you can find the news describing the security breaches. In the modern economy, where the Internet plays already a key role, all the connected businesses (and almost all businesses are connected) are on the risk caused by casual network scanning and brood force attacks. In addition to that, big companies and governments are quite often the attack targets for other companies, governments and criminals. Therefore, the risk of the information security is underestimated or overlooked. The motto “it won’t happen with me” is also very popular.
Just a brief example, a year ago in 2018 I was at a the Pathfinder conference in Berlin, where CEOs from different branches of the industry were delivering pitches. The CEO of the Ergo, which is one of the biggest insurance companies in the Europe, were speaking about cyber security and his guest was one of the practitioners in cyber security filed. When he said, “now we will scan your phones”, it was such a huge action in the hall, when people started switching off the Bluetooth and WiFi on their mobile phones.
Now imagine, that this happening with your IT and network infrastructure right now, and nobody explicitly tells that you. If somebody locks your phone with photos, it is already a personal tragedy. What about the consequences, if somebody steals the data of your customers? Or breaks the entire your network down?
Therefore, the topic today is how to protect the network infrastructure from being hacked.
What are we going to test?
We’ll cover two major tasks, which are slightly different from each other:
- Protection of the hosts connected to the network, which is a data plane security.
- Protection of the network devices itself, which is a control plane security.
Software version
The following software components are used in this lab.
Management host:
- CentOS 7.5.1804 with python 2.7.5
- Ansible 2.8.0
- Docker-CE 18.09
- Containerized NetBox
Enabler and monitoring infrastructure:
- Base Linux image for container: Alpine Linux 3.9
- DHCP: ISC DHCP 4.4.1-r2
- DNS: ISC BIND 9.12.3_p4-r2
- FTP: VSFTPD 3.0.3-r6
- HTTP: NGINX 1.14.2-r1
The Data Centre Fabric:
- Mellanox SN 2010
- Cumulus Linux 3.7.9
- Cisco IOS XR 6.5.1 [guest VNF]
- Nokia SR OS (VSR) 19.5.R1 [guest VNF]
More details about Data Centre Fabric you may find in the previous articles.
Topology
The physical topology of the previous lab works quite good for in this lab as well:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
| |
| /\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ |
| / Docker cloud \ |
| (c)karneliuk.com / +------+ \ |
| \ +---+ DHCP | / |
| Mellanox/Cumulus lab / | .2+------+ \ |
| \ | / |
| \ 172.17.0.0/16 | +------+ \ |
| +-----------+ \+------------+ +---+ DNS | / |
| | | | +----+ .3+------+ \ |
| | Mellanox | 169.254.255.0/24 | Management |.1 | / |
| | SN2010 | fc00:de:1:ffff::/64 | host | | +------+ \ |
| | | | | +---+ FTP | / |
| | mlx+cl | eth0 enp2s0f1 | carrier |\ | .4+------+ \ |
| | | .21 .1 | | \ | / |
| | | :21 :1 | +------+ | / | +------+ \ |
| | +------------------------+ br0 | | \ +---+ HTTP | / |
| | | | +------+ | \ .5+------+ \ |
| | | | | \ / |
| | | swp1 ens2f0 | +------+ +--+ \/\/\/\/\/\/\/ |
| | +------------------------+ br1 +-+ | |
| | | | +------+ | +-------------+ |
| | | | | | |
| | | swp7 ens2f1 | +------+ +---+ +-------+ | |
| | +------------------------+ br2 | +--+ | | |
| | | | +------+--+ | SR1 | | |
| | | | | +--+ | | |
| +-----------+ | +------+ | | +---+---+ | |
| | | br3 +-----+ | | |
| | +------+ | +---+---+ | |
| | +-----+ | | |
| | +------+ | XR1 | | |
| | | br4 +--------+ | | |
| | +------+ +-------+ | |
| | | |
| +-----------------------------+ |
| |
+-----------------------------------------------------------------------+
Details about the connection of the Mellanox SN 2010 to the HP lab server you can find in the previous lab.
The only difference in this lab comparing to the one, we discussed in the previous Segment Routing lab (link), is the direct link between Cisco IOS XR and Nokia SR OS VNFs. We build this link to get a triangle topology, where each network function is connected to each other. This will ease the comparison of the configuration of the different platforms.
You can use any hypervisor of your choice (KVM, VMWare Player/ESXI, etc) to run guest VNFs. For KVM you can use corresponding cheat sheet for VM creation.
The logical topology is updated in the way to reflect the introduce the link:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
| |
| +-----------------eBGP(IPv4/Unicast)-----------+ |
| v v |
| |
| +-----------+ 169.254.0.4/31 +-----------+ |
| sys: 10.0.0.1/32 | | to_XR1 g0/0/0/1 | | Lo0: 10.0.0.3/32 |
| Lo1: 172.16.1.1/32 | SR1 +----------------------------------+ XR1 | Lo1: 172.16.1.3/32 |
| Lo2: 172.16.2.1/32 | ASN:65001 | .4 .5 | ASN:65003 | Lo2: 172.16.2.3/32 |
| +-----+-----+ +-----+-----+ |
| | | |
| ^ .0 | to_SR1 g0/0/0/0 | .3 ^ |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | 169.254.0.0/31 +-----------+ 169.254.0.2/31 | | |
| | | | | | | |
| | +-----------------+ MLX-CL +----------------+ | |
| | .1 | ASN:65002 | .2 | |
| +eBGP(IPv4/Unicast)--> +-----------+ <-eBGP(IPv4/Unicast)+ |
| |
| lo: 10.0.0.2/32 |
| 172.16.1.2/32 |
| 172.16.2.2/32 |
| |
| (c)2019 karneliuk.com |
| |
+----------------------------------------------------------------------------------------------------+
It is possible to run any kind of routing protocol in the topology, but taking into account that we have run the BGP in the vast majority of our lab related to the service provider fabric project and in all the data centre fabric project, we continue to use the BGP. Again, the topology is quite a similar to the previous segment routing lab.
The topology and the configuration files you can find on my GitHub page.
Data plane protection
The first part of the discussion, as we highlighted in the beginning, is the data plane protection. The data plane protection is built in the network to protect the connected systems (laptops, servers, wifi devices, etc). There is a specific class of the network devices, such as firewalls and IPS/IPS systems, which are designed to protect the network. On the other hand, especially when we talk about the data centres, the amount of the traffic is typically so tremendous, that you would need to buy a lot of firewalls to control and inspect all the traffic. Some systems, e.g. payment systems complied with PCI DSS (link), require such protection, so you can’t avoid the investments in the firewalls. But in the vast majority of cases, where the data centre is a commercial, and its network infrastructure is shared across multiple customers, the network infrastructure itself (e.g. leafs (link)) can play also guarding role by allowing the traffic only from/to certain IP addresses or even ports.
Before we start, we do a basic connectivity check test to verify that all the Loopback IPs can reach each other (shown the example for SR1 – MLX-CL check, the same is done for XR1 but omitted for brevity):
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
PING 10.0.0.1 (10.0.0.1) from 10.0.0.2 : 56(84) bytes of data.
64 bytes from 10.0.0.1: icmp_seq=1 ttl=64 time=1.58 ms
--- 10.0.0.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.588/1.588/1.588/0.000 ms
cumulus@MLX-CL:mgmt-vrf:~$ ping 10.0.0.1 -I 172.16.1.2 -c 1
PING 10.0.0.1 (10.0.0.1) from 172.16.1.2 : 56(84) bytes of data.
64 bytes from 10.0.0.1: icmp_seq=1 ttl=64 time=2.36 ms
--- 10.0.0.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 2.364/2.364/2.364/0.000 ms
cumulus@MLX-CL:mgmt-vrf:~$ ping 10.0.0.1 -I 172.16.2.2 -c 1
PING 10.0.0.1 (10.0.0.1) from 172.16.2.2 : 56(84) bytes of data.
64 bytes from 10.0.0.1: icmp_seq=1 ttl=64 time=1.79 ms
--- 10.0.0.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.799/1.799/1.799/0.000 ms
cumulus@MLX-CL:mgmt-vrf:~$ ping 172.16.1.1 -I 10.0.0.2 -c 1
PING 172.16.1.1 (172.16.1.1) from 10.0.0.2 : 56(84) bytes of data.
64 bytes from 172.16.1.1: icmp_seq=1 ttl=64 time=2.18 ms
--- 172.16.1.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 2.188/2.188/2.188/0.000 ms
cumulus@MLX-CL:mgmt-vrf:~$ ping 172.16.1.1 -I 172.16.1.2 -c 1
PING 172.16.1.1 (172.16.1.1) from 172.16.1.2 : 56(84) bytes of data.
64 bytes from 172.16.1.1: icmp_seq=1 ttl=64 time=2.04 ms
--- 172.16.1.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 2.049/2.049/2.049/0.000 ms
cumulus@MLX-CL:mgmt-vrf:~$ ping 172.16.1.1 -I 172.16.2.2 -c 1
PING 172.16.1.1 (172.16.1.1) from 172.16.2.2 : 56(84) bytes of data.
64 bytes from 172.16.1.1: icmp_seq=1 ttl=64 time=1.39 ms
--- 172.16.1.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.397/1.397/1.397/0.000 ms
cumulus@MLX-CL:mgmt-vrf:~$ ping 172.16.2.1 -I 10.0.0.2 -c 1
PING 172.16.2.1 (172.16.2.1) from 10.0.0.2 : 56(84) bytes of data.
64 bytes from 172.16.2.1: icmp_seq=1 ttl=64 time=1.92 ms
--- 172.16.2.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.920/1.920/1.920/0.000 ms
cumulus@MLX-CL:mgmt-vrf:~$ ping 172.16.2.1 -I 172.16.1.2 -c 1
PING 172.16.2.1 (172.16.2.1) from 172.16.1.2 : 56(84) bytes of data.
64 bytes from 172.16.2.1: icmp_seq=1 ttl=64 time=1.21 ms
--- 172.16.2.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.213/1.213/1.213/0.000 ms
cumulus@MLX-CL:mgmt-vrf:~$ ping 172.16.2.1 -I 172.16.2.2 -c 1
PING 172.16.2.1 (172.16.2.1) from 172.16.2.2 : 56(84) bytes of data.
64 bytes from 172.16.2.1: icmp_seq=1 ttl=64 time=1.59 ms
--- 172.16.2.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.596/1.596/1.596/0.000 ms
As you can see, everyone can reach everyone. We want to reduce the amount of the communications. The scenario, which we are going to deploy in this lab, is the following:
- For the hosts sharing the 172.16.1.0/24 prefix we allow all IP traffic to/from this subnet. Meaning that only 172.16.1.1/32 or 172.16.1.2/32 could reach 172.16.1.3/32. If the communication is started from any other IP, e.g. 172.16.2.1/32, it should not be successful.
- For the hosts sharing the 172.16.2.0/24 prefix, we allow only HTTP traffic to this hosts from any subnet.
In a nutshell, we will cover two types of the protection: pure IP-based and port-based.
The main mechanisms, which is used to do such a protection is an access-list (ACL). Let’s take a look, how this protection is implemented in the network operation systems we have chosen: Nokia SR OS, Cisco IOS XR and Cumulus Linux running on Mellanox SN 2010 switch.
The latter is important for two reasons: in Cumulus VX the ACLs doesn’t work, and in Cumulus Linux on Broadcom switches they work a bit in a different manner.
#1.1. Nokia SR OS data plane protection
Let’s start the implementation of the ACLs from the network function running Nokia SR OS. From the name of the security tool (access-list), you can figure out, that the tasks are performed sequentially. Given the scenario described above, our list should contain the following sequience of the actions:
- Allow any traffic from 172.16.1.0/24 to 172.16.1.0/24 IPv4 addresses.
- Deny any traffic to 172.16.1.0/24 IPv4 addresses.
- Allow any traffic to 172.16.2.0/24 IPv4 addresses on the TCP port 80.
- Deny any traffic to 172.16.2.0/24 IPv4 addresses.
- Allow any other traffic.
Typically, you have a single access list per interface, therefore all the rules should be processed in such a way. Let’s translate the security rules above into Nokia SR OS syntax:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
A:admin@SR1# compare
configure {
+ filter {
+ ip-filter "ACL_IPV4_DP_PROTECTION" {
+ default-action accept
+ filter-id 10
+ entry 10 {
+ description "ALLOW TRAFFIC BETWEEN 172.16.1.0/24 HOSTS"
+ match {
+ src-ip {
+ address 172.16.1.0
+ mask 255.255.255.0
+ }
+ dst-ip {
+ address 172.16.1.0
+ mask 255.255.255.0
+ }
+ }
+ action {
+ accept
+ }
+ }
+ entry 20 {
+ description "DENY ANY TRAFFIC TO 172.16.1.0/24 HOSTS"
+ match {
+ dst-ip {
+ address 172.16.1.0
+ mask 255.255.255.0
+ }
+ }
+ action {
+ drop
+ }
+ }
+ entry 30 {
+ description "ALLOW HTTP TRAFFIC TO 172.16.2.0/24 SUBNET"
+ match {
+ protocol tcp
+ dst-ip {
+ address 172.16.2.0
+ mask 255.255.255.0
+ }
+ dst-port {
+ eq 80
+ }
+ }
+ action {
+ accept
+ }
+ }
+ entry 40 {
+ description "BLOCK TRAFFIC TO 172.16.2.0/24 SUBNET"
+ match {
+ dst-ip {
+ address 172.16.2.0
+ mask 255.255.255.0
+ }
+ }
+ action {
+ drop
+ }
+ }
+ }
+ }
}
*(gl)[]
A:admin@SR1# commit
All the entries contain the description, so you can easily read which entry corresponds to which security rule. The fifth rule, which should allow all the rest traffic is realized using default-action knob, so it doesn’t have any specific entry number.
One the access list is created, we apply it to the interfaces. In this lab, we would apply it to the interfaces towards the rest two routers:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
*(gl)[configure router "Base"]
A:admin@SR1# compare
interface "to_MLX-CL" {
+ ingress {
+ filter {
+ ip "ACL_IPV4_DP_PROTECTION"
+ }
+ }
}
interface "to_XR1" {
+ ingress {
+ filter {
+ ip "ACL_IPV4_DP_PROTECTION"
+ }
+ }
}
*(gl)[configure router "Base"]
A:admin@SR1# commit
So far, all the configuration was accepted and we can briefly verify it:
2
3
4
5
6
Egress Filter : none Ingress Filter : 10
Egr IPv6 Flt : none Ingr IPv6 Flt : none
Ingress stats : Disabled IPv6 DAD : Enabled
Ingress FP QGrp : (none) Egress Port QGrp : (none)
Ing FP QGrp Inst : (none) Egr Port QGrp Inst: (none)
And the content of the access-list itself:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
===============================================================================
IP Filter
===============================================================================
Filter Id : 10 Applied : Yes
Scope : Template Def. Action : Forward
Type : Normal
System filter : Unchained
Radius Ins Pt : n/a
CrCtl. Ins Pt : n/a
RadSh. Ins Pt : n/a
PccRl. Ins Pt : n/a
Entries : 4
Description : (Not Specified)
Filter Name : ACL_IPV4_DP_PROTECTION
-------------------------------------------------------------------------------
Filter Match Criteria : IP
-------------------------------------------------------------------------------
Entry : 10
Description : ALLOW TRAFFIC BETWEEN 172.16.1.0/24 HOSTS
Log Id : n/a
Src. IP : 172.16.1.0/24
Src. Port : n/a
Dest. IP : 172.16.1.0/24
Dest. Port : n/a
Protocol : Undefined Dscp : Undefined
ICMP Type : Undefined ICMP Code : Undefined
Fragment : Off Src Route Opt : Off
Sampling : Off Int. Sampling : On
IP-Option : 0/0 Multiple Option: Off
Tcp-flag : (Not Specified)
Option-pres : Off
Egress PBR : Disabled
Primary Action : Forward
Ing. Matches : 0 pkts
Egr. Matches : 0 pkts
! FURTHER OUTPUT IS OMITTED FOR BREVITY
The last thing we need to do in this part for Nokia is to verify that it works properly. As you remember, we speak about filtering traffic before the customers, so we shutdown temporary the link between XR1 and MLX-CL so that traffic between these nodes is transported over SR1:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
Sat Sep 28 17:49:23.304 UTC
Codes: C - connected, S - static, R - RIP, B - BGP, (>) - Diversion path
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP
i - ISIS, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, su - IS-IS summary null, * - candidate default
U - per-user static route, o - ODR, L - local, G - DAGR, l - LISP
A - access/subscriber, a - Application route
M - mobile route, r - RPL, t - Traffic Engineering, (!) - FRR Backup path
Gateway of last resort is not set
B 10.0.0.1/32 [20/0] via 169.254.0.4, 00:49:44
B 10.0.0.2/32 [20/0] via 169.254.0.4, 00:00:24
L 10.0.0.3/32 is directly connected, 00:51:42, Loopback0
C 169.254.0.2/31 is directly connected, 00:51:41, GigabitEthernet0/0/0/0
L 169.254.0.3/32 is directly connected, 00:51:41, GigabitEthernet0/0/0/0
C 169.254.0.4/31 is directly connected, 00:51:41, GigabitEthernet0/0/0/1
L 169.254.0.5/32 is directly connected, 00:51:41, GigabitEthernet0/0/0/1
B 172.16.1.1/32 [20/0] via 169.254.0.4, 00:49:44
B 172.16.1.2/32 [20/0] via 169.254.0.4, 00:00:24
L 172.16.1.3/32 is directly connected, 00:51:42, Loopback1
B 172.16.2.1/32 [20/0] via 169.254.0.4, 00:49:44
B 172.16.2.2/32 [20/0] via 169.254.0.4, 00:00:24
L 172.16.2.3/32 is directly connected, 00:51:42, Loopback2
After that, we do the verification. Let’s start with the verification of the first two rules related to hosts in the 172.16.1.0/24 prefix:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
Sat Sep 28 17:50:07.591 UTC
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.1.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/9 ms
RP/0/0/CPU0:XR1#ping 172.16.1.2 so 172.16.2.3
Sat Sep 28 17:50:09.851 UTC
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.1.2, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
RP/0/0/CPU0:XR1#ping 172.16.1.2 so 10.0.0.3
Sat Sep 28 17:52:23.222 UTC
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.1.2, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
Use command “show filter ip” shared above to check that counters are increasing.
Brilliant! The first two rules work as expected, so that hosts sharing 172.16.1.0/24 can communicate to each other, but no other host.
The test of the third rule is a bit tricky, as we don’t have running HTTP server. Nevertheless, that is not the problem as we don’t test the HTTP service, but rather access list. So we do the following test from the Cisco IOS XR VNF:
2
3
4
5
6
7
8
9
10
11
12
Trying 172.16.2.2...
Use specified source interface(Loopback1).
Use 172.16.1.3 as local address.
telnet: Unable to connect to remote host: Connection timed out
RP/0/0/CPU0:Sep 28 17:57:29.401 UTC: telnet[65717]: %IP-telnet-6-CONNECT : Couldn't connect to remote host 172.16.2.2(Connection timed out).
RP/0/0/CPU0:XR1#ping 172.16.2.2 so lo1
Sat Sep 28 17:57:41.510 UTC
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.2.2, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
As HTTP is just another TCP traffic, we try to generate the TCP on the port 80 using telnet built-in tool in the Cisco IOS XR to the customer port. Let’s take a look on the counters now, to check that our ACL is correct:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
===============================================================================
IP Filter
===============================================================================
Filter Id : 10 Applied : Yes
Scope : Template Def. Action : Forward
Type : Normal
System filter : Unchained
Radius Ins Pt : n/a
CrCtl. Ins Pt : n/a
RadSh. Ins Pt : n/a
PccRl. Ins Pt : n/a
Entries : 4
Description : (Not Specified)
Filter Name : ACL_IPV4_DP_PROTECTION
-------------------------------------------------------------------------------
Filter Match Criteria : IP
-------------------------------------------------------------------------------
Entry : 10
Ing. Matches : 10 pkts (1180 bytes)
Egr. Matches : 0 pkts
Entry : 20
Ing. Matches : 14 pkts (1412 bytes)
Egr. Matches : 0 pkts
Entry : 30
Ing. Matches : 4 pkts (264 bytes)
Egr. Matches : 0 pkts
Entry : 40
Ing. Matches : 5 pkts (590 bytes)
Egr. Matches : 0 pkts
===============================================================================
You see the counters in each rules, what confirms ACL is correct. The last rule, allow all the rest traffic, is confirmed by simple test:
2
3
4
5
6
Sat Sep 28 18:03:30.806 UTC
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
Don’t forget to bring the link between XR1 and MLX-CL up before going to the next point.
#1.2. Cisco IOS XR data plane protection
As the details steps are alredy provided above, we only need to implement that logic into the Cisco IOS XR syntax:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
Sat Sep 28 18:10:29.628 UTC
Building configuration...
!! IOS XR Configuration 6.5.1.34I
ipv4 access-list ACL_IPV4_DP_PROTECTION
9 remark "ALLOW TRAFFIC BETWEEN 172.16.1.0/24 HOSTS"
10 permit ipv4 172.16.1.0/24 172.16.1.0/24
19 remark "DENY ANY TRAFFIC TO 172.16.1.0/24 HOSTS"
20 deny ipv4 any 172.16.1.0/24
29 remark "ALLOW HTTP TRAFFIC TO 172.16.2.0/24 SUBNET"
30 permit tcp any 172.16.2.0/24 eq www
39 remark "BLOCK TRAFFIC TO 172.16.2.0/24 SUBNET"
40 deny ipv4 any 172.16.2.0/24
200 permit ipv4 any any
!
end
RP/0/0/CPU0:XR1(config)#commit
The Cisco IOS XR syntax is a bit more compact comparing to the Nokia SR OS, but it requires separate entries for the comments. Once the ACL is ready, we need to apply it to the interfaces:
2
3
4
5
6
7
8
9
10
11
12
13
Sat Sep 28 18:13:44.804 UTC
Building configuration...
!! IOS XR Configuration 6.5.1.34I
interface GigabitEthernet0/0/0/0
ipv4 access-group ACL_IPV4_DP_PROTECTION ingress
!
interface GigabitEthernet0/0/0/1
ipv4 access-group ACL_IPV4_DP_PROTECTION ingress
!
end
RP/0/0/CPU0:XR1(config)#commit
The verification of the ACL attached to the interfaces, as well as the content of the ACL, you can do in Cisco IOS XR in the following way:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
Sat Sep 28 18:15:12.328 UTC
GigabitEthernet0/0/0/0 is Up, ipv4 protocol is Up
Vrf is default (vrfid 0x60000000)
Internet address is 169.254.0.3/31
MTU is 1514 (1500 is available to IP)
Helper address is not set
Multicast reserved groups joined: 224.0.0.2 224.0.0.1
Directed broadcast forwarding is disabled
Outgoing access list is not set
Inbound common access list is not set, access list is ACL_IPV4_DP_PROTECTION
Proxy ARP is disabled
ICMP redirects are never sent
ICMP unreachables are always sent
ICMP mask replies are never sent
Table Id is 0xe0000000
RP/0/0/CPU0:XR1#show access-lists ipv4
Sat Sep 28 18:16:34.443 UTC
ipv4 access-list ACL_IPV4_DP_PROTECTION
9 remark "ALLOW TRAFFIC BETWEEN 172.16.1.0/24 HOSTS"
10 permit ipv4 172.16.1.0 0.0.0.255 172.16.1.0 0.0.0.255
19 remark "DENY ANY TRAFFIC TO 172.16.1.0/24 HOSTS"
20 deny ipv4 any 172.16.1.0 0.0.0.255
29 remark "ALLOW HTTP TRAFFIC TO 172.16.2.0/24 SUBNET"
30 permit tcp any 172.16.2.0 0.0.0.255 eq www
39 remark "BLOCK TRAFFIC TO 172.16.2.0/24 SUBNET"
40 deny ipv4 any 172.16.2.0 0.0.0.255
200 permit ipv4 any any (119 matches)
Before we started any tests, you might already see the hits to the default rule, which are coming from BGP, ARP and other protocols silently running on the interfaces.
Following the approach shared in the Linux part, we will bring the interface between MLX-CL and SR1 to force the traffic cross the XR1:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR,
> - selected route, * - FIB route
B>* 10.0.0.1/32 [20/0] via 169.254.0.3, swp7, 00:00:06
C>* 10.0.0.2/32 is directly connected, lo, 01:24:28
B>* 10.0.0.3/32 [20/0] via 169.254.0.3, swp7, 00:16:14
C>* 169.254.0.2/31 is directly connected, swp7, 00:16:23
B>* 172.16.1.1/32 [20/0] via 169.254.0.3, swp7, 00:00:06
C>* 172.16.1.2/32 is directly connected, lo, 01:24:28
B>* 172.16.1.3/32 [20/0] via 169.254.0.3, swp7, 00:16:14
B>* 172.16.2.1/32 [20/0] via 169.254.0.3, swp7, 00:00:06
C>* 172.16.2.2/32 is directly connected, lo, 01:24:28
B>* 172.16.2.3/32 [20/0] via 169.254.0.3, swp7, 00:16:14
Let’s start the test of the first two rules related to the hosts sharing the prefix 172.16.1.0/24:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
PING 172.16.1.1 (172.16.1.1) from 172.16.1.2 : 56(84) bytes of data.
64 bytes from 172.16.1.1: icmp_seq=1 ttl=63 time=3.69 ms
--- 172.16.1.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 3.695/3.695/3.695/0.000 ms
cumulus@MLX-CL:mgmt-vrf:~$ ping 172.16.1.1 -I 172.16.2.2 -c 1
PING 172.16.1.1 (172.16.1.1) from 172.16.2.2 : 56(84) bytes of data.
From 169.254.0.3 icmp_seq=1 Packet filtered
--- 172.16.1.1 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms
cumulus@MLX-CL:mgmt-vrf:~$ ping 172.16.1.1 -I 10.0.0.2 -c 1
PING 172.16.1.1 (172.16.1.1) from 10.0.0.2 : 56(84) bytes of data.
From 169.254.0.3 icmp_seq=1 Packet filtered
--- 172.16.1.1 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms
And brilliant again, as only the traffic from 172.16.1.2/32 can reach 172.16.1.1/32.
The second set of the tests is related to the HTTP traffic to the hosts in 172.16.2.0/24 range. As Cumulus Linux is Linux, so we can use a great Linux curl tool for that:
2
3
4
5
cumulus@MLX-CL:mgmt-vrf:~$ sudo ip vrf exec default curl -X GET 172.16.2.1:81 --interface 172.16.2.2
curl: (7) Failed to connect to 172.16.2.1 port 81: No route to host
Creds for this command to Attilla from Cumulus Networks.
In the first case we get a timeout, as there is no service running on the SR1, whereas in the second case we got a response from the XR1, which blocks our session immediately.
Let’s verify the counters on Cisco IOS XR router:
2
3
4
5
6
7
8
9
10
11
12
Sat Sep 28 18:47:19.536 UTC
ipv4 access-list ACL_IPV4_DP_PROTECTION
9 remark "ALLOW TRAFFIC BETWEEN 172.16.1.0/24 HOSTS"
10 permit ipv4 172.16.1.0 0.0.0.255 172.16.1.0 0.0.0.255 (6 matches)
19 remark "DENY ANY TRAFFIC TO 172.16.1.0/24 HOSTS"
20 deny ipv4 any 172.16.1.0 0.0.0.255 (2 matches)
29 remark "ALLOW HTTP TRAFFIC TO 172.16.2.0/24 SUBNET"
30 permit tcp any 172.16.2.0 0.0.0.255 eq www (4 matches)
39 remark "BLOCK TRAFFIC TO 172.16.2.0/24 SUBNET"
40 deny ipv4 any 172.16.2.0 0.0.0.255 (7 matches)
200 permit ipv4 any any (1493 matches)
Don’t forget to bring the link between the SR1 an MLX-CL up, before going further.
#1.3. Cumulus/Mellanox data plane protection
The final step in the data plane protection journey will be in Mellanox/Cumulus network function. Before this point the information was more or less familiar for a traditional network chaps, like me, but from now will be something new. Cumulus Linux relies on the Linux firewall called iptables, what makes its operation not an easy one, when you get to know it for the first time.
The official documentation for Cumulus Linux you can find on their site.
I strongly recommend you read the information in the link above, if you are new to the Linux FW. To sum up, in the iptables command you need to define:
- CHAIN, what defines where the access rule is applied. For traffic passing the switch it is FORWARD, whereas traffic to the switch is INPUT.
- MATCH, what identifies the traffic, which the rule should be applied to.
- JUMP, what is the action (I have no ideas, why it is called jump) that should be applied to the traffic.
Using this terminology, we create rules for our scenario in a separate file on our Mellanox SN 2010 running Cumulus Linux:
2
3
4
5
6
[iptables]
-A FORWARD --in-interface swp+ -s 172.16.1.0/24 -d 172.16.1.0/24 -j ACCEPT
-A FORWARD --in-interface swp+ -d 172.16.1.0/24 -j DROP
-A FORWARD --in-interface swp+ -d 172.16.2.0/24 -p tcp --dport 80 -j ACCEPT
-A FORWARD --in-interface swp+ -d 172.16.2.0/24 -j DROP
Let’s stay here for a moment to understand the key parts of the rules:
- -A means append the rule to the default trail (by default, there are no rules in the FORWARD chain)
- -d corresponds to the destination IP addresses
- -s corresponds to the source IP addresses
- –in-interface corresponds to incoming interface. Knob swp+ applies the rule to all interfaces starting from swp (i.e. all data plane interfaces).
- -p defines the protocol
- –dport specifies the particular TCP or UDP port.
All the details you can find in the iptables doc.
Once the rules are created, we need to install them (meaning, program to the ASIC). This done in Cumulus Linux in such a manner:
2
3
4
5
6
7
8
9
Reading rule file /etc/cumulus/acl/policy.d/00control_plane.rules ...
Processing rules in file /etc/cumulus/acl/policy.d/00control_plane.rules ...
Reading rule file /etc/cumulus/acl/policy.d/50_data_plane_protection.rules ...
Processing rules in file /etc/cumulus/acl/policy.d/50_data_plane_protection.rules ...
Reading rule file /etc/cumulus/acl/policy.d/99control_plane_catch_all.rules ...
Processing rules in file /etc/cumulus/acl/policy.d/99control_plane_catch_all.rules ...
Installing acl policy
done.
Among the default rules, you can see that the rules from our new file are installed as well. Let’s verify it is really so:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
-------------------------------
Listing rules of type iptables:
-------------------------------
TABLE filter :
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 DROP all -- swp+ any 240.0.0.0/5 anywhere
0 0 DROP all -- swp+ any loopback/8 anywhere
0 0 DROP all -- swp+ any base-address.mcast.net/4 anywhere
0 0 DROP all -- swp+ any 255.255.255.255 anywhere
! OUTPUT IS OMITTED FOR BREVITY
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- swp+ any 172.16.1.0/24 172.16.1.0/24
0 0 DROP all -- swp+ any anywhere 172.16.1.0/24
0 0 ACCEPT tcp -- swp+ any anywhere 172.16.2.0/24 tcp dpt:http
0 0 DROP all -- swp+ any anywhere 172.16.2.0/24
! FURTHER OUTPUT IS OMITTED
This command provides the output of the ACL for all the chains, so you need to scroll a bit further towards FORWARD.
As said before, there are no default rules for the FORWARD chain. Therefore, you can see only the rules we have created.
The final round of the verifications for the data plane protection we start with the bringing down the connectivity between SR1 and XR1:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
===============================================================================
Route Table (Router: Base)
===============================================================================
Dest Prefix[Flags] Type Proto Age Pref
Next Hop[Interface Name] Metric
-------------------------------------------------------------------------------
10.0.0.1/32 Local Local 02h40m54s 0
system 0
10.0.0.2/32 Remote BGP 00h44m29s 170
169.254.0.1 0
10.0.0.3/32 Remote BGP 00h00m47s 170
169.254.0.1 0
169.254.0.0/31 Local Local 02h40m40s 0
to_MLX-CL 0
169.254.0.4/31 Local Local 02h40m40s 0
to_XR1 0
172.16.1.1/32 Local Local 02h40m54s 0
Lo1 0
172.16.1.2/32 Remote BGP 00h44m29s 170
169.254.0.1 0
172.16.1.3/32 Remote BGP 00h00m47s 170
169.254.0.1 0
172.16.2.1/32 Local Local 02h40m54s 0
Lo2 0
172.16.2.2/32 Remote BGP 00h44m29s 170
169.254.0.1 0
172.16.2.3/32 Remote BGP 00h00m48s 170
169.254.0.1 0
-------------------------------------------------------------------------------
No. of Routes: 11
Flags: n = Number of times nexthop is repeated
B = BGP backup route available
L = LFA nexthop available
S = Sticky ECMP requested
===============================================================================
The first round of the verifications comes for the first two rules:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
A:admin@SR1# //ping 172.16.1.3 source 172.16.1.1 count 1
INFO: CLI #2051: Switching to the classic CLI engine
A:SR1# /ping 172.16.1.3 source 172.16.1.1 count 1
PING 172.16.1.3 56 data bytes
64 bytes from 172.16.1.3: icmp_seq=1 ttl=254 time=2.11ms.
---- 172.16.1.3 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 2.11ms, avg = 2.11ms, max = 2.11ms, stddev = 0.000ms
INFO: CLI #2052: Switching to the MD-CLI engine
[]
A:admin@SR1# //ping 172.16.1.3 source 172.16.2.1 count 1
INFO: CLI #2051: Switching to the classic CLI engine
A:SR1# /ping 172.16.1.3 source 172.16.2.1 count 1
PING 172.16.1.3 56 data bytes
Request timed out. icmp_seq=1.
---- 172.16.1.3 PING Statistics ----
1 packet transmitted, 0 packets received, 100% packet loss
INFO: CLI #2052: Switching to the MD-CLI engine
[]
A:admin@SR1# //ping 172.16.1.3 source 10.0.0.1 count 1
INFO: CLI #2051: Switching to the classic CLI engine
A:SR1# /ping 172.16.1.3 source 10.0.0.1 count 1
PING 172.16.1.3 56 data bytes
Request timed out. icmp_seq=1.
---- 172.16.1.3 PING Statistics ----
1 packet transmitted, 0 packets received, 100% packet loss
INFO: CLI #2052: Switching to the MD-CLI engine
And this is brilliant for the third time! Let’s test the second set of the rules (following the same approach using telnet tool to test TCP/80 rule):
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
A:admin@SR1# //telnet 172.16.2.3 80
INFO: CLI #2051: Switching to the classic CLI engine
A:SR1# /telnet 172.16.2.3 80
Trying 172.16.2.3 ...
telnet: Unable to connect to remote host: Socket is not connected
INFO: CLI #2052: Switching to the MD-CLI engine
[]
A:admin@SR1# //telnet 172.16.2.3 81
INFO: CLI #2051: Switching to the classic CLI engine
A:SR1# /telnet 172.16.2.3 81
Trying 172.16.2.3 ...
telnet: Unable to connect to remote host: Connection timed out
INFO: CLI #2052: Switching to the MD-CLI engine
Iptables works a bit in a different mode comparing to Cisco, that is why open session is immediately closed and blocked is just timed out.
Connectivity works as planned. Now we just test that the rest of the traffic is allowed:
2
3
4
5
6
7
8
9
10
11
A:admin@SR1# //ping 10.0.0.3 source 10.0.0.1 count 1
INFO: CLI #2051: Switching to the classic CLI engine
A:SR1# /ping 10.0.0.3 source 10.0.0.1 count 1
PING 10.0.0.3 56 data bytes
64 bytes from 10.0.0.3: icmp_seq=1 ttl=254 time=3.42ms.
---- 10.0.0.3 PING Statistics ----
1 packet transmitted, 1 packet received, 0.00% packet loss
round-trip min = 3.42ms, avg = 3.42ms, max = 3.42ms, stddev = 0.000ms
INFO: CLI #2052: Switching to the MD-CLI engine
Final counters verification at Mellanox/Cumulus:
2
3
4
5
6
7
8
9
! OUTPUT IS OMITTED
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
8 816 ACCEPT all -- swp+ any 172.16.1.0/24 172.16.1.0/24
2 204 DROP all -- swp+ any anywhere 172.16.1.0/24
1 78 ACCEPT tcp -- swp+ any anywhere 172.16.2.0/24 tcp dpt:http
2 156 DROP all -- swp+ any anywhere 172.16.2.0/24
! FURTEHR OUTPUT IS OMITTED
Bring the suspended interface up and go further to the control-plane policing.
Control plane protection
If the data plane protection is related to the connected customers, then control plane protection is related to the traffic destined to the router itself. This is mainly the traffic related to the network management (such as SSH, SNMP, etc) or so-called control-plane traffic (such as BGP, OSPF, etc).
In our scenario, we will do the following:
- Deny any SSH traffic on the data plane interfaces
- Allow BGP only from link subnets
- Block all the rest BGP traffic
- Allow all the rest traffic
#2.1. Nokia SR OS control plane protection
The configuration syntax for Nokia SR OS is almost the same as it was for the data plane with the exception, that the configuration is done in another context. This content is called CPM filter, where CP stands for control plane. You configure this filter and it is directly applied to the control plane of the router:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
A:admin@SR1# compare
configure {
system {
security {
+ cpm-filter {
+ default-action accept
+ ip-filter {
+ admin-state enable
+ entry 10 {
+ description "BLOCK SSH TRAFFIC"
+ match {
+ protocol tcp
+ dst-port {
+ eq 22
+ }
+ }
+ action {
+ drop
+ }
+ }
+ entry 20 {
+ description "ALLOW LINK BGP"
+ match {
+ protocol tcp
+ src-ip {
+ address 169.254.0.0/24
+ }
+ dst-ip {
+ address 169.254.0.0/24
+ }
+ dst-port {
+ eq 179
+ }
+ }
+ action {
+ accept
+ }
+ }
+ entry 30 {
+ description "BLOCK REST BGP"
+ match {
+ protocol tcp
+ dst-port {
+ eq 179
+ }
+ }
+ action {
+ drop
+ }
+ }
+ }
+ }
}
}
}
*(gl)[]
A:admin@SR1# commit
Once applied, let’s check, if we can SSH from Cisco IOS XR towards the Nokia SR OS router:
2
3
RP/0/0/CPU0:Sep 28 20:18:50.990 UTC: ssh_xr[65717]: %SECURITY-SSHD-3-ERR_ERRNO : Failed to connect - Connection timed out
The protection is working. We can take a look on the counters of the CPM filer to verify of works properly:
2
3
4
5
6
7
8
9
10
11
12
13
14
A:admin@SR1# show system security cpm-filter ip-filter
===============================================================================
CPM IP Filter (applied)
===============================================================================
Id Dropped Forwarded Description
-------------------------------------------------------------------------------
10 4 0 BLOCK SSH TRAFFIC
20 0 394 ALLOW LINK BGP
30 0 0 BLOCK REST BGP
-------------------------------------------------------------------------------
Num CPM IP filter entries: 3
===============================================================================
What is important to highlight, the CPM filter is working for the data plane interfaces, but not for the management interfaces located on the route processor. You have possibility to SSH to router if you have physical OOB network:
2
3
4
5
6
7
8
admin@169.254.255.31's password:
SR OS Software
Copyright (c) Nokia 2019. All Rights Reserved.
Trademarks
#2.2. Cisco IOS XR control plane protection
Let’s secure the control plane on the Cisco IOS XR device. To implement our scenario we do the following configuration:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
Sat Sep 28 20:38:08.921 UTC
Building configuration...
!! IOS XR Configuration 6.5.1.34I
control-plane
management-plane
out-of-band
vrf mgmt
interface MgmtEth0/0/CPU0/0
allow all
!
!
!
!
ipv4 access-list ACL_IPV4_DP_PROTECTION
49 remark "ALLOW LINK BGP"
50 permit tcp 169.254.0.0/24 169.254.0.0/24 eq bgp
59 remark "BLOCK ANOTHER BGP"
60 deny tcp any any eq bgp
!
ssh server v2
ssh server vrf mgmt
end
RP/0/0/CPU0:XR1(config)#commit
Further details on Cisco IOS XR hardening you can find in a specific guide.
As you can see, in the Cisco IOS XR we split configuration over various contexts:
- Management-related configuration is composed in the “control-plane management-plane” context.
- The protection of the incoming traffic is put into the data-plane ACLs.
Now, to check that this works, we SSH from the SR1 to the XR1:
2
Connect to address 10.0.0.3 failed
This is not working. Now we try from the management host:
2
3
4
5
6
IMPORTANT: READ CAREFULLY
Welcome to the Demo Version of Cisco IOS XRv (the "Software").
This is working. The final verification of the ACL, how we did that before:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Sat Sep 28 20:44:10.026 UTC
ipv4 access-list ACL_IPV4_DP_PROTECTION
9 remark "ALLOW TRAFFIC BETWEEN 172.16.1.0/24 HOSTS"
10 permit ipv4 172.16.1.0 0.0.0.255 172.16.1.0 0.0.0.255 (10 matches)
19 remark "DENY ANY TRAFFIC TO 172.16.1.0/24 HOSTS"
20 deny ipv4 any 172.16.1.0 0.0.0.255 (2 matches)
29 remark "ALLOW HTTP TRAFFIC TO 172.16.2.0/24 SUBNET"
30 permit tcp any 172.16.2.0 0.0.0.255 eq www (5 matches)
39 remark "BLOCK TRAFFIC TO 172.16.2.0/24 SUBNET"
40 deny ipv4 any 172.16.2.0 0.0.0.255 (7 matches)
49 remark "ALLOW LINK BGP"
50 permit tcp 169.254.0.0 0.0.0.255 169.254.0.0 0.0.0.255 eq bgp (225 matches)
59 remark "BLOCK ANOTHER BGP"
60 deny tcp any any eq bgp
200 permit ipv4 any any (6301 matches)
#2.3. Cumulus/Mellanox control plane protection
You have seen above two approaches to control-plane protection: centralized and distributed. Probably, it is a matter of habit, what is convenient.
As you have learned above, Cumulus Linux relies on iptables, therefore we have here also centralized approach. Comparing to the data plane filtering, we need to choose the proper INPUT chain. The overall process is the same: we create a file with the rules:
2
3
4
5
6
7
8
9
10
11
12
13
total 20
-rw-r--r-- 1 root root 7201 Sep 2 20:51 00control_plane.rules
-rw-r--r-- 1 root root 216 Sep 28 20:55 40_control_plane_protection.rules
-rw-r--r-- 1 root root 274 Sep 28 19:20 50_data_plane_protection.rules
-rw-r--r-- 1 root root 1182 Sep 2 20:51 99control_plane_catch_all.rules
cumulus@MLX-CL:mgmt-vrf:~$ cat /etc/cumulus/acl/policy.d/40_control_plane_protection.rules
[iptables]
-A INPUT --in-interface swp+ -p tcp --dport 22 -j DROP
-A INPUT --in-interface swp+ -s 169.254.0.0/24 -d 169.254.0.0/24 -p tcp --dport 179 -j ACCEPT
-A INPUT --in-interface swp+ -p tcp --dport 179 -j DROP
If there are no default FORWARD rules in Cumulus Linux, there are plenty of default control plane rules. As you can see, there are two default file with rules starting with 00 (applied first) and with 99 (applied last). We’ve created file starting with 40, so they will be applied after the 00, but before 99 (also before our 50 data plane rules).
Once the rules are created, we need to install them:
2
3
4
5
6
7
8
9
10
11
Reading rule file /etc/cumulus/acl/policy.d/00control_plane.rules ...
Processing rules in file /etc/cumulus/acl/policy.d/00control_plane.rules ...
Reading rule file /etc/cumulus/acl/policy.d/40_control_plane_protection.rules ...
Processing rules in file /etc/cumulus/acl/policy.d/40_control_plane_protection.rules ...
Reading rule file /etc/cumulus/acl/policy.d/50_data_plane_protection.rules ...
Processing rules in file /etc/cumulus/acl/policy.d/50_data_plane_protection.rules ...
Reading rule file /etc/cumulus/acl/policy.d/99control_plane_catch_all.rules ...
Processing rules in file /etc/cumulus/acl/policy.d/99control_plane_catch_all.rules ...
Installing acl policy
done.
After that we can start with the testing of the SSH from the Cisco IOS XR towards MLX-CL:
2
3
RP/0/0/CPU0:Sep 28 21:02:24.411 UTC: ssh_xr[65717]: %SECURITY-SSHD-3-ERR_ERRNO : Failed to connect - Connection timed out
And from the management host:
2
3
4
5
6
7
8
9
10
11
12
cumulus@169.254.255.21's password:
Welcome to Cumulus (R) Linux (R)
For support and online technical documentation, visit
http://www.cumulusnetworks.com/support
The registered trademark Linux (R) is used pursuant to a sublicense from LMI,
the exclusive licensee of Linus Torvalds, owner of the mark on a world-wide
basis.
cumulus@mlx-cl:mgmt-vrf:~$
Let’s take a look at the rules in the list of all iptables entries:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
-------------------------------
Listing rules of type iptables:
-------------------------------
TABLE filter :
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 DROP all -- swp+ any 240.0.0.0/5 anywhere
0 0 DROP all -- swp+ any loopback/8 anywhere
0 0 DROP all -- swp+ any base-address.mcast.net/4 anywhere
0 0 DROP all -- swp+ any 255.255.255.255 anywhere
0 0 SETCLASS udp -- swp+ any anywhere anywhere udp dpt:3785 SETCLASS class:7
0 0 POLICE udp -- any any anywhere anywhere udp dpt:3785 POLICE mode:pkt rate:2000 burst:2000
0 0 SETCLASS udp -- swp+ any anywhere anywhere udp dpt:3784 SETCLASS class:7
0 0 POLICE udp -- any any anywhere anywhere udp dpt:3784 POLICE mode:pkt rate:2000 burst:2000
0 0 SETCLASS udp -- swp+ any anywhere anywhere udp dpt:4784 SETCLASS class:7
0 0 POLICE udp -- any any anywhere anywhere udp dpt:4784 POLICE mode:pkt rate:2000 burst:2000
0 0 SETCLASS ospf -- swp+ any anywhere anywhere SETCLASS class:7
0 0 POLICE ospf -- any any anywhere anywhere POLICE mode:pkt rate:2000 burst:2000
0 0 SETCLASS pim -- swp+ any anywhere anywhere SETCLASS class:6
0 0 POLICE pim -- any any anywhere anywhere POLICE mode:pkt rate:2000 burst:2000
0 0 SETCLASS tcp -- swp+ any anywhere anywhere tcp dpt:bgp SETCLASS class:7
0 0 POLICE tcp -- any any anywhere anywhere tcp dpt:bgp POLICE mode:pkt rate:2000 burst:2000
0 0 SETCLASS tcp -- swp+ any anywhere anywhere tcp spt:bgp SETCLASS class:7
0 0 POLICE tcp -- any any anywhere anywhere tcp spt:bgp POLICE mode:pkt rate:2000 burst:2000
0 0 SETCLASS tcp -- swp+ any anywhere anywhere tcp dpt:5342 SETCLASS class:7
0 0 POLICE tcp -- any any anywhere anywhere tcp dpt:5342 POLICE mode:pkt rate:2000 burst:2000
0 0 SETCLASS tcp -- swp+ any anywhere anywhere tcp spt:5342 SETCLASS class:7
0 0 POLICE tcp -- any any anywhere anywhere tcp spt:5342 POLICE mode:pkt rate:2000 burst:2000
0 0 SETCLASS icmp -- swp+ any anywhere anywhere SETCLASS class:2
0 0 POLICE icmp -- any any anywhere anywhere POLICE mode:pkt rate:100 burst:40
0 0 SETCLASS udp -- swp+ any anywhere anywhere udp dpts:bootps:bootpc SETCLASS class:2
0 0 POLICE udp -- any any anywhere anywhere udp dpt:bootps POLICE mode:pkt rate:100 burst:100
0 0 POLICE udp -- any any anywhere anywhere udp dpt:bootpc POLICE mode:pkt rate:100 burst:100
0 0 SETCLASS tcp -- swp+ any anywhere anywhere tcp dpts:bootps:bootpc SETCLASS class:2
0 0 POLICE tcp -- any any anywhere anywhere tcp dpt:bootps POLICE mode:pkt rate:100 burst:100
0 0 POLICE tcp -- any any anywhere anywhere tcp dpt:bootpc POLICE mode:pkt rate:100 burst:100
0 0 SETCLASS udp -- swp+ any anywhere anywhere udp dpt:10001 SETCLASS class:3
0 0 POLICE udp -- any any anywhere anywhere udp dpt:10001 POLICE mode:pkt rate:2000 burst:2000
0 0 SETCLASS igmp -- swp+ any anywhere anywhere SETCLASS class:6
0 0 POLICE igmp -- any any anywhere anywhere POLICE mode:pkt rate:300 burst:100
0 0 DROP tcp -- swp+ any anywhere anywhere tcp dpt:ssh
0 0 ACCEPT tcp -- swp+ any link-local/24 link-local/24 tcp dpt:bgp
0 0 DROP tcp -- swp+ any anywhere anywhere tcp dpt:bgp
0 0 POLICE all -- swp+ any anywhere anywhere ADDRTYPE match dst-type LOCAL POLICE mode:pkt rate:1000 burst:1000 class:2
0 0 POLICE all -- swp+ any anywhere anywhere ADDRTYPE match dst-type IPROUTER POLICE mode:pkt rate:400 burst:100 class:2
0 0 SETCLASS all -- swp+ any anywhere anywhere SETCLASS class:0
There is a problem with the counters on the Mellanox platform running Cumulus Linux. Despite protection works properly, you don’t see the iptables hits.
That’s is how you can protect your network and the customers connected to your network.
You can find the final configuration files on the corresponding GitHub page.
Lessons learned
It took me two times more time to write this article, as I planned. I’ve spent plenty of time on fixing the iptables on the Linux lab server before got to the point of creating iptables rules in Cumulus Linux. So you should always add a huge gap of time to your activities, when you start something big, even if it is familiar to you. So, time is a key.
Conclusion
You might think that there is might be no particular logic in the recent articles, though they are strictly sequential. They are chained by the topics of the hardware dependencies, which exist in the real world and don’t exist in the VMs. The micro series of the 3 articles about Cumulus/Mellanox is over. But, following the motto of my ex-employer Vodafone “The Future is exciting. Ready?” Take care and goodbye!
Support us
P.S.
If you have further questions or you need help with your networks, I’m happy to assist you, just send me message (https://karneliuk.com/contact/). Also don’t forget to share the article on your social media, if you like it.
BR,
Anton Karneliuk
Note that on Cumulus you can add/edit ACLs in a much easier method using `net add acl` or `net add control-plane acl` or `sudo cl-acltool -L all`
Hey Pete,
thanks for the input, though I’m not sure `net add` is easier comparing to direct file modification.
Cheers,
Anton