Hello my friend,
In the vast majority of cases we speak about the network, network devices and network configuration. Which is absolutely legitimate, as we write about the networks. However, sometimes exactly the same network technologies live in the server world and do the same things under different names. So today we’ll take a look how to create the high-available server cluster using the Keepalived.
2
3
4
5
retrieval system, or transmitted in any form or by any
means, electronic, mechanical or photocopying, recording,
or otherwise, for commercial purposes without the
prior permission of the author.
Servers are perfect for automation
The network exists to span the applications and customers. However, it doesn’t host applications itself; instead, they are hosted on the servers. That’s why automation of the servers is very big topic and, in all honesty, the automation originally has started in the server world.
In the same time, the tools and approaches you learn at our network automation training are universal: they are applicable both for the network and servers automation. Automate all things!
At our network automation training we explore the Linux setup and administration so that you can build your Linux-based lab and manage such applications as KVM and Docker in addition to the Linux bridges and other types of networking. That is a very important part of the training and pleasant addition to the detailed overview and hands-on experience with NETCONF/RESTCONF/GNMI with YANG, JSON/XML/YAML/Protobuf and Python/Ansible/Bash in the context of the managing your network devices from Cisco, Nokia, Arista, and Cumulus.
Brief description
The high-availability is without doubts one of the corner stones of the modern applications. With more and more mission critical business process from financial reporting and forecasting to online purchase of everything to tele-operated surgery, there is no room for the application to be down. It requires the redundancy on multiple levels including the infrastructure. At the network level, the whole series about the Data Centre networking covers different scenarios of the network build including the fast reroute, CLOS topologies and so on. That’s why today we want to talk a bit about servers.
By no mean we pretend to cover all the aspects of the resilient infrastructure on servers side with this blogpost. Instead, it is just the first step in that direction.
The one of the protocols we’ve recently encountered during build of the servers cluster is called Keeplived. For those, who works on the server side, it is not new at all, as it exists already for a while. However, for us, networking folks, it was quite and interesting discovery. The Keeplaived has three main components:
- Various checks (TCP, HTTP GET, etc) to check if the remote service is available.
- Load balancing (which can also be used on the results of service check from the previous point).
- Failover based on the VRRP.
In today’s blogpost we’ll focus on the VRRP piece.
Topology
In this setup we are using the following topology. The network part is not particularly important here, as we are focusing on the servers and their configuration:
If you are interested in the multivendor data centre implementation and interoperability, read the corresponding series.
Implementation
Here you will learn how to install and configure the KeepaliveD and some other Linux components to allow it working.
#1. Install KeepaliveD
First of all, you need to install the keeplaived at your Linux. You can do that in any Linux distribution. We using the CentOS 8 in this lab. Unfortunately, given the latest news from Red Hat and IBM, the CentOS 8 is now being discounted. However, CentOS 7 will still be around till 2024, so you still can use it in your projects
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
Last metadata expiration check: 0:09:50 ago on Fri 25 Dec 2020 01:22:03 PM GMT.
Package keepalived-2.0.10-10.el8.x86_64 is already installed.
Dependencies resolved.
=========================================================================================================================================================================================
Package Architecture Version Repository Size
=========================================================================================================================================================================================
Upgrading:
keepalived x86_64 2.0.10-11.el8 AppStream 466 k
Transaction Summary
=========================================================================================================================================================================================
Upgrade 1 Package
Total download size: 466 k
Downloading Packages:
keepalived-2.0.10-11.el8.x86_64.rpm 795 kB/s | 466 kB 00:00
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 496 kB/s | 466 kB 00:00
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Running scriptlet: keepalived-2.0.10-11.el8.x86_64 1/1
Upgrading : keepalived-2.0.10-11.el8.x86_64 1/2
Running scriptlet: keepalived-2.0.10-11.el8.x86_64 1/2
Running scriptlet: keepalived-2.0.10-10.el8.x86_64 2/2
Cleanup : keepalived-2.0.10-10.el8.x86_64 2/2
Running scriptlet: keepalived-2.0.10-10.el8.x86_64 2/2
Verifying : keepalived-2.0.10-11.el8.x86_64 1/2
Verifying : keepalived-2.0.10-10.el8.x86_64 2/2
Installed products updated.
Upgraded:
keepalived-2.0.10-11.el8.x86_64
Complete!
Come to our network automation training to learn more about Linux.
Once the application is installed, you can validate that:
2
3
4
● keepalived.service - LVS and VRRP High Availability Monitor
Loaded: loaded (/usr/lib/systemd/system/keepalived.service; disabled; vendor preset: disabled)
Active: inactive (dead)
As you can see, by default the application is not enabled. We will do that once we will put the proper configuration in place.
#2. Configure KeepaliveD
The process of the configuration is relatively straightforward, if you are familiar with Linux:
- Modify the configuration file
- Restart the corresponding daemon
The configuration file for the KeepaliveD is located at /etc/keepalived/keepalived.conf. For the master in your cluster the configuration shall be the following:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
vrrp_instance VI_1 {
state MASTER
interface enp0s8
virtual_router_id 11
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass SUPER_PASS
}
virtual_ipaddress {
100.65.0.10/24
}
}
If you are familiar with the VRRP configuration on the routers, this snippet is self-explanatory. However, if you are not, it could cause some questions. Here are the details:
- name of the configuration group at vrrp_instance.
- default state of the host.
- interface, where the VRRP process is running.
- virtual_router_id which identifies the cluster. This parameter must be same for all deices in group.
- The VRRP priority (the host with the higher priority wins election and becomes active once).
- advert_int is a timer instructing the process how often the keep alive messages shall be sent.
- authentication of the hosts inside the group to prevent malicious hosts to take part.
- finally virtual_address contains IP address, which the hosts will be using as a virtual one.
This tools has quite good documentation, which you can use to implement this (or any further) use case.
The backup host shall have the following configuration:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
vrrp_instance VI_1 {
state BACKUP
interface enp0s8
virtual_router_id 11
priority 1
advert_int 1
authentication {
auth_type PASS
auth_pass SUPER_PASS
}
virtual_ipaddress {
100.65.0.10/24
}
}
Once the configuration is complete, you need to start the the keepalived.service on your Linux hosts:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
$ sudo systemctl status keepalived.service
● keepalived.service - LVS and VRRP High Availability Monitor
Loaded: loaded (/usr/lib/systemd/system/keepalived.service; disabled; vendor preset: disabled)
Active: active (running) since Fri 2020-12-25 14:20:25 GMT; 11s ago
Process: 34334 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 34336 (keepalived)
Tasks: 2 (limit: 23824)
Memory: 1.8M
CGroup: /system.slice/keepalived.service
├─34336 /usr/sbin/keepalived -D
└─34337 /usr/sbin/keepalived -D
Dec 25 14:20:28 c8-h1 Keepalived_vrrp[34337]: Sending gratuitous ARP on enp0s8 for 100.65.0.10
Dec 25 14:20:28 c8-h1 Keepalived_vrrp[34337]: Sending gratuitous ARP on enp0s8 for 100.65.0.10
Dec 25 14:20:28 c8-h1 Keepalived_vrrp[34337]: Sending gratuitous ARP on enp0s8 for 100.65.0.10
Dec 25 14:20:28 c8-h1 Keepalived_vrrp[34337]: Sending gratuitous ARP on enp0s8 for 100.65.0.10
Dec 25 14:20:33 c8-h1 Keepalived_vrrp[34337]: Sending gratuitous ARP on enp0s8 for 100.65.0.10
Dec 25 14:20:33 c8-h1 Keepalived_vrrp[34337]: (VI_1) Sending/queueing gratuitous ARPs on enp0s8 for 100.65.0.10
Dec 25 14:20:33 c8-h1 Keepalived_vrrp[34337]: Sending gratuitous ARP on enp0s8 for 100.65.0.10
Dec 25 14:20:33 c8-h1 Keepalived_vrrp[34337]: Sending gratuitous ARP on enp0s8 for 100.65.0.10
Dec 25 14:20:33 c8-h1 Keepalived_vrrp[34337]: Sending gratuitous ARP on enp0s8 for 100.65.0.10
Dec 25 14:20:33 c8-h1 Keepalived_vrrp[34337]: Sending gratuitous ARP on enp0s8 for 100.65.0.10
Once this configuration is complete, there is one more thing we need to do: change the built-in Linux firewall (iptabels in our case) configuration to allow the communication for the VRRP.
#3. Configure iptables
By default the Linux security rules are quite restrictive. As such you need to allow the traffic from the adjacent servers for the VRRP; otherwise, it will be dropped. To do that you should add the following rules:
If needed, your rule can be more restrictive (e.g., it can contain the source subnet or even the specific hosts, which are allowed to send VRRP packets).
Validation
#1. Control plane
The validation of the control plane for KeepaliveD is based on the default Linux principles: checking the log of the specific daemon. On the Master node you should see that it has entered the MASTER state and are sending gratuitous ARP:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
! SOME OUTPUT IS TRUNCATED
Dec 25 14:36:41 c8-h1 Keepalived[34897]: Starting Keepalived v2.0.10 (11/12,2018)
Dec 25 14:36:41 c8-h1 Keepalived[34897]: Running on Linux 4.18.0-193.28.1.el8_2.x86_64 #1 SMP Thu Oct 22 00:20:22 UTC 2020 (built for Linux 4.18.0)
Dec 25 14:36:41 c8-h1 Keepalived[34897]: Command line: '/usr/sbin/keepalived' '-D'
Dec 25 14:36:41 c8-h1 Keepalived[34897]: Opening file '/etc/keepalived/keepalived.conf'.
Dec 25 14:36:47 c8-h1 Keepalived[34904]: Starting VRRP child process, pid=34905
Dec 25 14:36:47 c8-h1 Keepalived_vrrp[34905]: Registering Kernel netlink reflector
Dec 25 14:36:47 c8-h1 Keepalived_vrrp[34905]: Registering Kernel netlink command channel
Dec 25 14:36:47 c8-h1 Keepalived_vrrp[34905]: Opening file '/etc/keepalived/keepalived.conf'.
Dec 25 14:36:47 c8-h1 Keepalived_vrrp[34905]: (Line 11) Truncating auth_pass to 8 characters
Dec 25 14:36:47 c8-h1 Keepalived_vrrp[34905]: Assigned address 100.65.0.11 for interface enp0s8
Dec 25 14:36:47 c8-h1 Keepalived_vrrp[34905]: Registering gratuitous ARP shared channel
Dec 25 14:36:47 c8-h1 Keepalived_vrrp[34905]: (VI_1) removing VIPs.
Dec 25 14:36:47 c8-h1 Keepalived_vrrp[34905]: (VI_1) Entering BACKUP STATE (init)
Dec 25 14:36:47 c8-h1 Keepalived_vrrp[34905]: VRRP sockpool: [ifindex(3), family(IPv4), proto(112), unicast(0), fd(10,11)]
Dec 25 14:36:47 c8-h1 Keepalived_vrrp[34905]: (VI_1) received lower priority (1) advert from 100.65.0.12 - discarding
Dec 25 14:36:47 c8-h1 Keepalived_vrrp[34905]: (VI_1) Backup received priority 0 advertisement
Dec 25 14:36:48 c8-h1 Keepalived_vrrp[34905]: (VI_1) Receive advertisement timeout
Dec 25 14:36:48 c8-h1 Keepalived_vrrp[34905]: (VI_1) Entering MASTER STATE
Dec 25 14:36:48 c8-h1 Keepalived_vrrp[34905]: (VI_1) setting VIPs.
Dec 25 14:36:48 c8-h1 Keepalived_vrrp[34905]: Sending gratuitous ARP on enp0s8 for 100.65.0.10
Dec 25 14:36:48 c8-h1 Keepalived_vrrp[34905]: (VI_1) Sending/queueing gratuitous ARPs on enp0s8 for 100.65.0.10
Dec 25 14:36:48 c8-h1 Keepalived_vrrp[34905]: Sending gratuitous ARP on enp0s8 for 100.65.0.10
On the BACKIP node you shall see that it is in the appropriate state as well:
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
! SOME OUTPUT IS TRUNCATED
Dec 25 14:36:49 localhost Keepalived[34662]: Starting Keepalived v2.0.10 (11/12,2018)
Dec 25 14:36:49 localhost Keepalived[34662]: Running on Linux 4.18.0-193.28.1.el8_2.x86_64 #1 SMP Thu Oct 22 00:20:22 UTC 2020 (built for Linux 4.18.0)
Dec 25 14:36:49 localhost Keepalived[34662]: Command line: '/usr/sbin/keepalived' '-D'
Dec 25 14:36:49 localhost Keepalived[34662]: Opening file '/etc/keepalived/keepalived.conf'.
Dec 25 14:36:49 localhost Keepalived[34664]: Starting VRRP child process, pid=34665
Dec 25 14:36:49 localhost Keepalived_vrrp[34665]: Registering Kernel netlink reflector
Dec 25 14:36:49 localhost Keepalived_vrrp[34665]: Registering Kernel netlink command channel
Dec 25 14:36:49 localhost Keepalived_vrrp[34665]: Opening file '/etc/keepalived/keepalived.conf'.
Dec 25 14:36:49 localhost Keepalived_vrrp[34665]: (Line 9) Truncating auth_pass to 8 characters
Dec 25 14:36:49 localhost Keepalived_vrrp[34665]: Assigned address 100.65.0.12 for interface enp0s8
Dec 25 14:36:49 localhost Keepalived_vrrp[34665]: Registering gratuitous ARP shared channel
Dec 25 14:36:49 localhost Keepalived_vrrp[34665]: (VI_1) removing VIPs.
Dec 25 14:36:49 localhost Keepalived_vrrp[34665]: (VI_1) Entering BACKUP STATE (init)
Dec 25 14:36:49 localhost Keepalived_vrrp[34665]: VRRP sockpool: [ifindex(3), family(IPv4), proto(112), unicast(0), fd(10,11)]
#2 Data plane
In this part you will the see the failover case. First of all, take a look on the topology provided above to remind the state. Here are the original state:
| Device | Type | Role |
|---|---|---|
| H1 | CentOS 8 | Master KeepliveD node |
| H2 | CentOS 8 | Backup KeepalieveD node |
| L1 | Cumulus Linux | connecting H1 |
| L2 | Cumulus Linux | connecting H2 |
| I1 | Cumulus Linux | Host somewhere in Internet (external to DC) |
As H1 is active, you will see the virtual IP address 100.65.0.10 at its interface, where keepalived is enabled:
2
3
4
5
6
7
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 08:00:27:7d:af:b4 brd ff:ff:ff:ff:ff:ff
inet 100.65.0.11/24 scope global enp0s8
valid_lft forever preferred_lft forever
inet 100.65.0.10/24 scope global secondary enp0s8
valid_lft forever preferred_lft forever
The H2 at this stage doesn’t have the IP address assigned:
2
3
4
5
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 08:00:27:70:07:ae brd ff:ff:ff:ff:ff:ff
inet 100.65.0.12/24 scope global enp0s8
valid_lft forever preferred_lft forever
In the provided configuration, the virtual IP addresses shares the MAC with the physical IP address of the interface (though it can be modified to use virtual MAC as well).
We run a standard EVPN anycast GW based on the asymmetric routing in our network, so the VLAN connecting both these servers presents on both L1 and L2. As such, you can see the IP and MAC of the virtual IP address from the servers:
2
3
4
5
6
100.65.0.10 dev vlan100 lladdr 08:00:27:7d:af:b4 REACHABLE
100.65.0.11 dev vlan100 lladdr 08:00:27:7d:af:b4 REACHABLE
100.65.0.10 dev vlan100-v0 lladdr 08:00:27:7d:af:b4 STALE
100.65.0.11 dev vlan100-v0 lladdr 08:00:27:7d:af:b4 STALE
100.65.0.12 dev vlan100 lladdr 08:00:27:70:07:ae REACHABLE
So the information seems to be learned and properly installed. Finally we can start our ping test from I1 host to observe the failover:
2
3
4
5
PING 100.65.0.10 (100.65.0.10) 56(84) bytes of data.
64 bytes from 100.65.0.10: icmp_seq=1 ttl=61 time=1.94 ms
64 bytes from 100.65.0.10: icmp_seq=2 ttl=61 time=2.14 ms
64 bytes from 100.65.0.10: icmp_seq=3 ttl=61 time=1.67 ms
#3. Faliover
Now, let’s shut down the enp0s8 interface on H1 to simulate the server outage:
2
3
4
5
6
7
8
H1$ ip addr show dev enp0s8
3: enp0s8: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
link/ether 08:00:27:7d:af:b4 brd ff:ff:ff:ff:ff:ff
inet 100.65.0.11/24 scope global enp0s8
valid_lft forever preferred_lft forever
The ping meanwhile continues from I1. For the convergence time it is getting dropped (for 5 seconds in our case) and the continue:
2
3
4
5
! OUTPUT IS OMITTED
64 bytes from 100.65.0.10: icmp_seq=31 ttl=61 time=3.45 ms
64 bytes from 100.65.0.10: icmp_seq=36 ttl=61 time=2.32 ms
! OUTPUT IS OMITTED
During that time the H2 is going from the backup to master state:
2
3
4
5
6
7
8
! SOME OUTPUT IS TRUNCATED
Dec 25 15:29:04 localhost Keepalived_vrrp[34665]: (VI_1) Receive advertisement timeout
Dec 25 15:29:04 localhost Keepalived_vrrp[34665]: (VI_1) Entering MASTER STATE
Dec 25 15:29:04 localhost Keepalived_vrrp[34665]: (VI_1) setting VIPs.
Dec 25 15:29:04 localhost Keepalived_vrrp[34665]: Sending gratuitous ARP on enp0s8 for 100.65.0.10
Dec 25 15:29:04 localhost Keepalived_vrrp[34665]: (VI_1) Sending/queueing gratuitous ARPs on enp0s8 for 100.65.0.10
Dec 25 15:29:04 localhost Keepalived_vrrp[34665]: Sending gratuitous ARP on enp0s8 for 100.65.0.10
At this point you could see that H2 owns the 100.65.0.10 IP address:
2
3
4
5
6
7
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 08:00:27:70:07:ae brd ff:ff:ff:ff:ff:ff
inet 100.65.0.12/24 scope global enp0s8
valid_lft forever preferred_lft forever
inet 100.65.0.10/24 scope global secondary enp0s8
valid_lft forever preferred_lft forever
And the network fabric has relearned the new ARP entry for the IP:
2
3
4
5
6
100.65.0.12 dev vlan100-v0 lladdr 08:00:27:70:07:ae STALE
100.65.0.10 dev vlan100 lladdr 08:00:27:70:07:ae REACHABLE
100.65.0.11 dev vlan100 lladdr 08:00:27:7d:af:b4 REACHABLE
100.65.0.10 dev vlan100-v0 lladdr 08:00:27:70:07:ae STALE
100.65.0.12 dev vlan100 lladdr 08:00:27:70:07:ae REACHABLE
So your service continue being provided.
When the primary host is coming back, it will take over the operation again, but this time it will be without any single packet drop.
Messages from H1 during the restoration of cluster:
2
3
4
5
6
7
8
9
10
11
! SOME OUTPUT IS TRUNCATED
Dec 25 15:37:19 c8-h1 Keepalived_vrrp[34905]: Netlink reports enp0s8 up
Dec 25 15:37:19 c8-h1 Keepalived_vrrp[34905]: (VI_1) Entering BACKUP STATE
Dec 25 15:37:20 c8-h1 Keepalived_vrrp[34905]: (VI_1) received lower priority (1) advert from 100.65.0.12 - discarding
Dec 25 15:37:21 c8-h1 Keepalived_vrrp[34905]: (VI_1) received lower priority (1) advert from 100.65.0.12 - discarding
Dec 25 15:37:22 c8-h1 Keepalived_vrrp[34905]: (VI_1) received lower priority (1) advert from 100.65.0.12 - discarding
Dec 25 15:37:23 c8-h1 Keepalived_vrrp[34905]: (VI_1) Receive advertisement timeout
Dec 25 15:37:23 c8-h1 Keepalived_vrrp[34905]: (VI_1) Entering MASTER STATE
Dec 25 15:37:23 c8-h1 Keepalived_vrrp[34905]: (VI_1) setting VIPs.
Dec 25 15:37:23 c8-h1 Keepalived_vrrp[34905]: Sending gratuitous ARP on enp0s8 for 100.65.0.10
Messages from H2:
2
3
4
5
6
! SOME OUTPUT IS TRUNCATED
Dec 25 15:37:15 localhost Keepalived_vrrp[34665]: Sending gratuitous ARP on enp0s8 for 100.65.0.10
Dec 25 15:37:23 localhost Keepalived_vrrp[34665]: (VI_1) Master received advert from 100.65.0.11 with higher priority 100, ours 1
Dec 25 15:37:23 localhost Keepalived_vrrp[34665]: (VI_1) Entering BACKUP STATE
Dec 25 15:37:23 localhost Keepalived_vrrp[34665]: (VI_1) removing VIPs.
The system is back to normal operation.
Lessons learned
Different Linux distributions have different security configurations. It took as bit to fix the iptables for this lab. One of the reasons is that we were installing the KeepaliveD on a fresh CentOS 8 setup, which doesn’t use iptables by default. Instead, it uses nftables. That’s why we first have enabled the iptables and then implemented the fix.
Conclusion
When you look on any application from the perspective of the end-user, it requires only one thing: make sure that application is properly working during the time the customer user needs that. Being a simple requirement on paper, it may be not easy to implement in reality. Knowing the various approaches, how to use on the network and/or server side, allows you do that and make sure your customers stay with you and bring their friends. Take care and good bye.
Support us
P.S.
If you have further questions or you need help with your networks, I’m happy to assist you, just send me message. Also don’t forget to share the article on your social media, if you like it.
BR,
Anton Karneliuk