With FreeBSD jails deployed around the world, static routing was getting a bit out of hand. Plus, when I needed to move a jail from one data center to another, I would have to update routing tables across multiple sites. Not ideal. Enter dynamic routing...
OSPF (open shortest path first) is an internal dynamic routing protocol that provides the autonomy that I needed and it's fairly easy to setup. This article does not cover configuration of VPN links, ZFS, or Freebsd jails, however it's recommended that you use seperate ZFS datasets per jail so that migration between hosts can be done with zfs send & receive.
In this scenario, we have five FreeBSD servers in two different data centers. Each physical server runs anywhere between three to ten jails. When jails are deployed, they are assigned a /32 IP on lo2. From here, pf handles inbound port forwarding and outbound NAT. Links between each server are provided by OpenVPN TAP interfaces. (I used TAP to pass layer 2 traffic. I seem to remember that I needed TAP interfaces due to needing GRE tunnels on top of TUN interfaces to get OSPF to communicate. I've heard TAP is slower than TUN so I may revisit this.)
In this example, we will use 172.16.2.0/24 as the range for OpenVPN P2P links and 172.16.3.0/24 as the range of IPs available for assignment to each jail. Previously, when deploying a jail, I assigned IPs based on the following groups:
- Server 1: 172.16.3.0/28
- Server 2: 172.16.3.16/28
- Server 3: 172.16.3.32/28
- Server 4: 172.16.3.48/28
- Server 5: 172.16.3.64/28
When statically routing, this made routing tables a bit smaller and easier to manage. However, when I needed to migrate a jail to a new host, I had to add a new /32 to all routing tables. Now, with OSPF, this is no longer an issue, nor is it required.
To get started, first we install the Quagga package:
pkg install quagga
This installs the zebra core routing daemon along with most popular dynamic routing protocols: ospf, bgp, rip, isis.
The two configuration files needed to get OSPFv2 running are /usr/local/etc/quagga/zebra.conf and /usr/local/etc/quagga/ospfd.conf.
(v2 = IPv4)
Starting with zebra.conf, we'll define the hostname and a management password:
hostname server1
password CorrectHorseBatteryStaple
(Modify the server name for each server. The rest can remain the same for all hosts.)
Second, we will populate the ospfd.conf file:
hostname server1
password CorrectHorseBatteryStaple
service advanced-vty
!
interface tap1
description TAP to server 1
ip ospf authentication message-digest
ip ospf message-digest-key 1 md5 CorrectHorseBatteryRoute
!
interface tap2
description TAP to server 2
ip ospf authentication message-digest
ip ospf message-digest-key 1 md5 CorrectHorseBatteryRoute
!
interface tap3
description TAP to server 3
ip ospf authentication message-digest
ip ospf message-digest-key 1 md5 CorrectHorseBatteryRoute
!
interface tap4
description TAP to server 4
ip ospf authentication message-digest
ip ospf message-digest-key 1 md5 CorrectHorseBatteryRoute
!
interface tap5
description TAP to server 5
ip ospf authentication message-digest
ip ospf message-digest-key 1 md5 CorrectHorseBatteryRoute
!
router ospf
ospf router-id 10.0.0.1
passive-interface default
no passive-interface tap1
no passive-interface tap2
no passive-interface tap3
no passive-interface tap4
no passive-interface tap5
network 172.16.2.0/23 area 0.0.0.0
area 0.0.0.0 authentication
!
line vty
!
(Again, modify the hostname and the router-id for each server. The interfaces should also reflect the local system's interface names but if the interface does not exist, it will ignore it. This allows you to use roughly the same configuration file on all hosts.)
To break this down:
service advanced-vtyallows you to skip theenorenablecommand. Since I'm the only one who uses this service, it's one less command to type.ip ospf authentication message-digestandip ospf message-diget-key...ignores non-authenticated OSPF communication. This is useful when communicating over the WAN and to prevent a replay attack. Since I'm using a VPN to communicate, I could exclude these.passive-interface defaultturns off the active communication of OSPF messages on all interfaces except for the interfaces listed asno passive-interface [interface name]. Since my ospf communication needs to leverage the VPNs, this prevents the servers from trying to send ospf data out the WAN interface (a firewall would work too).network 172.16.2.0/23 area 0.0.0.0lists a supernet of both172.16.2.0/24and172.16.3.0/24. This ensures routes for the jails are advertised along with the P2P links used by OpenVPN. The OpenVPN links are not required but can provide another IP to access your server if one of the links goes down. (See the suggested tasks below).
At this point, we can enable the services in rc.conf.local and start them:
sysrc quagga_enable=YES -f /etc/rc.conf.local
sysrc quagga_flags="-A 127.0.0.1" -f /etc/rc.conf.local
sysrc quagga_daemons="zebra ospfd" -f /etc/rc.conf.local
service start quagga
We bind the management interface to 127.0.0.1 so that it's only accessable to local telnet sessions. If you want to access this service remotely, you can bind to a remotely accessable IP. Remember telnet is not secure. If you need remote access, use a VPN.
To manage the services, you can telnet to your host's localhost address.
Use port 2601 for the zebra core:
telnet localhost 2601
Use 2604 for the ospf service:
telnet localhost 2604
Remember, this is accessible by non-root users so set a good password.
Suggested tasks
-
When deploying a dynamic routing solution with multiple paths to systems, I recommend you assign non 127.0.0.1, unique IP addresses to a secondary loopback adapter:
cloned_interfaces="lo1" ifconfig_lo1="inet 10.0.0.1/32"What this does is allows you to reach this IP for SSH or Nagios/Zabbix monitoring even when a VPN link goes down. OSPF will detect the change and your routing tables will start listing a new route to the loopback addresses.
Adding the following to your OSPF config would allow you to advertise the loopback address:network 10.0.0.0/24 area 0.0.0.0 -
Lastly, I suggest you telnet into both services (as shown above) and execute the following commands to prevent your passwords from being stored in the clear:
conf t service password-encryptionThis will change
password CorrectHorseBatteryStapleintopassword 8 aJIOIhdni897.
Testing & Status
show ip ospf neighbor should show you all of your neighboring systems. Referring to the diagram at the top of this article, server1 should only see two neighboring systems (server2 & server3):
server1# show ip ospf neighbor
Neighbor ID Pri State Dead Time Address Interface RXmtL RqstL DBsmL
10.0.0.2 1 Full/DR 35.051s 10.8.0.150 tap2:172.16.2.2 0 0 0
10.0.0.3 1 Full/Backup 36.209s 10.8.0.154 tap3:172.16.2.6 0 0 0
To show ospf's learned routes, telnet to 2604 (telnet localhost 2604) and issue a show ip ospf route:
server1# show ip ospf route
============ OSPF network routing table ============
...
N 172.16.3.0/32 [0] area: 0.0.0.0
directly attached to lo2
N 172.16.3.1/32 [0] area: 0.0.0.0
directly attached to lo2
N 172.16.3.2/32 [0] area: 0.0.0.0
directly attached to lo2
N 172.16.3.16/32 [11] area: 0.0.0.0
via 172.16.2.2, tap2
N 172.16.3.17/32 [11] area: 0.0.0.0
via 172.16.2.2, tap2
N 172.16.3.32/32 [10] area: 0.0.0.0
via 172.16.2.6, tap3
N 172.16.3.34/32 [10] area: 0.0.0.0
via 172.16.2.6, tap3
N 172.16.3.48/32 [20] area: 0.0.0.0
via 172.16.2.6, tap3
N 172.16.3.50/32 [20] area: 0.0.0.0
via 172.16.2.6, tap3
N 172.16.3.64/32 [25] area: 0.0.0.0
via 172.16.2.6, tap3
N 172.16.3.65/32 [25] area: 0.0.0.0
via 172.16.2.6, tap3
N 172.16.3.66/32 [25] area: 0.0.0.0
via 172.16.2.6, tap3
N 172.16.3.67/32 [25] area: 0.0.0.0
via 172.16.2.6, tap3
N 172.16.3.68/32 [25] area: 0.0.0.0
via 172.16.2.6, tap3
N 172.16.3.70/32 [25] area: 0.0.0.0
via 172.16.2.6, tap3
N 172.16.3.71/32 [25] area: 0.0.0.0
via 172.16.2.6, tap3
N 172.16.3.72/32 [25] area: 0.0.0.0
via 172.16.2.6, tap3
...
To show your kernel's routing table using zebra, telnet to 2601 (telnet localhost 2601) and issue a show ip route:
server1# show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, P - PIM, A - Babel, N - NHRP,
> - selected route, * - FIB route
K>* 0.0.0.0/0 via 123.123.123.1, vtnet0
...
O 172.16.3.0/32 [110/0] is directly connected, lo2, 01w2d13h
C>* 172.16.3.0/32 is directly connected, lo2
O 172.16.3.1/32 [110/0] is directly connected, lo2, 01w2d13h
C>* 172.16.3.1/32 is directly connected, lo2
O 172.16.3.2/32 [110/0] is directly connected, lo2, 01w2d13h
C>* 172.16.3.2/32 is directly connected, lo2
O>* 172.16.3.16/32 [110/11] via 172.16.2.2, tap2, 4d19h15m
O>* 172.16.3.17/32 [110/11] via 172.16.2.2, tap2, 4d19h15m
O>* 172.16.3.32/32 [110/10] via 172.16.2.6, tap3, 00:11:53
O>* 172.16.3.34/32 [110/10] via 172.16.2.6, tap3, 00:11:53
O>* 172.16.3.48/32 [110/20] via 172.16.2.6, tap3, 00:11:53
O>* 172.16.3.50/32 [110/20] via 172.16.2.6, tap3, 00:11:53
O>* 172.16.3.64/32 [110/25] via 172.16.2.6, tap3, 00:11:53
O>* 172.16.3.65/32 [110/25] via 172.16.2.6, tap3, 00:11:53
O>* 172.16.3.66/32 [110/25] via 172.16.2.6, tap3, 00:11:53
O>* 172.16.3.67/32 [110/25] via 172.16.2.6, tap3, 00:11:53
O>* 172.16.3.68/32 [110/25] via 172.16.2.6, tap3, 00:11:53
O>* 172.16.3.70/32 [110/25] via 172.16.2.6, tap3, 00:11:53
O>* 172.16.3.71/32 [110/25] via 172.16.2.6, tap3, 00:11:53
O>* 172.16.3.72/32 [110/25] via 172.16.2.6, tap3, 00:11:53
...
Further reading
Here's a write-up on how to use Ansible to deploy ospfd.conf files.