In a previous configuration, each FreeBSD jail was placed into its own /30 subnet assigned to respective VLAN. This allowed me to filter traffic to and from each individual jail from a pfsense firewall.

Now that I have a better understanding of what traffic I wanted to permit, I decided to expand the subnets and grouped similar systems. This was a fun weekend project that made me pull out my hair (my fault for chainging too many things at once - I knew better), but it was the nightly cronjobs that puzzled me for a few weeks afterwards.

Every night, I would receive a cronjob email:

Cron Daemon <root@fqdn>
newsyslog: pid file doesn't exist: /var/run/syslog.pid

I logged into the server, it was right, it didn't exist:

# service syslogd status
syslogd is not running.
#service syslogd start
Starting syslogd.
syslogd: child pid 53653 exited with return code 1
/etc/rc.d/syslogd: WARNING: failed to start syslogd
# ls -lah /var/run/syslog*
-rw-r--r--  1 root  wheel     0B May  6 07:36 /var/run/syslogd.sockets

And it failed to start. With the fresh memory that I had recently done some maintenance on the jails, I remembered that most of them got a new IP address in the new expanded subnets.

# grep syslogd /etc/rc.*
/etc/rc.conf.local:syslogd_flags="-c -b 172.16.1.36"

And there's the issue, the IP address that I used to bind the syslogd listener to was incorrect! After I updated the IP, the daemon started successfully:

#service syslogd start
Starting syslogd.

Using the rc.conf flags above, I bound the syslogd daemon to the jail's IP. By default syslogd will bind to * and conflict with other jails.

Now, what should I have done to prevent this? Use Ansible!