Up until the latest update to FreeBSD 12.1-RELEASE-p7, limiting jails to a few cores was two lines in the jail.conf
config file. After the update, I had a jail refuse to start and display the following error: cpuset: setaffinity: Resource deadlock avoided
In my jail.conf
I use the exec.poststart
entry to limit the CPUs to which each jail can spawn:
...
exec.poststart = "cpuset -c -l ${cpuset} -j ${name}";
...
Farther down in the same file, I define the jail's allowed CPUs using a variable:
...
$cpuset = "7-8";
...
In this example, all processes on that specific jail are spawned on cpu 7 or 8. Easy.
The error above only appeared on a single jail, the one hosting my internal bind9
DNS services. After a few Google searches, it appears I'm not alone. Same service, same error.
The poster goes into an explanation as to how the bind daemon, named
, will spawn one process for each core on the system. They go on to publish a script, called by the same exec.poststart
stage, that appears to capture all processes spawned by that jail. Then it continues to limit the jail's processes to the desired cores. Or at least I think that's what it says. It's in Russian.
The Fix
My takeaway from the research above was the part about named
spawning a process for each core. As a test, I first let the jail run on all cores by removing the cpuset
restraints. This allowed the jail to start, but as expected, named
processes were started and bound to each cpu core.
Next, I tried disabling the named
service, restoring the cpuset
constraints, then I tried to start the jail. Success! The jail would start. That confirms named
is the culprit.
Next, I logged into the jail and started named
manually: Success! It started and is only bound to the desired cores!
And now for the super elaborate fix: crontab -e
Yup, I know it's hack-y but it does the job. I left the service disabled in /etc/rc.cond.local
and instead started it via cron:
@reboot sleep 5 && service named onestart
This will allow the jail to start, let the exec.poststart
phase execute, then the process starts after constraints are put in place.