I use Ansible to deploy FreeBSD jails to my servers. I was running into an issue where the jails were starting without network connectivity. This led me down a path of researching the best way to accomplish my isolated network configuration for untrusted jails.

The Setup

I use a single 10G network interface on my main server. This provides connectivity to the baremetal host and also trunks about 30 VLANs to provide connectivity to various jails.

Originally I defined all VLANs in the /etc/rc.conf.local file under the cloned_interfaces= setting. This worked, but as I integrated more of the jail provisioning process into Ansible, the modifications became a bit more cumbersome.

Now, I take advantage of the /etc/rc.conf.d/netif directory and make minimal changes to the /etc/rc.conf.local file.

Ansible

Using some community modules and pre-deployment prompts, I can now reliably deploy jails with VNET interfaces bridged to VLANs. I'll break down the playbook below and explain each module.

- name: VNet settings
  when: hostvars['localhost'].vnet | default(false) | bool
  block:
    - name: Add vlan to rc.conf.local interface list
      community.general.sysrc:
        name: "vlans_{{ hostvars['localhost'].interface }}"
        state: value_present
        value: "{{ hostvars['localhost'].vlan }}"
        path: /etc/rc.conf.local

    - name: Create persistent vlan entry
      ansible.builtin.template:
        src: templates/netif.j2
        dest: "/etc/rc.conf.d/netif/vlan{{ hostvars['localhost'].vlan }}.conf"
        mode: "0640"

    - name: Bring up vlan interface
      ansible.builtin.command:
        cmd: "ifconfig vlan{{ hostvars['localhost'].vlan }} create vlan {{ hostvars['localhost'].vlan }}
          vlandev {{ hostvars['localhost'].interface }} name vlan{{ hostvars['localhost'].vlan }} up"
      when: "'vlan' + hostvars['localhost'].vlan not in ansible_facts.interfaces"

    - name: Pause for 4 seconds to wait for vlan interface to come up with proper MTU.
      ansible.builtin.pause:
        seconds: 4

- name: Start jail
  ansible.builtin.command:
    cmd: "jail -c {{ hostvars['localhost'].jail }}"
  register: result
  until: result.rc == 0
  retries: 10
  delay: 10
  changed_when: "'Starting sshd' in result.stdout"

- name: Setup DNS
  ansible.builtin.template:
    src: resolv.conf.j2
    dest: "{{ jpath }}/etc/resolv.conf"
    mode: "0644"

- name: Install required packages
  ansible.builtin.command:
    cmd: "jexec {{ hostvars['localhost'].jail }} env ASSUME_ALWAYS_YES=yes pkg install -y sudo python security/pam_ssh_agent_auth ca_root_nss"
  register: packages
  changed_when: packages.rc == "0"
  until: packages.rc == 0
  retries: 100
  delay: 10

Block

I start with a block module. Since most of my servers are VPS' hosted with a single IP address, they don't use VLANs or VNETs. The block module includes a when conditional that only runs if the fact vnet is true.

sysrc

Using the sysrc module, I ensure the VLAN number is included in the list of VLANs for the server's interface. The sysrc module performs an append where it will only add the value if it does not exist in the current value. This allows me to deploy multiple jails in a single VLAN but only add a single VLAN ID to he vlans_<intf>= value in /etc/rc.conf.local.
The interface and vlan facts are set during the pre-deployment process. Depending on the server selected, Ansible gathers facts using an include_vars for the destination host.

template (vlan config)

The template module builds the VLAN rc file in the /etc/rc.conf.d/netif/ directory. Ansible is idempotent so if this file already exists (meaning the VLAN already exists), it will simple mark this task as "OK" and move on.

command (bring up vlan)

This module brings up the VLAN interface, but like the previous task, Ansible is aware of the system's state. Using the host's interface facts, Ansible will only perform this task if the VLAN is missing from the server's interface list.

pause

Pausing the playbook here was the final key to successfully bringing up the interface. By default, most Ethernet network interfaces operate at 1500 mtu. My server's physical 10G interface is configured for 9000 mtu. When the VLAN comes up with the command module, it takes a second or two for the VLAN interface to update its mtu to 9000. Without the pause, the jail's bridge would inherit the original 1500 mtu from the VLAN interface, then drop the VLAN interface from the bridge when the VLAN mtu was updated. In short, this allows the VLAN interface to settle before bringing up the jail.

command (start jail)

This module starts the jail. I can't remember why I added the loop. It might be left over from when I was troubleshooting the VLAN mtu issue.

template (DNS)

Using the template module, I configure the jail's DNS settings.

command (install packages)

The final check that everything was configured correctly is the command module that tries to install the initial packages. It calls the pkg utility from the host using jexec to install some baseline packages needed for Ansible. It also registers the result and will continue looping 100 times until the command's exit code is "0". Until then, it will keep trying every 10 seconds. During this time, I can correct any configuration errors on my part (switch VLAN config, firewall VLAN interface, NAT settings, etc...)