During a late night of troubleshooting a Zabbix queue issue, I convinced myself that the network setup for my jailed environment was faulty. After some rest, it turns out my issue was completely unrelated, but regardless, here's the story of how I migrated from ezjail-managed jails (with multi-FIBs bound to VLANs) to jail.conf (with vnet, bridges, and vlans).
Let's start with the ezjail setup.

Just a warning, I use /31 subnets in this example. The IP address 10.10.10.0 is used as a host address. This is valid. Don't confuse it with a network address. Technically, it is the network address, but in a /31 subnet, it's also a host.

Here's the facts (pre-migration):

1. ZFS is used. This is applied in the ezjail.conf file to identify the pool and main dataset. Ezjail creates a basejail and newjail dataset automatically during the initial install command. When started , these are nullfs mounted into each jail where the jail uses its internal fstab file to mount multiple directories from the /basejail/ directory inside the jail. This allows ezjail to freebsd-update a single "basejail" which automatically applies to all jails. Good for saving space but adds a layer of complexity.

2. Each jail needed network isolation. FreeBSD now include multi-FIBs in the kernel by default so there's no need to recompile the kernel. I'm unsure at what version this happened and my searching has failed me. Need more coffee.
   

   • The jail's IP address is assigned to the VLAN interface when the jail is started.

   To accomplish the isolation, each jail needs it's own routing table (FIB). This needs to be enabled at boot. Add the following to /boot/loader.conf.local:

   net.fibs="16"
   net.add_addr_allfibs="0"
   

   • Number depends on what you need.
   • The allfibs line is needed for FreeBSD 12+. It keeps the host IP addresses from being applied to all FIBs (I think - but it was needed).

   Since all jails will be sharing the same physical interface to communicate with the router/firewall, we need to define a VLAN.
   Each jail will get the following lines in rc.conf.local:

   cloned_interfaces="vlan11"
   ifconfig_vlan11="vlan 11 vlandev ix0 description <jail name>"
   static_routes="vlan11 vlan11_gw"
   route_vlan11="-net 10.10.10.0/31 -interface vlan11 -fib 1"
   route_vlan11_gw="default 10.10.10.0  -fib 1"
   

   • Small caveat, the cloned_interfaces isn't duplicated, instead just add the vlan to the list: "vlan11 vlan12 vlan13..."

   After deploying the new jail with ezjail, the config file will include the IP address assignment:

   export jail_<jail name>_ip="vlan11|10.10.10.0/31"
   

   To ensure the jail uses the appropriate routing table, the following needs to be added to the jail's configuration file in /usr/local/etc/ezjail/<jail name>.conf:

   export jail_<jail name>_fib="1"
   

   At this point the host, after a reboot, will bring up the VLAN interface and the jail will use the new FIB1 routing table instead of the host's routing table when communicating across VLAN11.

3. Finally, cpuset is used for a few jails to keep them from hogging all the cycles:

   export jail_<jail name>_cpuset="2-3,19"
   

Now, here's the same end-state but with jail.conf and vnet.

1. Let's start with the datasets from ezjail. Since the jails are all "thin" and rely on the basejail nullfs mount, we need to inflate the jail's dataset. Start by stopping the jail and a quick snapshot to be safe:

   ezjail stop <jail name>
   zfs snapshot zroot/usr/jails/<jailname>@something_descriptive
   

   Now inflate the jail:

   fetch https://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/12.0-RELEASE/base.txz -o /tmp/base.txz
   fetch https://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/12.0-RELEASE/lib32.txz -o /tmp/lib32.txz
   tar -xkf /tmp/base.txz -C /usr/jails/<jail name>
   tar -xkf /tmp/lib32.txz -C /usr/jails/<jail name>
   

   • Use the tar "-k" option to keep it from overwriting existing files.

2. From what I understand vnet uses an ethernet pair or epair to pass traffic between the jail and the host. This feels more similar to the traditional virtualiztion NICs where you get a tap interface on the host that's linked to the VM. One side of the epair (a) stays on the host while the other side (b) is inside the jail. This allows the jail to have it's own network stack. No need for extra FIBs on the host as the routing table now lives in the jail.
   Here's how it should look (using VLAN11 as an example):
   

   • The IP address is assigned in the jail to the epairXb interface.

   We need to do a few things before the jail is started. First, we need to ensure the VLAN is configured and up. This happens in /etc/rc.conf.local just like the ezjail setup but a bit less:

   cloned_interfaces="vlan11"
   ifconfig_vlan11="vlan 11 vlandev ix0 description vlan11-<jail name>"
   

   Next, we need an epair and a bridge interface to bridge it to the VLAN interface. Since there's about 20 jails on this server, we need an automated way to create & destroy these. This is where jail.conf comes in. Not only will we use this to define our jails, it allows us to automate tasks before and after the jail is started. What's cool about jail.conf is its abilitiy to template your configuations and provide global settings that you can override as needed.

3. jail.conf
   Keeping the legacy ezjail configuration in mind, here's the new jail.conf config:

   # Global
   $domain = "example.net";
   $subdomain = "";
   $fast = "/storage2/jails/${name}${subdomain}.${domain}";
   exec.start = "/bin/sh /etc/rc";
   exec.stop = "/bin/sh /etc/rc.shutdown";
   exec.clean;
   mount.devfs;
   sysvshm="new";
   sysvsem="new";
   allow.raw_sockets = 0;
   allow.set_hostname = 0;
   allow.sysvipc = 0;
   enforce_statfs = "2";
   path = "/storage/jails/${name}${subdomain}.${domain}";
   host.hostname = "${name}${subdomain}.${domain}";
   
   # Networking
   $uplinkdev        = "vlan${vlan}";
   $epid             = "${ip}";
   $subnet           = "10.10.10.";
   $cidr             = "/31";
   $ipv4_addr        = "${subnet}${ip}${cidr}";
   vnet;
   vnet.interface    = "jail_${name}";
   
   # Start and Stop
   exec.prestart     = "ifconfig bridge${vlan} > /dev/null 2> /dev/null            || ( ifconfig bridge${vlan} create up description VLAN_${vlan} && ifconfig bridge${vlan} addm $uplinkdev )";
   exec.prestart    += "ifconfig epair${epid} create up description jail_${name}   || echo 'Skipped creating epair (exists?)'";
   exec.prestart    += "ifconfig bridge${vlan} addm epair${epid}a                  || echo 'Skipped adding bridge member (already member?)'";
   exec.created      = "ifconfig epair${epid}b name jail_${name}                   || echo 'Skipped renaming ifdev to jail_${name} (looks bad...)'";
   exec.created     += "echo \"hostname='${host.hostname}'\" > $path/etc/rc.conf.d/hostname";
   exec.start        = "ifconfig jail_${name} inet ${ipv4_addr}";
   exec.start       += "route -n add -inet default ${subnet}${gw}";
   exec.start       += "/bin/sh /etc/rc";
   exec.poststart    = "cpuset -j ${name} -l ${cpuset}";
   exec.stop         = "/bin/sh /etc/rc.shutdown";
   exec.poststop     = "ifconfig bridge${vlan} deletem epair${ip}a";
   exec.poststop    += "ifconfig epair${epid}a destroy";
   
   # Jails
   ...
   

   • Custom variables are defined with a leading $. Built in variables and commands lack the leading $.

   Let me explain the method to the madness:

   • Most jails have the same root domain but some have a subdomain. The $subdomain is set to nothing at first but then populated as needed in the individual jail sections below.
   • $fast is just the location of my SSD/NVMe datasets. If I set path = $fast in a jail's config, it overwrites the default path = which resides on spinning HDDs.
   • host.hostname allows me to define jails with short names, yet the jail gets the full FQDN when started. This makes my jail commands shorter. What? I'm lazy.

   Networking:
   Before I explain this section, let's use this jail config as reference:

   jail1 {
       $ip = 1;
       $gw = 0;
       $vlan = "11";
       path = $fast;
       $cpuset = "8-11";
   }
   

   • $uplinkdev is used when we bring up the bridge. We ensure the VLAN is a member of the bridge. This is populated by the jail's $vlan variable before being added to the bridge.
   • $epid is the epair ID. Since each jail gets a unique /31 subnet on the host, I can use the jail's IP address as the epair's ID.
   • $subnet and $cidr is defined so that the jail config is smaller. I only need to define the last octet with $ip. If a jail needs a custom subnet or mask, I can define it in the jail section.
   • $ipv4_addr allows me to define parts of the full inet IP as needed. This is then used to configure the interface in the jail. Yes, I could probably use this in the exec.start section directly.
   • vnet; defines that we want to use vnet in the jail. vnet.interface is used to rename the interface inside the jail. Without this, the jail would see something like epair30b as its interface. If a jail fails to start, the interfaces are usually left defined. This helps identify the leftover cruft and destroy them.
   • Ansible handles DNS configuation so that's excluded from my setup. If you need it, refer to the reddit link below.

   The start and stop section instructs the host or jail to perform tasks. Here, we need to do a few tasks:

   • Bring up the bridge
     • Rename the bridge to the VLAN number.
   • Add the vlan to the bridge
   • Create the epair
     • Rename the epair to the jail's IP (last octet).
   • Add the epair to the bridge
   • Define the jail's hostname
   • Define the jail's IP (after it starts)
   • Define the jail's default route (gateway) (after it starts)
   • Limit the jail's CPU use with cpuset
   • Define the teardown process when the jail stops.
     • Remove the epair from the bridge
     • Destroy the epair

Lastly, we need to disable ezjail and enable jail. Edit your /etc/rc.conf.local:

sysrc -f /etc/rc.conf.local -x ezjail_enable="YES"
sysrc -f /etc/rc.conf.local jail_enable="YES"
sysrc -f /etc/rc.conf.local jail_reverse_stop="YES"

• jail_reverse_stop is handy if you have jails that rely on another jail (i.e. a database jail). You'll need to define depend = "<db_jail_name>"; in jail.conf for the dependent jail. This will ensure that jail is started before, and shutdown after, the dependent jail.

At this point, we should be able to start the jails with the jail command:

jail -c <jail name>

Stopping jails is done with:

jail -r <jail name>

Restarting:

jail -rc <jail name>

Credit due to a few sources that helped during this adventure:

• https://www.cyberciti.biz/faq/how-to-configure-a-freebsd-jail-with-vnet-and-zfs/
• https://www.reddit.com/r/freebsd/comments/ahdbbq/howto_jails_freebsd_12_vnet_zfs/