Allan Jude, or as I like to refer to him as the "ZFS whisperer", has provided weekly tidbits about the proper deployment of ZFS on FreeBSD. He even co-authored a book about it. Dude knows his stuff.

By not adhering to his teachings, I discovered (the hard way) why you should create and label your GPT partitions...

I have a Norco RPC-4220 4U rackmount chassis, housing 20 HDDs and 2 SSDs. The HDDs are all mounted in hot-swap sleds, but for reasons that will bore you, they're not all installed.

A few weeks ago, I discovered I needed a couple. A "RAID10" to be exact, for some bhyve VMs. So, I installed 5x old 500GB HDDs. I usually encrypt all disks just so that when I need to retire a disk, I don't have to worry about trying to delete the data.

# gpart process performed for each disk
gpart create -s gpt da12
gpart add -t freebsd-zfs -a 4k -b 1M da12

# geli process performed for each disk
geli init -s 4096 -e AES-XTS -l 256 -K "/root/geli.key" "/dev/da12p1"
geli attach -k /root/geli.key /dev/da12p1

zpool create pool mirror da12p1.eli da13p1.eli mirror da14p1.eli da15p1.eli
zpool add pool spare da16p1.eli

I completed the bhyve install, deployed a guest system and was happy with the results. A few days later, I installed a few more disks to use as a backup destination for Bareos. This time I installed 2x 2TB HDDs in the fist two slots of the chassis. I performed roughly the same procedure as above but didn't create partitions and only created a single mirror (RAID1) with no spare.

zpool create pool mirror da20.eli da21.eli

In an embarrassing attempt to troubleshoot some networking issues with a handful of jails on the same server, I rebooted the server. This is where Allan's teachings caught up to me.

As I said above, I encrypt my disks. Therefore, when I boot the system, I have to decrypt the Geli volumes and import the pool.

devices=(da12p1 da13p1 da14p1 da15p1 da16p1)

read -s -p 'Decryption password: ' pass
for name in "${devices[@]}"; do
    echo "Decrypting: $name"
		echo -n "$pass" | geli attach -j - -k /root/geli.key "/dev/$name" || exit 1

sleep 2

echo "Importing pool: $zpool"
zpool import "$zpool"
zpool status

Upon running this script after the system rebooted, I was met with errors. You see, when I first built the RAID10, the system assigned /dev/da12-16 to my HDDs. When I installed the additional two disks, they got /dev/20-21. Upon rebooting, FreeBSD reassigned the disks based on their position in the chassis.

The Problem

  1. On the RAID1, I didn't use partitions.
  2. On the RAID10, I used partitions, but I didn't label the partitions.

The Fix

Since the RAID1 was new and didn't contain any data, I started over by destroying the zvol zpool destroy [pool name] and creating labeled GPT partitions.

As for the RAID10, it already contained data so destroying it was not an option, however, I had the luxury of stopping the VMs, offlineing the pool, and unmounting the decypted volume.

zpool export pool
# geli command performed for each disk
geli detach /dev/da12p1.eli

GPT labels can be used to identify disks in ZFS. All I needed to do was modify the partitions and add labels! Luckily, gpart modify came to the rescue.
Repeating the following task for each disk, I polled for a serial number and added it as the label to the partition:

camcontrol identify da12 | grep 'serial number'
gpart modify -i1 -l ASDFGH1234 da12

Next, I modified and ran the decryption script.

devices=(ASDFGH1234 ASDFGH1235 ASDFGH1236 ASDFGH1237 ASDFGH1238)

read -s -p 'Decryption password: ' pass
for name in "${devices[@]}"; do
    echo "Decrypting: $name"
	echo -n "$pass" | geli attach -j - -k /root/geli.key "/dev/gpt/$name" || exit 1

sleep 2

echo "Importing pool: $zpool"
zpool import "$zpool"
zpool status

Zpool status now shows the gpt label and the zvol is online!

NAME                       STATE     READ WRITE CKSUM
pool                       ONLINE       0     0     0
  mirror-0                 ONLINE       0     0     0
    gpt/ASDFGH1234         ONLINE       0     0     0
    gpt/ASDFGH1235         ONLINE       0     0     0
  mirror-1                 ONLINE       0     0     0
    gpt/ASDFGH1236         ONLINE       0     0     0
    gpt/ASDFGH1237         ONLINE       0     0     0
	  gpt/ASDFGH1238  AVAIL