Allan Jude, or as I like to refer to him as the "ZFS whisperer", has provided weekly tidbits about the proper deployment of ZFS on FreeBSD. He even co-authored a book about it. Dude knows his stuff.
By not adhering to his teachings, I discovered (the hard way) why you should create and label your GPT partitions...
I have a Norco RPC-4220 4U rackmount chassis, housing 20 HDDs and 2 SSDs. The HDDs are all mounted in hot-swap sleds, but for reasons that will bore you, they're not all installed.
A few weeks ago, I discovered I needed a couple. A "RAID10" to be exact, for some bhyve VMs. So, I installed 5x old 500GB HDDs. I usually encrypt all disks just so that when I need to retire a disk, I don't have to worry about trying to delete the data.
# gpart process performed for each disk
gpart create -s gpt da12
gpart add -t freebsd-zfs -a 4k -b 1M da12
# geli process performed for each disk
geli init -s 4096 -e AES-XTS -l 256 -K "/root/geli.key" "/dev/da12p1"
geli attach -k /root/geli.key /dev/da12p1
zpool create pool mirror da12p1.eli da13p1.eli mirror da14p1.eli da15p1.eli
zpool add pool spare da16p1.eli
I completed the bhyve install, deployed a guest system and was happy with the results. A few days later, I installed a few more disks to use as a backup destination for Bareos. This time I installed 2x 2TB HDDs in the fist two slots of the chassis. I performed roughly the same procedure as above but didn't create partitions and only created a single mirror (RAID1) with no spare.
zpool create pool mirror da20.eli da21.eli
In an embarrassing attempt to troubleshoot some networking issues with a handful of jails on the same server, I rebooted the server. This is where Allan's teachings caught up to me.
As I said above, I encrypt my disks. Therefore, when I boot the system, I have to decrypt the Geli volumes and import the pool.
zpool='pool'
devices=(da12p1 da13p1 da14p1 da15p1 da16p1)
read -s -p 'Decryption password: ' pass
echo
for name in "${devices[@]}"; do
echo "Decrypting: $name"
echo -n "$pass" | geli attach -j - -k /root/geli.key "/dev/$name" || exit 1
done
sleep 2
echo "Importing pool: $zpool"
zpool import "$zpool"
zpool status
Upon running this script after the system rebooted, I was met with errors. You see, when I first built the RAID10, the system assigned /dev/da12-16 to my HDDs. When I installed the additional two disks, they got /dev/20-21. Upon rebooting, FreeBSD reassigned the disks based on their position in the chassis.
The Problem
- On the RAID1, I didn't use partitions.
- On the RAID10, I used partitions, but I didn't label the partitions.
The Fix
Since the RAID1 was new and didn't contain any data, I started over by destroying the zvol zpool destroy [pool name] and creating labeled GPT partitions.
As for the RAID10, it already contained data so destroying it was not an option, however, I had the luxury of stopping the VMs, offlineing the pool, and unmounting the decypted volume.
zpool export pool
# geli command performed for each disk
geli detach /dev/da12p1.eli
GPT labels can be used to identify disks in ZFS. All I needed to do was modify the partitions and add labels! Luckily, gpart modify came to the rescue.
Repeating the following task for each disk, I polled for a serial number and added it as the label to the partition:
camcontrol identify da12 | grep 'serial number'
gpart modify -i1 -l ASDFGH1234 da12
Next, I modified and ran the decryption script.
...
devices=(ASDFGH1234 ASDFGH1235 ASDFGH1236 ASDFGH1237 ASDFGH1238)
read -s -p 'Decryption password: ' pass
echo
for name in "${devices[@]}"; do
echo "Decrypting: $name"
echo -n "$pass" | geli attach -j - -k /root/geli.key "/dev/gpt/$name" || exit 1
done
sleep 2
echo "Importing pool: $zpool"
zpool import "$zpool"
zpool status
Zpool status now shows the gpt label and the zvol is online!
NAME STATE READ WRITE CKSUM
pool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
gpt/ASDFGH1234 ONLINE 0 0 0
gpt/ASDFGH1235 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
gpt/ASDFGH1236 ONLINE 0 0 0
gpt/ASDFGH1237 ONLINE 0 0 0
spares
gpt/ASDFGH1238 AVAIL