FreeBSD encrypted root on ZFS
Preface
In 2023, I still run my own mailserver. Yes, because I like to keep control of (at least part) my own digital life, and I enjoy having multiple domain names on which I have stuff on. However, I was paying 30€/month to AWS to get in exchange 2 cores, 2GiBs of RAM and 40G of disk, barely sufficient to run IMAP+SMTP+MySQL+Clamd, let alone any form of spam protection or full-text search on email bodies.
So, I was paying a lot of money to run a shitty service, and I even though about shutting everything off and move my mail and my web sites onto some form of fully hosted service.
I still want to do it
Say what, to host four domains with just some email redirects plus the web sites I run, I would have spent more I was paying to also cripple me to some service vendor and their politics.
So, I wanted to run FreeBSD and I started scouting on the ISPs page until I decided to review Hetzner and netcup, that both offer aggressive pricing and a old fashioned VPS and little more.
Settling on a vendor
Eventually, I settled on a netcup VPS 1000 that gives me, for 1/3 of the price I was paying to AWS, 4 times the resources: 6 cores, 8GiB of RAM, 160GiB of RAID10 SSD and an uncrippled, completely totally free FreeBSD installation.
However, the base image that Netcup provides has some limitations:
- It runs on UFS
- It is lacking a swap partition
- It has no encryption
Making a plan
As I was already into the configuration stage and I didn’t want to restart from scratch (this is an old-fashioned server, manually managed, no automation) I decided to:
- Spin up temporary servers on hetzner to experiment
- Peruse for the incantation required to have a full disk encryption bootable machine
- Copy over the / from the netcup server to hetzner and see whether it boots
- Rinse and repeat
- Once the incantation is stable:
- Boot a hetzner target server to temporarily hold all the data
- Reboot the netcup source server from a CD so to rsync over all the data to hetzner
- Scratch the netcup server disk and recreate all the partitions and filesystems the way I like
- Rsync all data back from hetzner to netcup and reboot
Executing it
Turns out, it actually works. I started using the FreeBSD installation CD, to then realise I didn’t need the installer at all, because I already had a live system I was migrating, so I ended up using mfsbsd to both spin up the target server, and as well to boot the source server when it was time to copy everything back and forth.
Starting from this freebsd forum thread and this wiki page for zfs boot I ended up cooking the following incantation:
Reboot from ramdisk and copy over the data to the temp server
This configures the network, updates rsync to the latest version, mounts the current filesystem in /mnt and rsyncs everything over to a temporary storage location
ifconfig vtnet0 inet6 2a03:4000:2:33c::42 prefixlen 64
route -6 add default fe80::1%vtnet0
echo 'nameserver 2a03:4000:0:1::e1e6' > /etc/resolv.conf
pkg install rsync
pkg upgrade libiconv
mount /dev/vtbd0p2 /mnt
cd /mnt
rsync --archive --recursive --times --executability --hard-links \
--links --perms --compress --exclude .sujournal --exclude .swapfile \
--exclude .snap --exclude 'dev/*' --exclude 'srv/www/*/dev/*' \
. root@m17.openssl.it:/mnt
Create the partitions
Here we create a boot partition holding the gptboot executable, whose
responsibility is to load and execute the freebsd loader from the clear
text /boot partition.
Then we create a swap partition and eventually a zfs partition that
will contain our ZFS pool.
gpart destroy -F vtbd0
gpart create -s GPT vtbd0
gpart add -s 472 -t freebsd-boot vtbd0
gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 vtbd0
gpart add -s 1G -t freebsd-ufs -l boot vtbd0
gpart set -a bootme -i 2 vtbd0
gpart add -s 2G -t freebsd-swap -l swap vtbd0
gpart add -t freebsd-zfs -l root vtbd0
Create /boot and the encrypted root device
Here we create a UFS filesystem for the unencrypted /boot partition
that’ll hold the kernel and loader, and part of the encryption key used
to encrypt the root. That key alone is not sufficient to gain access to
the filesystem, as also an additional passphrase is needed.
newfs -O 2 -U -m 8 -o space /dev/vtbd0p2
mkdir /tmp/ufsboot
mount /dev/vtbd0p2 /tmp/ufsboot
mkdir -p /tmp/ufsboot/boot/geli
dd if=/dev/random of=/tmp/ufsboot/boot/geli/vtbd0p4.key bs=64 count=1
geli init -e AES-XTS -l 256 -s 4096 -bd -K /tmp/ufsboot/boot/geli/vtbd0p4.key /dev/vtbd0p4
cp /var/backups/vtbd0p4.eli /tmp/ufsboot/boot/geli
geli attach -k /tmp/ufsboot/boot/geli/vtbd0p4.key /dev/vtbd0p4
Create ZFS pool
This is my layout, that I mostly use to limit executability of paths that should not be executable, and also for ease of snapshotting separate parts of the filesystem that need different retention strategies
zpool create -R /mnt -O canmount=off -O mountpoint=none -O atime=off -O compression=lz4 tank /dev/vtbd0p4.eli
zfs create -o mountpoint=/ tank/ROOT
zfs create -o mountpoint=/tmp -o exec=off -o setuid=off tank/tmp
zfs create -o canmount=off -o mountpoint=/usr tank/usr
zfs create -o setuid=off tank/usr/ports
zfs create -o canmount=off -o mountpoint=/var tank/var
zfs create -o exec=off -o setuid=off tank/var/log
zfs create -o atime=on -o exec=off -o setuid=off tank/var/spool
zfs create -o exec=off -o setuid=off tank/var/tmp
zfs create -o canmount=off -o mountpoint=/srv tank/srv
zfs create -o exec=off -o setuid=off tank/srv/mail
zfs create -o exec=off -o setuid=off tank/srv/www
Eventually, mount the unencrypted UFS boot partition below the ZFS fs hierarchy,
umount /dev/vtbd0p2
mkdir /mnt/ufsboot
mount /dev/vtbd0p2 /mnt/ufsboot
Copy everything back!
Now it’s time to get back the stuff from the temporary location it was placed to, and write it onto the new shiny ZFS pool on the GELI-encrypted root:
rsync --archive --recursive --times --executability --hard-links \
--links --perms --compress root@m17.openssl.it:/mnt/ /mnt
mv /mnt/boot/* /mnt/ufsboot/boot
rm -rf /mnt/boot
ln -s ufsboot/boot /mnt
We use a symlink to point /boot to /ufsboot/boot, so the system will behave
as if /boot was a normal directory in /. It’s required to keep a /boot
subdir in the boot partition because plenty of loader code depends on
hardcoded /boot paths.
What’s left
/etc/fstab, with encrypted swap of course:
/dev/vtbd0p2 /ufsboot ufs rw 0 1
/dev/vtbd0p3.eli none swap sw,ealgo=AES-XTS,keylen=128,sectorsize=4096 0 0
/boot/loader.conf.d/geli.conf:
geom_eli_load="YES"
geli_vtbd0p4_keyfile0_load="YES"
geli_vtbd0p4_keyfile0_type="vtbd0p4:geli_keyfile0"
geli_vtbd0p4_keyfile0_name="/boot/geli/vtbd0p4.key"
zfs_load="YES"
vfs.root.mountfrom="zfs:tank/ROOT"
/etc/rc.conf:
zfs_enable="YES"
Did it blend?
Yes of course it did! And it’s happily working since :-)
03:44:10 root@m42:/srv/www/sindro.me/staging
# uname -a
FreeBSD m42.openssl.it 13.2-RELEASE-p2 FreeBSD 13.2-RELEASE-p2 GENERIC amd64
03:44:13 root@m42:/srv/www/sindro.me/staging
# df -hT
Filesystem Type Size Used Avail Capacity Mounted on
tank/ROOT zfs 140G 6.7G 134G 5% /
devfs devfs 1.0K 1.0K 0B 100% /dev
/dev/vtbd0p2 ufs 992M 189M 723M 21% /ufsboot
tank/var/spool zfs 134G 1.1M 134G 0% /var/spool
tank/tmp zfs 134G 220K 134G 0% /tmp
tank/srv/mail zfs 138G 4.8G 134G 3% /srv/mail
tank/srv/www zfs 136G 2.1G 134G 2% /srv/www
tank/var/log zfs 134G 13M 134G 0% /var/log
tank/var/tmp zfs 134G 224K 134G 0% /var/tmp
tank/usr/ports zfs 136G 2.6G 134G 2% /usr/ports
/dev nullfs 1.0K 1.0K 0B 100% /srv/www/admin.openssl.it/dev
/dev nullfs 1.0K 1.0K 0B 100% /srv/www/mail.openssl.it/dev
/dev nullfs 1.0K 1.0K 0B 100% /srv/www/nhaima.org/dev
/dev nullfs 1.0K 1.0K 0B 100% /srv/www/spadaspa.it/dev
tank/usr/src zfs 134G 773M 134G 1% /usr/src
tank/usr/obj zfs 134G 96K 134G 0% /usr/obj