Preface

So I started running home assistant at home on a raspberry PI 5 machine and I just installed HAOS on an SD. I then started growing deeply uncomfortable on storing credentials in the HA filesystem in clear text (any obfuscation is not enough).

Considering configuring an encrypted root with HAOS is simply not possible without forking it, and also considering that dedicating a RPI5 entirely to HAOS is a waste of resources, I decided to add an SSD to the Pi, boot it with raspbian and then run HAOS inside a VM.

This way, I can have an encrypted root on the main host, thus encrypting the entire HAOS VM.

Furthermore I can now snapshot the entire HAOS VM and I have much more flexibility in managing it. Last but not least, I can also use the remaining RPI CPU and RAM for something else.

Requirements

First, a big thank you to this post that gave me the initial pointers on how to set this up. But that 2021 post is now slightly outdated, and many steps are no longer necessary.

• An RPI5 with debian 13 or newer
• A decent and reliable USB stick that can be fully erased
• A keyboard and monitor directly connected to the RPI

Overview

The overall idea is:

• We prepare an initrd that contains the resize2fs tool that allows shrinking and enlarging ext4 filesystems
• We configure the system to boot off an encrypted root that does not exist yet, thus forcing the system to fall into the initrd.
• While in initrd, we shrink the root filesystem to the smallest possible size
• We copy the root filesystem from the cleartext device over to the USB stick
• We create the encrypted device using LUKS
• We copy back the root filesystem from the USB stick back to the encrypted device
• We extend the root filesystem to the maximum size
• We configure SSH in the initrd so that you can unlock it even after you’ve deployed your raspberry pi in a location without a keyboard or screen.

Ready? Set, go!

I presented a talk at All Systems Go 2025, the foundational Linux userspace conference. The conference is organised mostly by the systemd team, and it’s a yearly meeting for all people working on Linux systems software.

This year’s theme has mostly been “containers, containers, containers” with many new features in systemd to support containerisation and as well practical experiences from people working in the field on how they’re using systemd and collateral software to build container infrastructures.

I presented together with my colleague Serge Dubrouski our work in building an Operating System at Meta scale. We run an image-based operating system, but the company comes from two decades of updating the OS online, so we had to design a suitable migration strategy and set the foundation for the future.

We describe how we cut CentOS releases from upstream, the OSS tools we’ve built to create OS images, and the internal technology (MetalOS) that we came up with to build an OS that runs on millions of Linux servers.

You can also download the video for offline viewing.

Questions? Comments? Rant below! 🤣

TL;DR

FreeBSD: How to block port scanners from enumerating open ports on your server, by using fail2ban and an ASCII representation of pf logs.

Preface

I use fail2ban to keep away attackers and bots alike that attempt to scan my websites or brute force my mailboxes. Fail2ban works by scanning log files for specific patterns and keeping a count of matches per IP, and allows the systems administrator to define what to do when that count exceeds a defined threshold.

The patterns are indicative of malicious activity, such as attempting to guess a mailbox password, or attempt to scan a web site space for vulnerabilities.

The action to perform is most of the time to block the offending IP address via the machine firewall, but fail2ban supports any mechanism that you can conceive, as long as it can be enacted by a UNIX command.

PF and its logs

On my FreeBSD server I use the excellent pf packet filter to policy incoming traffic and to perform traffic normalization.

The PF logging mechanism is very UNIX-y, as it provides a virtual network interface (pflog0) onto which the initial bytes of packets blocked by a rule that has the log specifier are forwarded, so that real-time block logs can be inspected via a simple:

# tcpdump -eni pflog0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on pflog0, link-type PFLOG (OpenBSD pflog file), capture size 262144 bytes
01:48:13.748353 rule 1/0(match): block in on vtnet0: 121.224.77.46.41854 > 46.38.233.77.6379: Flags [S], seq 1929621329, win 29200, options [mss 1460,sackOK,TS val 840989709 ecr 0,nop,wscale 7], length 0
01:48:15.726215 rule 1/0(match): block in on vtnet0: 192.241.235.20.37422 > 46.38.233.77.5632: UDP, length 3
01:48:17.993439 rule 1/0(match): block in on vtnet0: 145.239.244.34.54154 > 46.38.233.77.1024: Flags [S], seq 3365362952, win 1024, length 0
^C
3 packets captured
3 packets received by filter
0 packets dropped by kernel

These logs can be saved by pflogd into a pcap format file in /var/log/pflog, that can be used for async troubleshooting and inspection, as well using tcpdump or anything that can parse pcap files (such as wireshark).

Preface

In 2023, I still run my own mailserver. Yes, because I like to keep control of (at least part) my own digital life, and I enjoy having multiple domain names on which I have stuff on. However, I was paying 30€/month to AWS to get in exchange 2 cores, 2GiBs of RAM and 40G of disk, barely sufficient to run IMAP+SMTP+MySQL+Clamd, let alone any form of spam protection or full-text search on email bodies.

So, I was paying a lot of money to run a shitty service, and I even though about shutting everything off and move my mail and my web sites onto some form of fully hosted service.

I still want to do it

Say what, to host four domains with just some email redirects plus the web sites I run, I would have spent more I was paying to also cripple me to some service vendor and their politics.

So, I wanted to run FreeBSD and I started scouting on the ISPs page until I decided to review Hetzner and netcup, that both offer aggressive pricing and a old fashioned VPS and little more.

Settling on a vendor

Eventually, I settled on a netcup VPS 1000 that gives me, for 1/3 of the price I was paying to AWS, 4 times the resources: 6 cores, 8GiB of RAM, 160GiB of RAID10 SSD and an uncrippled, completely totally free FreeBSD installation.

However, the base image that Netcup provides has some limitations:

• It runs on UFS
• It is lacking a swap partition
• It has no encryption

Making a plan

As I was already into the configuration stage and I didn’t want to restart from scratch (this is an old-fashioned server, manually managed, no automation) I decided to:

Il vero sistemista e’ un po’ come il meccanico di una volta, quello che se gli portavi la macchina per rifare la convergenza e quando arrivavi sentiva che il minimo non andava bene, ti faceva la convergenza, e giustamente la pagavi, ma poi ti sistemava anche il minimo e non ti chiedeva nulla, lo faceva perche’ non sopportava di sentire una macchina che non era a punto come si deve.

Era quello che da ogni minimo e impercettibile rumore indovinava subito qualsiasi problema, anche quello di cui il cliente non si era ancora accorto.

Era quello che dopo cena a casa con la famiglia, tornava in officina, dove potevi vedere le luci accese fino a notte tarda, perche’ stava lavorando al “suo” gioiello, una qualche macchina semi d’epoca recuperata chissa’ dove che con passione piano piano sistemava fino a farla tornare nuova.

Ecco, il sistemista e’ come quel meccanico, e le sue auto sono i server.

Fonte: Veteran Unix Admins

English version

The real sysadmin is like the old-fashioned car mechanic, the one you brought your car to adjust the wheels’ convergence and when you got into his garage he heard also your engline while idling didn’t have the right RPM. He then fixed the wheels’ convergence and you paid him for it, but he also fixed the engine idling RPM without asking you nothing - he did it because he couldn’t stand a car that was not set up properly.

He is the one that from every tiny and imperceptible noise immediately guessed every car problem, even those the customer did not yet realize.

In its own words:

Sources: 55179.13.c, 55471.c

Source code differences between two consecutive versions of the Security.framework, a MacOS/iOS component. The seemingly innocuous extra goto fail; is the cause of a severe security flaw in most Apple products.

Because I have been busy doing this:

Abstract: add SID APT source, configure APT Pinning to give squeeze packages priority over SID ones, rebuild the nodejs package under squeeze.

• Add SID APT source by creating /etc/apt/sources.list.d/sid.list (use your nearest mirror):

deb http://ftp.us.debian.org/debian/ sid main
deb-src http://ftp.us.debian.org/debian/ sid main

• Configure APT pinning by creating /etc/apt/preferences.d/sid:

Package: *
Pin: release a=unstable
Pin-Priority: 50

• Install the latest version of libv8 manually, libv8-3.8.9.20 at the time of writing this:

apt-get install libv8-3.8.9.20

• Download the nodejs package sources, dependencies and build them:

cd
apt-get source nodejs
apt-get build-dep nodejs
cd nodejs-*
debuild -nc -uc

• If you encounter build-dependency errors, you should try first to lower the dependency in debian/control, both in Build-Depends and in Depends and re-run debuild. If the build fails (e.g. with undefined reference to 'ev_run') the previous version is missing required functions. So, you must install the updated versions of the required dependencies (e.g. libev4) from sid, using apt-get install name=version e.g. libev4=1:4.11-1. I suggest this because you’ll have to manually update packages installed from sid, so the lesser, the best.

• Install the generated package

dpkg -i nodejs_*.deb nodejs-dev*.deb

• Profit :-)

So you have a Linux VM you use for development, because you want to mirror the production environment as closely as possible. You have many applications to deal with, they have to be running at the same time because they are nifty REST JSON web services.

You are very tired to remember which one you put on port 8081, and your configuration files slowly become a real mess. So you set up IP address aliases in for the network interface and decide to assign even host names – /etc/hosts is just fine – for each app.

Then, in such a setup, why would you still need to run them on ports higher than 1024? Wouldn’t be just great to type the application name in the browser address bar? Indeed it is, but it’s better to not run them as root, anyway.

The solution are Linux capabilities (see also here). The one that interests us is cap_net_bind_service: it gives a process the right to bind well-known ports (< 1024). If you use an interpreted language, of course you’ll have to add the capability to the interpreter itself. That’s why there’s development in the title of this article – you should not set this up on a production server, if you don’t know what you are doing.

One final quirk: if you happen to dlopen() shared objects that dynamically link towards libraries outside the canonical paths, you cannot load them via LD_LIBRARY_PATH (e.g. the SYBASE.sh) as it is ignored for setcap-ped processes. You should better move the library paths into an /etc/ld.so.conf.d snippet.

“If it is good, they stop making it”, the payoff printed on the conference necklaces, distributed to every participant, along with an über-l33t badge customized with our nickname and the key hash.

Being my first experience at an international security conf (I’ve only been to the ccc2k+7 camp), and being a ph outsider ‘cause I never participated to previous editions, the boot keynote held by FX, staffer and frontman, has been enlightening: “you ought to be here!”, he yelled while pointing at the stage, wearing a white shirt with the Phenoelit logo printed on both arms.

“This conference has never started on time”, he continued, “so there was no reason to do that for this last one”. the schedule is straightforward: party, the next days talks from 12.00PM to 7.30PM, then party, and the last days talks from 12.00PM to 5.30PM. definitely a setup well-playing with the available alcohol :-D.

Afterwards, another speaker informed us that the wi-fi access keys we received at the registration allows us to use a 6 APs/3 repeaters beast driven by an OpenBSD box – they want the audience to hack it because, well, “you are the Worst Case Scenario.” :-)

Then, the funny Hacker Hacker video was presented:

:-D

After a lousy and not so exciting first night (due to tiredness), we’ll wait and see what the next day would bring.

Sniffjoke – a sniffer-evasion toolkit

High capacity sniffers used in big cos and on border national gateways that collect user generated traffic on order to find possibly “criminal” patterns are today generally available for bandwidth to the 10Gbps, there will be soon appliances that’ll process streams of 100Gbps. Sniffjoke, by vecna and evilaliv3 is a tool that can inject into TCP connections outsider packets that will fool the intercepting sniffer but with no remarkable effect on the receiver. these packets for instance trick the sniffer into thinking that the connection has been reset even it is not true – by injecting a wrong-checksummed RST or a packet with a TTL less than 1 of the hop count – or try to consume its processing power by using known vendor-specific interpretations of the TCP RFC. Details: website, slides, wireshark thread.