Backfilling Two Years of Logs: Enterprise-Grade Observability on a Raspberry Pi

Wed, 08 Apr 2026 00:00:00 +0000

I have a FreeBSD server called m42 that’s been running for years. It handles email (Postfix + Dovecot + Rspamd), web (nginx with a dozen vhosts), firewall (pf), and all the usual suspects. It generates thousands of log entries per day across four distinct formats: BSD syslog, fail2ban, pf packet filter, and nginx access/error logs.

I even wrote pfasciilogd back in 2023 to convert pf’s binary logs into ASCII text so fail2ban could parse them — a foundational piece that now feeds structured firewall telemetry into the whole pipeline.

I also have two years of monthly backups sitting in restic snapshots. That’s roughly 25 million log lines, just… sitting there. A goldmine of security telemetry, traffic patterns, and attack data — completely unindexed and unsearchable.

I built a full observability stack on a Raspberry Pi 5 at home — VictoriaLogs for storage, Telegraf for processing, Grafana for visualization — and then I backfilled every single one of those 25 million entries through the exact same pipeline that processes live data. With full enrichment: GeoIP geolocation, ASN identification, and reverse DNS resolution for every IP address.

This is enterprise-grade log management. Running on a $80 single-board computer. In my living room.

Why backfills are hard (and usually skipped)

Let’s be honest: nobody does backfills. They’re the broccoli of operations work. You know you should, but the effort-to-reward ratio feels terrible.

The problem is that a backfill isn’t just “load old data.” Your pipeline has evolved. The parsing rules you wrote six months ago don’t match today’s processors. The enrichment you added last week doesn’t exist in your old backfill scripts. You end up with two classes of data: rich live entries and impoverished historical ones.

Observability on Marcello Barnaba

Backfilling Two Years of Logs: Enterprise-Grade Observability on a Raspberry Pi

Why backfills are hard (and usually skipped)