The Homelab: Hardware and Architecture

Every post on this blog runs on the same hardware: a rack in my basement that I built and operate as production. When something breaks down there it’s a real incident, and a few of those incidents have their own posts here.

The honest origin is a coworker. At my old job there was a network engineer named Kevin, the guy who knew everything about the company network because he’d built most of it. He was generous with his time and liked talking shop, and I’d walk away from those conversations having understood maybe a third of what he said. That kept happening, and eventually I had to admit the problem wasn’t that Kevin was deep. Networking was my weak spot. A gap like that doesn’t close by reading, and it shouldn’t close by experimenting on an employer’s infrastructure, so I built a place where the things I break are mine.

The same reasoning applied to more than networking. Managed cloud hides the integration work behind a console, and the integration work was the part I actually wanted to learn. Down here, if the CNI is misconfigured or storage doesn’t mount or the resolver dies on boot, there’s no support ticket to open. It’s on me to figure out why. The bulk of the build took about six months, and it’s been my production infrastructure since.

The rack

A 27U locking cabinet on casters in a basement corner, fed by two dedicated 20A circuits. Each circuit powers its own UPS and each UPS feeds its own PDU, so a tripped breaker or a dead UPS takes down half the rack instead of all of it.

The switch is the only box with dual power supplies, one leg on each UPS, because it’s the one thing that can’t lose power. Everything else picks a side.

The compute lives in SilverStone rackmount cases. Consumer hardware in real rack cases gets me modern CPUs and GPUs without the noise and power draw of retired enterprise gear off eBay. And most of it wasn’t bought for this. Aside from the backend server, the rack itself, the cases, and the firewall, everything here was built from parts pulled out of computers I already owned. That’s the actual reason the lab runs consumer parts instead of ECC and datacenter gear, and the non-ECC tradeoff has already cost me once.

Four machines, four roles.

Firewall. 2U. Ryzen 5 7600X, 32GB DDR5, pfSense on a mirrored ZFS boot pair. Routing, VLANs, VPN, IDS/IPS, DNS. It’s sized like a server because it does server work, deep packet inspection on every WAN packet and recursive DNS for the whole network. It’s supposed to be the most boring machine in the rack. It has more war stories than the other three combined.

Backend. 5U. Ryzen 9 9950X, 256GB of DDR5-6000, RTX 5090. Kubernetes control plane plus worker, and where the Spark driver lands. The 256GB covers the driver, a MySQL pod, and everything else that schedules here, with headroom left for mistakes.

Frontend. 4U. Ryzen 9 7900X, 128GB DDR5, RX 7900 XTX. The second worker, running the Spark executors and the other MySQL pod. Two physical nodes is deliberate. Pod scheduling, node affinity, and network policy don’t become real problems until a second machine exists.

NAS. 4U. i9-12900KS, 128GB DDR5, RTX 3090, running TrueNAS SCALE. All the persistent storage, every backup target, centralized logging, and GPU work for a few self-hosted services. It has one quirk that’s written in the runbook in bold. With memory fast-boot disabled, which is deliberate after a training failure, a cold boot spends about fifteen minutes retraining RAM before the box answers a single ping. The first time it happened I was convinced I’d killed it.

The network

The core is a Cisco Nexus 93180YC-FX, 48 SFP28 ports at 25G plus 100G uplinks, running NX-OS, deployed purely as a layer 2 switch. Routing between segments is the firewall’s job. The switch just moves packets at line rate and stays out of the way. It’s more switch than a basement needs, and that’s why I bought it.

The three lab hosts connect at 25GbE over SFP28 optics and OM4 fiber, with jumbo frames end to end on the lab segment. The firewall hangs off a 10G trunk carrying the VLANs. One of the two 10G ports on its NIC failed early with a link-flap condition, so the trunk lives permanently on the survivor.

Traffic is split into VLANs for the trusted home network, the lab, infrastructure management, and switch management, with the management segments isolated from everything else. Remote access is WireGuard. Nothing else is exposed to the internet.

The 25G fabric gets used. NFS-backed Kubernetes volumes and Spark shuffle traffic both cross it, and the NFS mounts run multiple parallel TCP connections per host to actually fill the pipe. There’s not much point tuning storage if the network under it can’t carry what the disks serve.

Storage

Five ZFS pools on the NAS, split by role instead of one big pool:

• a mirrored SSD boot pool
• a bulk HDD mirror for backups, replicas, and anything that values capacity over speed
• a small SSD mirror just for infrastructure config, the data you cry about losing
• an NVMe mirror for hot data, including the MySQL data the cluster mounts
• NVMe scratch for staging and transient work that nothing needs to survive

Everything hot gets scheduled snapshots, hourly on the busiest datasets and daily elsewhere, and replicates on-box to the bulk pool. Irreplaceable data also gets a long-retention snapshot window, weeks of history instead of days, which is my undo button for deletions and ransomware.

The backup design gets this much attention because it’s earned it, twice. Once when both members of the hot NVMe mirror hit the same firmware bug within hours of each other and the pool went from DEGRADED to SUSPENDED overnight. And once when an audit found a replication task that had been running green every night while copying an empty parent dataset, for months. Zero data lost either time, but only because there were layers. No single layer turned out to be trustworthy on its own.

The Kubernetes cluster

Two bare-metal nodes built with kubeadm, Calico with VXLAN for the CNI. No managed control plane, no cloud load balancers, no CSI driver written by someone else that I get to blame. On top of it run Spark via the Spark Operator, Airflow for orchestration, a MySQL StatefulSet with GTID-based primary/secondary replication, and an Apache Iceberg lakehouse on MinIO behind a Polaris REST catalog.

That stack runs the StockAlgo pipeline, 11 technical indicators across 13,000+ symbols, every hour. If you read the post about building that pipeline by hand in MySQL, this cluster is where version 2 lives.

The reason for self-hosting all of it is that Spark, Iceberg, Polaris, and MinIO sit exactly where Glue, Athena, and S3 would sit in a managed setup. Wiring them together myself made every integration seam explicit. When the catalog and the object store disagree, there’s no vendor to point at, so I have to actually understand the handshake.

Security

Deny-by-default firewall rules on every interface, with the broad pass rules removed. Suricata running inline as IPS on the WAN. DNS-level blocklisting at roughly 534,000 entries. An internal certificate authority on ECDSA P-384 issues real HTTPS certificates for every internal service, so nothing on this network talks plaintext and my devices trust the CA properly instead of clicking through warnings.

The posture mirrors what a small production environment should look like, and running it surfaces the same problems production teams deal with. The IDS has blocked my own legitimate traffic more than once, certificate lifecycles need actual management, and every lockdown decision picks a fight with usability.

Observability

Prometheus scrapes node_exporter on all four hosts plus SNMP from the firewall, Grafana sits on top, and every host forwards syslog to a centralized receiver on the NAS, where the archive lands on ZFS and rotates daily.

That archive has settled more than one argument with my own incident notes. Months into a recurring DNS failure, it’s what let me check my earlier theories against what had actually happened, timestamp by timestamp, and some of those theories didn’t survive. When my memory and the archive disagree, the archive wins.

When the power actually goes out

Power loss triggers UPS-driven staged shutdowns in dependency order. Compute nodes shut down first on a battery timer, storage after, and the firewall holds out until low-battery. The order matters because the shutdown path itself crosses the network, and a storage box that powers off before the hosts still writing to it turns a power blip into a recovery project.

The runtime math isn’t static either. GPU load on the backend meaningfully shortens how long its UPS lasts, which is the kind of thing you only find out by measuring, and worth re-measuring before anything power-hungry gets added.

Posts about this lab

• I Built Airflow, Spark, and Iceberg by Hand in MySQL: the pipeline that runs on this cluster

Deep dives on the rest, the resolver saga, the backup audit, the mirror failure, the watchdog stack, publish here weekly through the fall, and this list grows as they land.