My Backups Ran Green Every Night and Backed Up Nothing

For months, my nightly replication task ran green. Every morning it reported success. Then I audited it, and the replica it had been maintaining was 15 megabytes. The source was about 198 gigabytes.

The task wasn’t broken. It did exactly what it was configured to do, every night, and what it was configured to do was back up an empty container. That distinction, between a task succeeding and data being protected, is today’s post.

Some background for anyone who hasn’t read the lab writeup: all my hot data lives on an NVMe pool on a TrueNAS box, and every dataset that matters is supposed to snapshot on a schedule and replicate to a bulk HDD pool on the same machine. My MySQL data is in there. My application databases are there as well. When I finally sat down and audited the whole setup, “supposed to” turned out to be carrying a lot of weight.

Finding #1: the empty parent

ZFS datasets nest. A parent dataset like fast/apps is a real filesystem, but in my layout it’s mostly a container: the actual data lives in child datasets underneath it, one per application, each its own filesystem with its own snapshots. The parent itself holds almost nothing.

The semantics are what bit me. A snapshot of the parent covers only the parent. The difference is one flag:

zfs snapshot    fast/apps@nightly   # the container, ~nothing
zfs snapshot -r fast/apps@nightly   # the container and every child

Replication works the same way. A non-recursive send of the parent transfers the container and stops.

My periodic snapshot task on fast/apps was non-recursive. So was the replication task shipping those snapshots to the bulk pool. Two independent flags, both wrong since the day I created them. And nothing in the stack could have flagged it, because nothing in the stack was failing. The snapshot and the replication tasks did what they were told to do. Every night, a fresh copy of an empty container arrived on schedule, and every layer reported success.

The audit itself was one comparison:

Both flags flipped to recursive, and the first honest run moved the missing ~198G. The children that replica was supposed to protect, application databases included, had never been copied once in the task’s lifetime. I enjoyed the idea of periodic snapshots. I loved having them.

Finding #2: the task with the wrong name

The same audit found a replication task named backend-fast-to-bulk. Read that name and you’d say it backs up the backend host’s data. What it replicated was fast/workspaces, my dev workspaces dataset, 91.4G of it, into a replica dataset also named after the backend.

So the workspaces data was protected the whole time, by accident, under a name that says something else. Meanwhile the task list quietly taught me a misleading fact about my coverage. I believed the backend data was handled because a green task said so, and I believed workspaces was a gap.

The fix was renaming the task and the target dataset to say what they do. The detail that made it painless was that zfs rename on the target preserves the snapshot lineage, so the incremental replication chain continued unbroken under the corrected name. No re-seed, no gap in history, it just worked.

The coverage pass

After two tasks had misled me while working correctly, I stopped reading the task list and inventoried from the data side instead. I walked every dataset that mattered, checked its replica, and compared sizes. That pass added what had been silently missing, and the end state looks like this:

The 12 week row is deliberate. That long window is the deletion and ransomware undo button. If something destructive lands there, I don’t need last night’s copy, I need eleven weeks ago.

The new standing rule would have caught finding #1 on night one. After any replication task’s first run, compare the replica’s size to the source and list its children. It takes thirty seconds.

The dataset the GUI can’t see

One dataset resisted the built in tooling entirely. TrueNAS manages its own application runtime dataset, the one holding live state for every containerized app, and hides it from the GUI’s dataset pickers. The platform manages it, so the platform doesn’t offer it. Which means my favorite point and click snapshot and replication tasks can’t touch the one dataset where a corrupted app database would hurt most.

A small cron script now does what the tasks would have, and since the last post’s config was real, this one is too, verbatim:

#!/bin/sh
set -e
SRC=fast/ix-apps/app_mounts
DST=bulk/replicas/fast-appmounts
NOW=$(date +%Y%m%d-%H%M)
zfs snapshot -r ${SRC}@cron-${NOW}
BASE=""
if zfs list ${DST} >/dev/null 2>&1; then
  CAND=$(zfs list -t snapshot -o name -s creation -H ${DST} 2>/dev/null | grep @cron- | tail -1 | cut -d@ -f2)
  if [ -n "$CAND" ] && zfs list -t snapshot ${SRC}@${CAND} >/dev/null 2>&1; then
    BASE=$CAND
  fi
fi
if [ -n "$BASE" ]; then
  zfs send -R -i @${BASE} ${SRC}@cron-${NOW} | zfs recv -F ${DST}
else
  zfs send -R ${SRC}@cron-${NOW} | zfs recv -F ${DST}
fi
zfs list -t snapshot -o name -s creation -H ${SRC} | grep "^${SRC}@cron-" | head -n -14 | xargs -r -n1 zfs destroy -r

Recursive snapshot, incremental send when a base exists, full send when it doesn’t, and a prune at the end keeping the newest 14 snapshots on the source. First verified run: 11.0G, all nine application children present. (It’s #!/bin/sh with set -e, and if you read the MySQL post you know this platform and I have already had the sh versus bash conversation.)

Getting to that version took two attempts, and the bug in my first attempt is the one that will bite anyone writing incremental send scripts.

Trap one: stale snapshots on the target. The target dataset still held old snapshots from a manual one off copy, with no common ancestor to the new chain. An incremental send has nothing to bind to. The root dataset task hit the same wall, [EFAULT] No incremental base. The fix is to destroy the target’s stale state and do exactly one full send, then go incremental forever after. A replica’s snapshot history has to be a suffix of the source’s history, or incrementals can’t work.

Trap two: deriving the incremental base from the wrong side. My first script chose the incremental base from the source’s snapshot list to take the newest source snapshot that should already exist on the target, and send the delta from there. That works until the target’s state diverges from the source’s assumptions, which is precisely what trap one’s cleanup had just caused. The correct authority is the destination, and you can see the fix in the CAND block above. It asks the target for the newest @cron- snapshot it holds, then confirms that snapshot still exists on the source too, and only then uses it as the base. Either check failing means BASE stays empty and the script falls back to a full send instead of a doomed incremental. The only base an incremental send can bind to is one both sides hold. It hasn’t broken since.

What I changed my mind about

Before this audit I thought of backup verification as a restore testing problem which periodically proved you could get the data back. That’s still true, and still on the calendar. But every failure here would have been caught by something much cheaper, because in every case a component reported success at its own layer while the thing I cared about never happened. To catch this I can compare what I have versus what I should have. Replica size against source size, children against children, and a task list against the datasets themselves.

I’d already built this idea once without recognizing how general it was. In the v1 pipeline, I originally deleted empty transaction rows, and in v2 I deliberately keep zero success runs in run_log, because that row is how a silent failure becomes visible. This audit was the same principle applied to storage, with higher stakes.

One replica from the audit is still sitting at a suspicious 96K, unresolved. I haven’t figured it out yet, and I’m not calling it probably fine, since that’s what I assumed about the last two tasks.