BackReel: restoring a Kubernetes cluster to any second

The problem

Backup tools restore only to the moments of their scheduled backups. History tools see every change, but restore nothing. And cloud providers cap how often a disk can be snapshotted, so their APIs cannot reach the second.

BackReel restores a Kubernetes cluster, its objects and its volumes, as it was at any second, with no scheduled backups.

Two rules

• No access to etcd. Everything goes through the Kubernetes API, so managed clusters stay within reach.
• Every restore states its real precision. A volume with no data is flagged before the restore, and a gap in the history is never shown as a quiet period.

Objects

A recorder follows every resource through the API: keyframes, the events between them, and the gaps in the history, flagged as such. history, at and diff read that history without a cluster. A namespace is restored as it was, after a dry run and a diff.

Secrets are encrypted with age before they are written, and are not recorded at all without a public key. The history goes to an S3-compatible bucket; with Object Lock and a default retention, deleting the cluster or the history cannot erase it, and the recorder warns when the bucket has neither.

Volumes

An agent written in Rust serves a ublk disk, in user space, and journals every write, with a consistent point at each flush. A write is acknowledged only once its record, and every record before it, is in the journal: whatever a crash leaves out of the journal was never acknowledged.

In Kubernetes, a CSI driver written in Go wraps the real one and stacks the agent on each volume. The journal, cut into segments, goes to the bucket, with a VolumeSnapshot as keyframe. restore-volume picks the latest keyframe that covers the requested instant and makes a new volume from it; an instant no keyframe covers is refused.

What is proven

• PostgreSQL under load, rebuilt at random instants: no row written after the instant, no corruption.
• The agent killed with -9 under load: a new agent takes the disk over in under a second, and no acknowledged write is lost.
• 510 restores at random instants, across 5 kills: not one acknowledged write lost.
• In Kubernetes, under pgbench, the agent killed twice and the node plugin restarted are pauses under a second: no failed transaction.
• A disk rebuilt from a snapshot taken under load and the segments in the bucket is the same, byte for byte, as the disk itself.
• Twelve instants drawn at random under pgbench, restored as new volumes: every acknowledged row there, none written after, every database sound.
• The node lost with writes never flushed: the restored volume is the disk, byte for byte, and not what the journal had shipped of those writes.

What it costs

Measured and published in the repository, with its method. Under PostgreSQL, the ublk disk without its journal costs 32% of the throughput; the journal takes another 46%: 1,992 transactions per second instead of 3,708, and a p99 latency of 5.0 ms instead of 2.1 ms. Two changes, a single sync at each flush and journal writes through io_uring, raised the share of throughput kept from 36% to 54%. These figures come from a local virtual machine, not a cloud cluster.

Where the project stands

The object recorder, the block journal outside Kubernetes, then inside Kubernetes with its CSI driver: those three phases are done. The demo on a real cloud cluster remains.