What a shared Terraform state really costs

A command meant to delete a DNS record destroyed a SaaS company’s production infrastructure on a Friday afternoon. Three people spent the weekend rebuilding it.

What follows is not an account of an operator error. It is an analysis of what allowed an operator error to reach that far.

The system as it was

A single Terraform state described the entire infrastructure: DNS records, Kubernetes clusters, application interfaces, container images. Everything that existed lived in one file, stored in a shared storage account and applied from workstations.

There is nothing unusual about this setup. It is in fact the default trajectory: you declare one resource, then a second, and nobody ever explicitly decides to put everything together. It is simply what happens when nothing is decided.

What happened

An operation on a minor resource. An environment variable missing at execution time. The command applied to the entire state.

By the time anyone understood what was happening, production no longer existed.

The wrong conclusion

It is tempting to stop at “we should have been more careful”.

That is the conclusion that guarantees the incident will happen again, because it changes nothing in the system. It turns an architectural failure into a failure of vigilance, and vigilance is not a property you can reliably demand, least of all at four o’clock on a Friday, in a two-person team that has spent the week on something else.

A setup in which a moment of inattention destroys production is a broken setup. That holds whoever makes the mistake, and it holds on the days when nobody makes it.

The root cause

A single state means there is no limit on the blast radius.

This is the central point, and it deserves to be stated precisely: in this configuration, every command, whatever its intent and whatever resource it targets, runs with the technical ability to destroy everything. The gap between “I am deleting a DNS record” and “I am destroying the production clusters” is not protected by a barrier. It is protected by the command being well formed.

Three other gaps made the incident worse.

No separation between environments. Nothing technically distinguished what could be destroyed without consequence from what could not.

No systematic review of the plan before apply. Terraform announces what it is about to do; that still has to be a mandatory step rather than a good habit.

No backup of application workloads on Kubernetes. This is the gap that cost the most time over the weekend. Rebuilding infrastructure described in code is slow but mechanical. Reconstructing the application state that lived inside it, without a backup, is far less so.

What was fixed

Cluster backups were put in place straight away with Velero: Kubernetes resources and persistent volumes. It was the most costly gap and the quickest to close.

But a backup proves nothing until it has been restored. So the setup was put to the test: Kubernetes resources and persistent volumes restored from Velero backups into a test cluster, through to the applications actually restarting.

That test gives what no green report gives: the real time to restore service, the order in which services have to come back, and the list of what does not come back on its own. Between two tested restores, a backup is not a guarantee, it is a hypothesis.

Splitting the state is a heavier fix: it means reworking how the code is organized, not adding a tool.

What I do differently today

On the infrastructure I am building now, four decisions follow directly from this incident.

The state is split along two axes. Lifecycle first: what must never be destroyed lives in a separate state from what is meant to be. Rate of change second: resources modified every day do not share a state with those nobody touches. A command run within the scope of the DNS records no longer has the technical ability to reach the cluster.

The execution identity is dedicated and scoped. Terraform does not run under my account, but under an identity that holds only the permissions for what it manages: no billing rights, no identity administration rights. The audit logs distinguish what I did from what the tool did.

The state is protected against its own destruction. The storage that holds the state lives in a separate project from the one it describes, and carries an explicit deletion lock. The principle: a recovery mechanism must not live inside what it repairs.

No long-lived secrets. Authentication relies on short-lived tokens derived from an identity, rather than a key sitting on a disk.

What I take away from it

The blast radius of a command is an architectural decision, made long before the incident and usually without anyone noticing. Nobody decides to put everything in the same state; you simply fail to decide otherwise, which amounts to the same thing on the day it matters.

And a backup is only worth its last successful restore.