Rollback is a pipeline feature, not a runbook

A runbook says: to roll back, redeploy the previous version. A rollback that works under pressure is the pipeline's ability to make that sentence true in under five minutes, at 3 a.m., driven by someone who did not ship the change. No schema, config value, or app store stands in the way.

Most pipelines cannot. The previous version is a tag someone can overwrite. The migration that shipped with the build dropped a column. The config change that caused the incident was made in a console and left no artifact. The person on call has never run the rollback command. None of those are runbook problems. They are properties of the pipeline, and each is fixable on a normal Tuesday.

This is what the pipeline has to provide, in the order it usually fails.

Immutable, addressable artifacts

The rollback target must be a thing with a name that cannot change meaning. A container image digest. A versioned binary in a registry with deletion disabled for the last N releases. A bundle hash. Not latest, not a branch, not "the previous deploy," which is a query, not a name.

Two properties follow. The artifact must still exist: registries with aggressive garbage collection have removed the previous image before the incident that needed it. Set retention to keep at least the last several production artifacts per service, and check that the rule is applied rather than assumed. And the deploy tool must accept the identifier directly, so that rollback is the deploy command with a different argument. A separate rollback path runs only during incidents, which makes it the least tested code in the repo.

The pipeline should also record, per environment, which digest is live and which was live before it. An incident that begins with "what was running before this" is already 10 minutes long.

Roll forward or roll back, and when each is honest

Roll back: put the previous artifact into production. Roll forward: ship a new artifact with a fix. Both are legitimate. The dishonest version of each is common.

Roll back is honest when the previous artifact can serve current traffic against current state. No schema change it cannot read, no config it does not understand, no message format from a newer producer it will reject. When those hold, rollback is a traffic shift or a redeploy of a known artifact, takes minutes, and requires no understanding of the bug. That last point is the reason to prefer it during an incident: rollback is the only mitigation that does not need a diagnosis first.

Roll forward is honest when the previous artifact cannot run (the data has moved on, the dependency it needs is gone). It is also honest when the fix is a one-line revert that clears the full pipeline faster than the rollback path would. It is dishonest when it is chosen because the team never built a working rollback and named the gap a philosophy. "We always roll forward" is a fine policy for a team whose forward pipeline is 8 minutes end to end and whose changes are small. Otherwise it is a bet that every incident will wait while the fix is found, written, reviewed, built, and rolled out.

A pipeline that supports both lets the on-call choose in the moment. The decision rule is short: if the previous artifact is compatible with the current state, roll back first and diagnose second; otherwise roll forward with a revert, through the pipeline, not around it.

Migrations that ride the pipeline without blocking it

The single most common thing that makes rollback impossible is a database migration bundled with the code that needs it. Ship version N+1, which renames a column; roll back to N, which reads the old name; N is now broken against the schema N+1 left behind.

The fix is a discipline, not a tool: expand/contract, written up as ParallelChange on martinfowler.com. Every schema change is split into an additive phase and a removal phase, with a code release in between that works against both shapes.

Expand: add the new column, table, or index. Remove and rename nothing. Deploy. Every running version of the code can still read and write.

Migrate: ship code that writes both old and new, or reads new with a fallback to old. Backfill the new shape from the old. This code works whether or not the schema is expanded, so it rolls back cleanly.

Contract: once every consumer reads the new shape and the backfill is verified, ship the removal of the old column as its own release.

The rule for the pipeline: a release may contain expand migrations with code, or a contract migration alone, never a change whose rollback requires reversing a schema operation. Migrations run as a pipeline stage before the code rollout, and a migration's own gate is a dry-run against a production-shaped copy that reports lock duration and row count. A migration that holds a table lock for 40 seconds is a deploy-caused incident under a different name.

One consequence teams miss: this makes down-migrations mostly unnecessary. If every schema change is additive at release time, rolling back the code never touches the schema. The down-migration that undoes the expand can wait for a calm afternoon, if it is needed at all.

Sponsored:

Config and secrets are deploys

A config change made through a console has no artifact, no diff, no previous version, and no rollout tiers. It is deployed to 100% instantly, and its rollback is someone remembering what the value was.

Treat every config change as a deploy: versioned in the same place as the code or in a config repository with the same review, shipped by the same pipeline through the same tiers, recorded in the same deploy log with a before-and-after. A feature flag flip is a config change and gets the same treatment. That is why flag systems with an audit log and a "revert to previous state" control are worth what they cost.

Secrets rotate through the pipeline the same way, with one added rule: the old secret stays valid for a window after the new one ships, long enough to roll back the code that consumes it. Rotating a credential and deploying the code that uses the new one as a single step means the rollback target holds a dead credential.

Once a team has made its code deploys safe, the config path is where the incidents move. The code pipeline got the gates. The config path did not, and it became the exception that is now the risk.

What cannot roll back in seconds

Some things have no traffic weight to turn down. Release engineering for them does not pretend otherwise; it moves the reversibility somewhere else.

Mobile apps. Once a build reaches devices, you do not control when users update, and both app stores treat a rollback as a new submission with its own review. What exists instead: the stores' own staged rollouts (Google Play's percentage rollouts, the App Store's phased release over a week), which let you halt distribution before most users have the build. Server-side kill switches, so the risky behavior in a shipped client can be turned off remotely. And a hard rule that the server stays compatible with the last several client versions, because the old client is the rollback and it will be in the field for months.

Firmware and devices. The reversal mechanism is dual partitions: write the new image to the inactive slot, boot it, and if it fails a health check within a window, the bootloader falls back to the previous slot. Android's A/B seamless updates are the widely known public example. Fleet rollout is staged by device cohort with a bake measured in days, not minutes, because the failure signal is slow to arrive.

Data backfills. A job that has rewritten a billion rows does not roll back; it rolls forward with another job. The engineering is in the shape of the job: idempotent so it can be rerun, checkpointed so it can be stopped and resumed, run against a sample first with the result verified, and, where the data matters, written to a new location with a cutover rather than in place.

Anything external: an email that was sent, a webhook that was delivered, a payment that was captured. The pipeline cannot reverse these, so it gates them. Outbound side effects live behind their own flag and their own canary, separate from the code rollout, and the release of a change that touches them is the slowest one in the ladder.

Rollback as a drill

A rollback that has only ever run during incidents has been tested under the worst conditions and at no other time. The way out is to run it under good conditions on a schedule.

The minimal drill: once a month, per service, roll production back to the previous artifact during business hours, with the on-call driving and the tooling doing what it would do at 3 a.m. Measure the time from decision to all traffic on the previous artifact. Record it. Roll forward again.

The drill finds what the runbook does not: the artifact that was garbage-collected, the migration that left the previous version unable to boot, the config value that was changed in the console and not in the repo, the permission the on-call does not have, the command that needs a flag nobody remembers. Each of those is a pipeline fix, and each is cheaper to make at 2 p.m.

Google's public writing on its disaster recovery testing program, DiRT, makes the broader case: the exercises exist to find the things a system's owners believe work and have never run. A rollback drill is that program at the size of one service.

A useful extension is to make the drill a pipeline stage. A weekly job picks a low-risk service, rolls it back, verifies, rolls forward, and posts the timing. When the timing exceeds the target it opens a ticket. Rollback time becomes a number the team watches rather than one it discovers.

Rollback is the only mitigation that does not need a diagnosis first. That is worth an afternoon a month.

What a deploy-caused incident should leave behind

The output of a deploy-caused incident review is not a checklist item ("verify staging before deploying") and not a training ("engineers should be more careful"). Those are ways of asking humans to run a gate manually, forever. The output is a change to the pipeline that makes the class of failure mechanically impossible or mechanically caught.

The translation, by failure class:

A build that was fine in staging and broke at scale: the rollout ladder had no tier that saw the failure, or the tier had no check on the signal that showed it. Add the check to the tier where the signal exists, with the threshold that would have caught this one.

A migration that blocked rollback: the pipeline accepted a release containing a destructive schema change alongside code. Add a migration linter that rejects drops, renames, and type narrowing in a release that also ships code, and requires them to ship alone.

A config change made outside the pipeline: the console still allows it. Remove write access from the console, route changes through the repo, and alert on drift between declared and live config.

A rollback that took 40 minutes: whichever step took the time becomes the drill's target, and the drill's threshold becomes an alert.

A change nobody could attribute: the artifact had no provenance, or the deploy log did not record who promoted what. Add the record.

The test for a good action item: can the pipeline enforce it without anyone remembering. If the answer is no, the review is not finished. John Allspaw's writing on blameless postmortems makes the same point from the other direction: the review's job is to understand how the outcome made sense to the people in it at the time. "Be more careful" is what a review says when it stopped early.

Can we roll this back in under five minutes

Run each row against a service you own. Every entry in the right-hand column is a pipeline task, and most are an afternoon.

On Monday

Pick the service that would hurt most and run the table. Fix nothing yet; fill in the column. Then do one thing: roll it back, during the day, on purpose, and time it. The number you get, and the step that consumed most of it, is the first pipeline change. The rest of the table is the backlog, ordered by which failing row would have cost the most in the last incident you remember.