Anatomy of a deploy pipeline: merged commit to live traffic

A deploy pipeline is a sequence of gates. Each gate exists to catch one class of bad change, and each one costs minutes. Teams that deploy dozens of times a day do not have fewer gates than teams that deploy weekly. They have gates that are cheaper, automated, and placed where they can see the failure they were built to see.

Eight stages, in order. For each: what it catches, what it costs, and the common way it is wired wrong. Then a decision table for the three forks every pipeline takes, and a way to draw your own and find the stage you only think you have.

The pipeline described here is a composite, not a specific company's. Where a public write-up shows the same shape, it is named.

Stage 1: the merge queue keeps main green

What it does: serializes merges so that every commit on main was tested against the exact main it landed on, not the main from when the pull request was opened.

Without a queue, two PRs that each pass CI against an older main can land back to back and break the build together. The break is nobody's fault, and it lands directly in front of the deploy. Every deploy behind it now waits for a fix or a revert, and the change that eventually ships is a batch of everything that queued up while it waited. Batches are how a deploy goes from one suspect to twelve.

What it catches: semantic conflicts. Two green PRs that are red together.

Cost: one CI run of latency per merge, spent in the queue rather than on the branch. Once the merge rate exceeds what serial testing can absorb, batched queues (test a group of PRs as one, bisect on failure) cut this to a fraction of a run per PR. The threshold is arithmetic, not folklore: when merges arrive faster than one CI run can serialize them, optimistic rebasing has already stopped working and the queue is mandatory. Uber's "Keeping Master Green at Scale" paper is the public account of what that looks like at monorepo scale.

Wired wrong: the queue runs a lighter test suite than the PR check to keep it fast. Then the queue catches nothing the PR check did not, and the team has paid the latency for a guarantee it did not buy. The queue's test set is the one that defines green; if that is too slow, the fix is in the test suite, not in the queue.

Stage 2: build once, promote the artifact

What it does: turns the merged commit into one immutable artifact (container image, binary, bundle) with a content-addressed identifier, and that same artifact travels through every environment to production.

What it catches: environment drift disguised as application bugs. If staging ran a different build than production, a staging pass told you about staging.

Cost: the build, once. Typically 3–15 minutes depending on caching. The cost you avoid is the one paid by the rebuild-per-environment pipeline: the same minutes again at every stage, plus the deploys that break because a dependency resolved differently on Tuesday than it did on Monday.

Wired wrong: the pipeline builds once, but the deploy step references a mutable tag (latest, staging, a branch name) instead of the digest. Two deploys of "the same tag" ten minutes apart can be different bytes. The artifact identity that reaches production must be a digest or a version that cannot be reassigned, and the deploy log must record which one went out.

Stage 3: provenance, or the artifact you can name

What it does: attaches to the artifact a record of what it is. Source commit, build inputs, the pipeline run that produced it, and a signature proving the build service produced it rather than a laptop.

What it catches: two things. During an incident, the on-call answers "what is running" in one lookup instead of an archaeology session. During a supply-chain event, whether the image in production came from your build or from somewhere else.

Cost: seconds at build time to attach attestations, and a policy check at deploy time that rejects unsigned or unattested artifacts. The SLSA framework describes the levels; most teams get most of the value from a hosted build service that emits signed provenance and a deploy step that refuses anything without it.

Wired wrong: provenance exists but nothing checks it. The signature is generated because a template did it, and the deploy step never verifies. An unchecked signature is a checksum with better marketing.

Stage 4: pre-production verification

What it does: deploys the artifact to an environment that is not production and runs the checks that need a running system. Integration tests, contract tests against real dependency versions, migration dry-runs, a smoke suite.

What it catches: the class of failure that unit tests cannot see. A wrong environment variable. A service that boots and then cannot reach its database. A migration that takes a table lock for 40 seconds.

Cost: 5–20 minutes, plus the ongoing cost of keeping the environment from drifting. This is the stage where teams over-invest. A staging environment that mirrors production perfectly does not exist; one that runs the same artifact with the same config shape and the same migration path gets most of the signal at a fraction of the upkeep.

Wired wrong: pre-prod becomes the QA gate. Someone has to check staging before production is allowed. Now the pipeline waits for a human between merge and the first production traffic, and the deploy batch grows while it waits. The human check is the least reproducible gate in the whole system. Pre-prod should run automated checks and either promote or stop. A human belongs later, at a different kind of decision.

Sponsored:

Stage 5: progressive rollout tiers

What it does: exposes the artifact to production traffic in increasing slices. A common ladder: one instance (the canary), then a small percentage, then a region, then everything. Some teams put an internal-users tier first so employees hit the build before customers do.

Facebook's "Rapid release at massive scale," the account of moving from a release branch to quasi-continuous push, describes the same shape: employees, then a small production percentage, then full. Slack's "Deploys at Slack" walks staged percentages with checks at each tier and a person who owns the deploy. The ladder is the industry's convergent answer; the rungs differ per team.

What it catches: whatever the previous four stages could not, which is everything that depends on real traffic, real data, and real scale.

Cost: the sum of the bake times at each tier. A four-tier ladder with 10-minute bakes is 40 minutes from first production instance to full rollout. That number is the honest deploy latency of a mature pipeline. Teams that quote "two minutes to production" usually mean two minutes to the canary.

Wired wrong: the tiers exist but traffic is not partitioned by them. A "5% canary" that is one pod behind a load balancer with sticky sessions can be receiving 0.4% of requests, or 20%. Verify the slice with the request-rate metric, not the pod count.

Stage 6: automated checks at each tier

What it does: at every tier, compares the new slice against the old one on a fixed set of signals and either promotes, holds, or aborts without a human.

The signal set is small and boring: error rate (5xx and application-level failures), latency at p50 and p99, saturation (CPU, memory, connection pools), and one or two business counters the service exists to move (checkouts, messages sent, jobs completed). Compare the canary against a baseline of the old build deployed at the same moment on the same hardware, not against the whole fleet. The fleet has warm caches and long-lived connections, and the canary has neither.

What it catches: regressions large enough to be visible in the slice. The arithmetic of what that means is its own piece; the short version is that a 5% slice watched for 5 minutes sees only large regressions, and the pipeline should say so rather than imply otherwise.

Cost: the bake time, already counted. Plus the engineering to define thresholds per service, which is the part most teams skip and the reason most canary gates never abort anything.

Wired wrong: the check is "did the deploy succeed" (pods healthy, health endpoint returns 200) rather than "did the service get worse." A health check that returns 200 while the service returns 500 to users is the most common way a rollout advances through a broken tier. The gate has to look at what users see.

Stage 7: the button a human still presses

What it does: at one tier, usually before the last, a human confirms the promotion.

The purpose is not to re-verify the automated checks. It is to hold the one decision the checks cannot make: is now a good time. An incident is in progress on a neighboring service. A marketing event starts in 20 minutes. A dependency's on-call just posted something in the channel. The automated gates know none of this.

What it catches: context. Bad timing, not bad code.

Cost: the human's response latency. Keep the button in the tool the on-call already has open; a promotion that requires finding the right dashboard is a promotion that waits an hour.

Wired wrong: the button sits at the beginning (approve before any production traffic) rather than near the end. Placed early, it delays the first real signal and turns the person into a stamp for code they cannot evaluate. Placed late, after the automated tiers have reported, it is a genuine decision with real information behind it.

The other way it goes wrong: the button is a chat message. "Deploying, anyone object?" is not a gate. It is a broadcast with a timeout.

Stage 8: the post-deploy verification window

What it does: after 100%, keeps watching for a defined period, with the previous artifact one command away, before the deploy is declared done and the next one is allowed to start.

Some regressions are invisible at the tier level and visible only at full scale: a cache hit rate that decays over an hour, a memory leak, a batch job that runs on the hour and hits the new code path. The window catches them while the cause is still obvious, because only one deploy is in the suspect set.

What it catches: slow-onset regressions and the ones that depend on full traffic.

Cost: the window, which serializes deploys. A 30-minute window caps a service at roughly 48 deploys a day. Teams that need more shorten the window per service, based on how quickly its regressions have historically shown up. That is a number worth measuring rather than guessing.

Wired wrong: the window has no end condition. "Done" is implicit, the next deploy starts whenever, and two deploys overlap. When the alert fires there are two suspects and the rollback target is unclear. The window needs an explicit close, and the next deploy waits for it.

Most pipelines have a box for every stage. The question is which boxes have a gate inside and which have only a name.

Three forks, one table

The forks are independent. Build-once with push and imperative deploys is a fine pipeline. A pipeline that is failing rarely gets fixed by switching sides at a fork; it gets fixed by putting a gate in a box that had none.

Draw your own, then find the missing stage

Take a whiteboard. Draw the eight stages as boxes in a row. Under each box, write three things about your pipeline as it exists today, not as the config suggests it should.

One: what artifact leaves the box. If the answer at stage 2 is a tag rather than a digest, write the tag.

Two: what stops the box from advancing. Not "tests" but the specific check and its threshold. If nothing stops it, write "nothing." If a person looks at something, write the person's name and what they look at.

Three: how long the box took on the last five deploys. Pull it from the pipeline log; do not estimate.

Then look for the box where the answer to the second question is "nothing" or a name. That is the stage you do not have. The common ones: a merge queue that runs a subset of the suite; a staging deploy that no automated check depends on; a canary tier whose only condition is "pods healthy"; a post-deploy window with no close.

Pick one. Wire the gate. Measure the minutes it adds and the failures it catches over the next month, and put both numbers next to the box. Then draw the pipeline again.