Progressive delivery that actually stops a bad build

Most canary setups do not stop bad builds. They stop builds that crash on boot, which the health check would have caught anyway, and they wave through the ones that raise the error rate by half a percent, which is the regression that pages someone two days later.

The reason is arithmetic, not tooling. A canary is a sample, and a small sample watched briefly cannot tell a small change from noise. The teams whose rollouts do stop have done the math for their own traffic, picked signals with enough events in them, written the abort criteria down in advance, and let a program apply them.

The arithmetic of a small canary

Take a service that handles 40 req/s. Not a giant; a normal internal API. Its baseline error rate is 0.5%.

Route 5% of traffic to a canary. The canary gets 2 req/s. Watch it for 5 minutes: 600 requests.

At 0.5%, the canary should produce about 3 errors in that window. Errors at this rate arrive roughly as a Poisson process, so 3 is the mean and the spread around it is wide. Observing 6 or more errors when the true mean is 3 happens about 8% of the time by chance alone. An abort threshold at 6 errors therefore fires falsely on roughly one deploy in twelve. A team that ships twenty times a day will abort a good build almost daily, and within a month someone will raise the threshold.

Raise it to 8. Now a false abort happens about 1.2% of the time, which is tolerable. But consider a real regression: the new build doubles the error rate to 1.0%. The canary now produces a mean of 6 errors in the window. The chance it produces 8 or more, and trips the threshold, is about 26%.

On this service, with this canary size and this bake time, a build that doubles the error rate gets through the gate three times out of four. A regression of half a percentage point on top of a 0.5% baseline is not small. It is a 2x increase, and the canary cannot see it.

The levers are the ones in the setup. More requests in the sample: a bigger slice, a longer bake, or both. To separate 0.5% from 1.0% with reasonable confidence in both directions (few false aborts, few misses), the canary needs on the order of a few thousand requests, not 600. At 2 req/s that is 20–30 minutes, and longer when the comparison is against a baseline that is itself noisy. A 25% slice gets there in about 5 minutes, at the cost of exposing a quarter of users to the build.

Latency percentiles are worse. A p99 computed from 600 requests is the sixth-worst request; one garbage collection pause moves it. Comparing canary p99 to baseline p99 over 5 minutes at this volume is comparing two dice rolls.

Bake time and slice size are not defaults; they are per-service numbers derived from the service's traffic and the size of regression the team wants to catch. For low-traffic services, an error-rate canary is a boot check. The team should know that, and put its verification somewhere the volume exists: shadow traffic in pre-prod, a synthetic probe with a known answer, a longer post-deploy window.

Waves, and what each one is for

A single canary is the first rung. The ladder above it exists because different failures show up at different scales.

One instance catches crash-on-boot, bad config, a missing secret. Minutes.

A percentage (5–10%) catches request-path regressions large enough for the math above. Tens of minutes.

One region or availability zone catches failures that depend on a full slice of infrastructure: a dependency in that region that behaves differently, a data-locality assumption, a cache that is cold everywhere at once. A regional wave also gives you a rollback that is a traffic shift rather than a redeploy: send the region's traffic elsewhere and its users are on the old build in seconds.

Everything catches what needs full scale: connection pool exhaustion at peak, a queue that backs up only when every producer is on the new code.

The number of rungs is a cost decision. Every rung adds its bake to the deploy. A four-rung ladder with 10-minute bakes has a 40-minute floor from first production instance to full rollout. For most services that is fine; for the one that ships hotfixes it is not, which is why ladders are per service and not per company.

Sponsored:

Choosing the signals

A canary analysis compares the new slice to a baseline on a fixed set of signals. Keep the set small, because each signal is another chance for a false abort, and every signal must have enough events in the bake window to say something.

Error rate is the primary. Count transport-level failures (5xx) and application-level ones (a 200 whose body reports failure). Segment by endpoint when the service has one endpoint that carries the traffic and twenty that do not; a regression in a low-volume endpoint disappears in the aggregate.

Latency: p50 for the common path, p99 for the tail. Trust the tail only when the sample is large enough for it to mean something. At low volume, watch p50 and p90 and accept that the tail is unobservable in the window.

Saturation: CPU, memory, file descriptors, connection pool usage, thread counts. These catch the leak and the accidental N+1 that do not show up as errors yet. A canary burning 30% more CPU than baseline for the same traffic is a regression whether or not a request failed.

One business metric: the thing the service exists to do. Orders placed, messages delivered, searches returning results. It is the check on all the others, because a build can be green on every technical signal and still have broken the feature. The count is usually too low for statistics, so treat it as a floor (abort if it drops to zero or near it) rather than a comparison.

Compare against the right baseline. The old build on the fleet has warm caches, long-lived connections, and hours of JIT. The canary has none of those. The honest comparison is against a baseline group deployed at the same moment on the same class of hardware running the previous artifact. Netflix's automated canary analysis, open-sourced with Google as Kayenta for Spinnaker, is built around exactly this baseline-versus-canary pairing.

Abort criteria, written before the deploy

Write the rule down before the rollout starts, and let a program evaluate it, not whoever is watching.

A workable shape:

signal        window   abort when
error_rate    10m      canary > baseline + 0.5pp  AND canary errors >= 10
p50_latency   10m      canary > baseline * 1.25
p99_latency   10m      canary > baseline * 1.5    (only if requests >= 5000)
cpu_per_req   10m      canary > baseline * 1.3
orders        10m      canary count == 0 while baseline count > 20
 
hold (extend window once) when any signal passes half its abort delta

Three features of that table matter more than the numbers. The absolute floor on the error count stops a single bad request from aborting a low-volume canary. The sample-size condition on p99 keeps the tail from firing on noise. Every threshold has a window, and the window is the bake, so "promote" means the whole window passed.

The hold state is the third. An inconclusive reading (not enough events, one signal marginal) extends the tier's bake instead of promoting or aborting. Most rollouts that get stopped should be stopped by hold-then-abort, not by an instant trip on the first bad minute.

A program, not a dashboard

A human watching a dashboard during a rollout does three things badly. They look at whichever graph is most visually dramatic, which is usually the noisiest one. They apply a threshold invented in the moment that nobody can reproduce afterward. And they get called away.

The automated version does one thing: applies the table at the end of each window and emits promote, hold, or abort. Its output is a single line in the deploy log with the numbers that produced the decision, so the next engineer can see why.

The human's job moves. Before the deploy: set the thresholds and review them when a service's traffic changes. During the deploy: press the button at the last tier, where the decision is about context (is now a good time) rather than about the graphs. After an abort: read the decision line and decide whether the abort was right.

The dashboard still exists. It is for the human after the abort, not instead of it.

Flags decouple deploy from release

Pete Hodgson's Feature Toggles is the reference for this, and the clearest statement of why deploy and release are separate words. Deploy: the artifact is running in production. Release: users can reach the behavior. A release toggle lets a build ship dark, sit in production under real conditions, and turn on later for 1% of users, then 10%, then all, from a control plane that is not the deploy pipeline.

For progressive delivery this changes what the canary has to catch. Infrastructure regressions (crash, memory, latency on existing paths) still ride the deploy rollout. Behavioral changes ride the flag rollout, where the population can be chosen (internal users, one cohort, one region) and the rollback is a config flip with no artifact involved.

Hodgson's piece is equally clear about the cost, and it is the part teams skip. A release toggle is a branch in the code with two live paths. The test matrix doubles per flag until the flag is removed. Each flag needs an owner, an expiry, and a removal PR that is part of the launch rather than a cleanup someone will get to. A codebase with two hundred flags of unknown state is not decoupled from anything; it is a deploy pipeline with a second, undocumented rollout mechanism. The invoice for the lifecycle tax arrives at the worst moment: someone flips an old flag during an incident and finds out what the other path does.

Ops toggles, the kill switches, are the good kind of long-lived flag: few, documented, and exercised regularly. Everything else expires.

Testing in production, with names for the mechanisms

Cindy Sridharan's "Testing in Production, the safe way" turned the slogan into a taxonomy, and the taxonomy is what makes it usable. She splits production testing into three phases, and the phases map onto everything above.

Deploy phase, before any user sees the new build: integration testing against real dependencies, shadowing (send a copy of live traffic to the new build and discard its responses), tap-compare (send the same request to old and new and diff the answers), load testing, config validation. Shadowing and tap-compare are how a low-traffic service gets around the canary arithmetic: 100% of traffic reaches the new build and no user sees its output.

Release phase, when users start to hit it: canarying, the monitoring that drives the analysis, exception tracking, traffic shaping to control the slice.

Post-release, once it is fully out: feature flags and dark launches, A/B tests, the telemetry that says whether it worked, chaos experiments.

The reason to credit the taxonomy is that it names what a canary is and is not. A canary is one release-phase mechanism. Shadowing is a deploy-phase mechanism that a canary cannot replace and that many services need more. A team that "tests in production" with a canary alone has picked one row of the table.

A rollout being stopped

Suppose a team deploys a build of an order service that handles about 2,000 req/s across 40 instances. Numbers are illustrative; the shape is the point.

14:02. Merge queue lands the commit. Build starts.

14:09. Artifact built, signed, pushed by digest. Pre-prod deploy and smoke suite start.

14:15. Smoke suite green. Rollout begins: one canary instance behind the load balancer, plus one baseline instance running the previous artifact, deployed alongside on the same node type. Each takes about 50 req/s.

14:16. Canary passes boot check and health probe. Tier 1 bake, 5 minutes.

14:21. Tier 1 analysis, 15,000 requests each side: error rate canary 0.4% vs baseline 0.5% (61 vs 74 errors); p50 within 3%; CPU per request within 5%. Promote.

14:21. Tier 2: 10% of traffic to the canary group, now four instances, with a four-instance baseline group. Bake, 10 minutes.

14:27. Six minutes in, about 72,000 requests each side. Canary error rate 0.9% (650 errors) against baseline 0.5% (360). Over the count floor. Under the +0.5pp abort line, but past half of it. State: hold. The tier's window extends by 5 minutes instead of promoting at 14:31.

14:31. Ten minutes in, 120,000 requests each side. Canary 1.1% (1,320 errors) vs baseline 0.5% (600). Both abort conditions met. State: abort.

14:31. Automated rollback: traffic weight for the canary group goes to zero within the load balancer's drain interval, about 30 seconds. Canary instances are kept running, isolated from traffic, for diagnosis. The deploy log records the decision line with the numbers. The on-call gets the line and a link.

14:32. All user traffic is on the previous artifact. Users were on the bad build for 11 minutes at 10%, and 6 minutes at one instance before that.

14:40. On-call reads the canary's logs. The errors are timeouts to an inventory service from a new code path that makes two calls where the old one made one. Under load, the second call is what trips the client-side deadline. Pre-prod did not catch it because pre-prod's inventory service answers in 2 ms.

14:55. The team decides: revert now through the queue, fix the deadline later. The revert is a new artifact and it goes through the same ladder.

Three details in that sequence carry the design. The hold at 14:27 kept the rollout from promoting on a marginal reading and from aborting on one; either would have been a guess. The abort was a traffic shift, which is why it took 30 seconds and not a redeploy. And the canary instances were kept, not killed, because the evidence was on them.

What to change on Monday

Pull the request rate for the three services you deploy most. Compute the number of requests the canary tier sees in its bake window. If that number is under a few thousand, the tier is a boot check. Lengthen the bake, widen the slice, add shadowing in pre-prod, or accept it and say so in the runbook.

Then find the abort criteria. If they live in someone's head, write the table. If the table exists, add the error-count floor, the sample-size condition on the tail, and a hold state. Run one week of deploys with the analysis in log-only mode and count how often it would have aborted. That number tells you whether the thresholds are right before they start stopping real builds.