DISCUSSION

releaseLinkincident.io ↗

We turned off Pub/Sub and nobody noticed

240M messages a day split 50/50 across Pub/Sub and NATS, cut over topic by topic, then verified by deleting the NATS cluster in production. Nobody noticed.

Title card reading “We turned off Pub/Sub and nobody noticed”, with rows of alert icons on a branching line

Why we picked it · the editor's summary

incident.io wanted Pub/Sub off its single-point-of-failure list without a big-bang migration, so it wrote a load-balancer layer over two brokers. Publishes split evenly; a circuit breaker fails a broker out on errors; consumers drain whichever side has the deeper queue, with delay-based scheduling so neither starves. The proof was chaos testing in production: disable NATS, confirm nothing changed for customers, then disable Pub/Sub, same result, against a 99.99% availability target. No cost, latency, or timeline figures are given. The recommendations are to write the broker requirements down before choosing one and to keep exercising the failover rather than trusting it. If your queue has one broker, the abstraction is the part to copy.

Read on incident.io ↗14m read

Comments

Loading comments…

Back to all articles