Transforming How We Run Kafka at Honeycomb

Why we picked it · the editor's summary

Honeycomb moved its clusters from self-hosted Confluent with ZooKeeper to Apache Kafka 4.1.1 in KRaft mode on EKS. The design came out of a December 2025 incident that forced a full cluster evacuation and showed that the ingest service could reset offsets and switch clusters, which became the migration primitive. Each run sequenced producer cutover before consumer cutover, accepted a deliberate window of downtime between them, and was watched on purpose-built OpenTelemetry dashboards. Practice compressed a run from four to five hours to two to three, and the last one shipped while the lead was on leave, which was the point. Broker replacement used to take 48 to 72 hours; the post says it is faster now without a number. Parsons's caveat is that your circumstances will vary. Rehearse the rollback before the cutover, then rehearse it again.