Error budget burned by retry storm: pre-release and first 30 minutes checklists

Client retries amplify a partial failure into a full outage.

The pre-release checklist maps each item to a test class from the incident page. The first-30-minutes checklist covers detection, containment, the rollback decision and customer communication.

7 pre-release items · 7 first-30-minutes items · verified 2026-10-22

Pre-release checklist

PR-01Drop a share of the feature flags service's database connections in staging while the writer database runs at peak ingestion load, and confirm retries back off instead of multiplying. chaos
PR-02Open the database circuit breaker in a drill and confirm read-only flag evaluation keeps serving. chaos
PR-03Switch flag delivery to the legacy routing path with empty caches in a drill and confirm SDKs fall back to polling instead of retrying the stream. chaos
PR-04Replay a reconnect of every server-side SDK against the streaming service and its load balancer, and confirm both stay responsive. load
PR-05Run the 300 millisecond database connection timeout against a writer database at peak ingestion load, and fail when connection failures exceed the 1 second baseline. load
PR-06Keep the canary window open after a timeout change until the writer database has passed a peak ingestion period, and halt promotion on any rise in pod crash loops or connection pool timeouts. canary
PR-07Alert on the ratio of SDK retry requests to first requests during a rollout, and halt the rollout when the ratio rises above baseline. canary

First 30 minutes

FM-01Check whether connection errors rose after a timeout, routing or load-reduction change, and note that change's deploy time.
FM-02Read service logs for connection pool timeout errors and count the pods in crash loops.
FM-03Compare retry request volume with normal request volume to confirm a retry storm.
FM-04Rate-limit client retries at the load balancer so the overwhelmed service or writer database can recover.
FM-05Roll back the timeout or routing change, and deploy the previous values as a manual configuration change if the deployment tool's rollback fails.
FM-06Switch SDKs from streaming to polling where the client supports it.
FM-07Post a status update with the share of failed flag evaluations, the affected regions and whether read-only flags are affected.

Download the checklist

Pre-release checklist (Markdown)1.7 KB · v1.0.0
First 30 minutes checklist (Markdown)1.1 KB · v1.0.0

Only in this checklist

PR-04 is the only item among 5 published checklists in this group that relies on load and performance tests: Replay a reconnect of every server-side SDK against the streaming service and its load balancer, and confirm both stay responsive.

Questions

What should be checked before a release to prevent Error budget burned by retry storm?

7 pre-release items, PR-01 to PR-07. PR-01: Drop a share of the feature flags service's database connections in staging while the writer database runs at peak ingestion load, and confirm retries back off instead of multiplying.

Is there a rollback plan for Error budget burned by retry storm?

The first-30-minutes checklist has 7 items. FM-01: Check whether connection errors rose after a timeout, routing or load-reduction change, and note that change's deploy time.

What should a team do in the first 30 minutes after Error budget burned by retry storm is detected?

The first-30-minutes checklist has 7 items. FM-01: Check whether connection errors rose after a timeout, routing or load-reduction change, and note that change's deploy time.

Is a Error budget burned by retry storm checklist still needed if deployment is automated?

Yes. Automation runs the steps; the checklist names what the tests must cover: chaos and resilience tests, load and performance tests and canary and progressive delivery checks.

Which pipeline stages does the Error budget burned by retry storm checklist cover?

The pre-release items map to the pre-deploy and post-deploy stages.

In what formats can the Error budget burned by retry storm checklist be downloaded?

Both checklists download as Markdown files: /downloads/error-budget-burned-by-retry-storm-pre-release.md and /downloads/error-budget-burned-by-retry-storm-first-30-minutes.md, licensed CC BY 4.0.