# Error budget burned by retry storm: pre-release checklist

Source: https://releasefailures.com/incidents/error-budget-burned-by-retry-storm/
Content version: 1.0.0
Verified on: 2026-10-22
Licence: CC BY 4.0

Definition: Client retries amplify a partial failure into a full outage.
- [ ] PR-01 Drop a share of the feature flags service's database connections in staging while the writer database runs at peak ingestion load, and confirm retries back off instead of multiplying. (test class: Chaos and resilience tests)
- [ ] PR-02 Open the database circuit breaker in a drill and confirm read-only flag evaluation keeps serving. (test class: Chaos and resilience tests)
- [ ] PR-03 Switch flag delivery to the legacy routing path with empty caches in a drill and confirm SDKs fall back to polling instead of retrying the stream. (test class: Chaos and resilience tests)
- [ ] PR-04 Replay a reconnect of every server-side SDK against the streaming service and its load balancer, and confirm both stay responsive. (test class: Load and performance tests)
- [ ] PR-05 Run the 300 millisecond database connection timeout against a writer database at peak ingestion load, and fail when connection failures exceed the 1 second baseline. (test class: Load and performance tests)
- [ ] PR-06 Keep the canary window open after a timeout change until the writer database has passed a peak ingestion period, and halt promotion on any rise in pod crash loops or connection pool timeouts. (test class: Canary and progressive delivery checks)
- [ ] PR-07 Alert on the ratio of SDK retry requests to first requests during a rollout, and halt the rollout when the ratio rises above baseline. (test class: Canary and progressive delivery checks)