Rate limit misconfigured: pre-release and first 30 minutes checklists

A rate limit change blocks legitimate traffic or stops protecting an endpoint.

The pre-release checklist maps each item to a test class from the incident page. The first-30-minutes checklist covers detection, containment, the rollback decision and customer communication.

9 pre-release items · 8 first-30-minutes items · verified 2026-10-21

Pre-release checklist

PR-01Diff the rate limit values of each public API between the release and production and block the change when any limit drops without an approved change record. config
PR-02Apply the rate limit scaling flag to a partial subset of users in pre-production and confirm the global limiter still admits requests from users outside the subset. config
PR-03Send one request as a client on the rate limiting allowlist through the changed proxy configuration and confirm the request bypasses the limit. config
PR-04Replay a five-minute window of production traffic that includes allowlisted and Git over HTTPS clients and confirm the bypass count stays at its production level. load
PR-05Load test session starts at peak rate against the updated internal rate limit and confirm sessions start without queuing. load
PR-06Load test guild loading right after a session backlog clears and confirm the internal service's scaling targets keep latency within budget. load
PR-07Add an alert for when the share of HTTP 429 responses rises above twice its 7-day baseline after any rate limit or proxy change. canary
PR-08Halt a partial flag rollout automatically when 403 responses from the rate limiter rise above baseline. canary
PR-09Alert when the count of requests bypassing rate limiting falls by more than 50 percent within one 5-minute window. canary

First 30 minutes

FM-01Count 429 and 403 responses by client group, including allowlisted clients, and compare their start time with the last rate limit, proxy or flag change.
FM-02Read the configuration log for proxy and rate limit changes in the hour before the first support ticket.
FM-03If a flag or config change lines up with the spike, revert it to the previous working version before investigating further.
FM-04If sessions are queued behind the limit, bypass the internal rate limit temporarily so pending sessions can start.
FM-05Scale up the downstream service before the session backlog is released so recovery does not stall on its scaling targets.
FM-06Reply to every support ticket about rate limiting with the affected protocols and the time of the revert.
FM-07Publish a status update that names the affected API or protocol, the status code users see and the start time.
FM-08Close the incident only after the 429 share and the bypass count return to their pre-change levels.

Download the checklist

Pre-release checklist (Markdown)2.0 KB · v1.0.0
First 30 minutes checklist (Markdown)1.3 KB · v1.0.0

Only in this checklist

PR-04 is the only item among 5 published checklists in this group that relies on load and performance tests: Replay a five-minute window of production traffic that includes allowlisted and Git over HTTPS clients and confirm the bypass count stays at its production level.

Questions

What should be checked before a release to prevent Rate limit misconfigured?

9 pre-release items, PR-01 to PR-09. PR-01: Diff the rate limit values of each public API between the release and production and block the change when any limit drops without an approved change record.

Is there a rollback plan for Rate limit misconfigured?

The first-30-minutes checklist has 8 items. FM-01: Count 429 and 403 responses by client group, including allowlisted clients, and compare their start time with the last rate limit, proxy or flag change.

What should a team do in the first 30 minutes after Rate limit misconfigured is detected?

Which configuration checks apply to Rate limit misconfigured?

PR-01: Diff the rate limit values of each public API between the release and production and block the change when any limit drops without an approved change record. PR-02: Apply the rate limit scaling flag to a partial subset of users in pre-production and confirm the global limiter still admits requests from users outside the subset. PR-03: Send one request as a client on the rate limiting allowlist through the changed proxy configuration and confirm the request bypasses the limit.

Is a Rate limit misconfigured checklist still needed if deployment is automated?

Yes. Automation runs the steps; the checklist names what the tests must cover: configuration and environment tests, load and performance tests and canary and progressive delivery checks.

Which pipeline stages does the Rate limit misconfigured checklist cover?

The pre-release items map to the pre-deploy and post-deploy stages.