Release Incident Playbook

Rate limit misconfigured: symptoms, cause, and the test that catches it

A rate limit change blocks legitimate traffic or stops protecting an endpoint.

Rate limit misconfigured is documented by 4 accepted public reports from 4 organisations, 4 of them published in the last 5 years.

  • Likelihood: medium (4 public reports in 5 years)
  • Impact: medium (4 of 4 reports: availability)

4 public reports · verified 2026-10-21

Symptoms

User-visible

  • The global rate limiter returned 403 errors for 100 percent of requests to the affected product for 25 minutes.[1]
  • Customers saw more HTTP 429 responses on web, API and Git traffic, mostly Git over HTTPS.[3]
  • Customers in about 50 root namespaces were rate limited, mainly those on the rate limiting allowlist.[3]
  • Some customers could not use the public search API between 1:34 PM and 2:43 PM EDT on 11 March 2025.[2]
  • Users could not start sessions, and some guilds were unavailable or slow to load during recovery.[4]

Monitoring-visible

  • HTTP 429 errors spiked from 05:55 UTC after the proxy configuration change.[3]
  • Requests bypassing rate limiting dropped from about 2,000,000 to 15,000 per 5-minute window.[3]
  • A customer reached support through the paging channel before the team found the 429 spike, and four support tickets were raised about rate limiting.[3]

Root causes

  1. stage: config A partial deployment of a feature flag meant to scale down rate limiting for a subset of users put the global rate limiter into an invalid state that limited all requests.[1]
  2. stage: config A proxy configuration change altered how application-level rate limiting was handled, so traffic that had been bypassing rate limits started receiving 429 errors.[3]
  3. stage: config An unintended configuration change lowered the rate limits of a public search API.[2]
  4. stage: config Updated internal rate limiting stopped sessions from starting, and the scaling targets of an internal service then limited guild loading during recovery.[4]

Public examples

  1. [1]GitHub, 2025-10-08. A partial deployment of a feature flag meant to scale down rate limiting for a subset of Copilot users put the global rate limiter into an invalid state, so it limited 100 percent of requests with 403 errors for 25 minutes. primary GitHub Availability Report: September 2025
  2. [2]HubSpot, 2025-03-12. An unintended configuration change lowered the rate limits of HubSpot's public search API, and some customers could not use the API for about 70 minutes. primary HubSpot's public search API was unavailable for some users
  3. [3]GitLab, 2024-06-20. A configuration change to HAProxy changed how application level rate limiting was handled, so traffic that had been bypassing rate limits, mainly allowlisted customers and Git over HTTP users, started receiving 429 errors for about four hours. primary Incident Review: Uptick in 429 errors, unexpected authentication errors
  4. [4]Discord, 2024-03-05. Updated internal rate limiting stopped Discord sessions from starting until the rate limit was temporarily bypassed, after which guild loading was delayed while an internal service scaled up. primary Issue starting sessions, loading guilds

Test classes that catch it

Test classStageLimitationBasis
Configuration and environment testspre-deployA check of limit values per environment passes when each value is valid, so it misses the invalid limiter state that a partial flag rollout to a subset of users produced.by design analysis
Load and performance testspre-deployA load test that sends traffic only from clients outside the allowlist never exercises the bypass path that allowlisted and Git over HTTPS traffic depended on.by design analysis
Canary and progressive delivery checkspost-deployA canary that tracks only 5xx error rates misses a rise in 429 and 403 responses, which are the status codes the misconfigured limiters returned.by design analysis

Risk

  • Likelihood: medium (4 public reports in 5 years)
  • Impact: medium (4 of 4 reports: availability)
Counts behind the grades
Accepted public reports4
Published in the last 5 years4
Impact stated: availability4

How the grades are computed

Pre-release checklist

  • PR-01
  • PR-02
  • PR-03
  • PR-04
  • PR-05
  • PR-06
  • PR-07
  • PR-08
  • PR-09

First 30 minutes

  • FM-01
  • FM-02
  • FM-03
  • FM-04
  • FM-05
  • FM-06
  • FM-07
  • FM-08

Download the checklist

What catches this: catalogue entries

Listed because their own documentation or site names configuration and environment tests, load and performance tests, canary and progressive delivery checks on the verification date. Alphabetical. Not a recommendation.

Tools whose documentation names these test classes

Providers whose site names these test classes

Only on this page

In report [3], rate limit misconfigured hit the customers the limiter was meant to exempt. A proxy configuration change altered how application-level rate limiting was handled. Requests bypassing rate limiting dropped from about 2,000,000 to 15,000 per 5-minute window. Customers in about 50 root namespaces, mainly those on the allowlist, received 429 errors for about four hours.

A load test that sends traffic only from ordinary clients never exercises that bypass path. A customer reached support before the team found the 429 spike. The follow-ups in report [3] add an alert on increases in 429 responses and show configuration management changes in the configuration log. The review notes an earlier incident with a similar rate limiting misconfiguration.

Questions

What causes rate limit misconfigured?

All four reports trace it to a config change. The changes were a partial flag rollout [1], an unintended lower limit [2], a proxy change that removed a bypass [3] and updated internal rate limiting [4].

What are the symptoms of rate limit misconfigured?

Users receive 403 or 429 responses or cannot start sessions. In one report, the rate limiter limited 100 percent of requests for 25 minutes [1].

Why did tests not catch rate limit misconfigured?

In one report, an undetected edge case put the limiter into an invalid state only under a partial flag rollout [1]. The follow-ups in that report add more coverage of rate limit scaling tests in pre-production validation.

Which test class catches rate limit misconfigured, and what does it miss?

Configuration and environment tests, load and performance tests and canary checks are listed, all with the by-design label. A canary that tracks only 5xx error rates misses the 429 and 403 responses that rate limiters return.

What should be checked before a release to prevent rate limit misconfigured?

PR-03 sends one request as an allowlisted client through the changed proxy configuration. PR-07 adds an alert on increases in the share of HTTP 429 responses.

Should a team roll back or fix forward when rate limit misconfigured happens?

Three reports describe a revert or bypass: the flag revert recovered immediately [1], the proxy change returned to the previous working version [3], and the internal limit was temporarily bypassed [4].

Release Incident Playbook (2026). Rate limit misconfigured: symptoms, cause, and the test that catches it, content version 1.0.0. https://releasefailures.com/incidents/rate-limit-misconfigured/. Accessed 2026-10-21.