Release Incident Playbook

Third-party rate limit hit: symptoms, cause, and the test that catches it

A new call pattern exceeds a provider's quota and requests are rejected.

Third-party rate limit hit is documented by 2 accepted public reports from 2 organisations, 2 of them published in the last 5 years.

  • Likelihood: low (2 public reports in 5 years)
  • Impact: medium (2 of 2 reports: availability)

2 public reports · verified 2026-10-21

Symptoms

User-visible

  • Remote entities backed by code repositories returned code host API rate limit errors.[1]
  • Services, environments, pipelines and templates stored in repositories failed to load.[1]
  • Pipeline executions that depended on remote YAML were blocked.[1]
  • 67 percent of users could not create or start cloud development environments.[2]
  • Event processing that normally took minutes was delayed by hours.[3]

Monitoring-visible

  • The provider answered with HTTP 429 Too Many Requests and an X-RateLimit-Remaining header at 0.[3]
  • Thousands of events failed even though each one was retried 5 times with backoff.[3]

Root causes

  1. stage: code A release made webhook event processing clear the sync cache on every event. Every entity fetch then went straight to the code host API.[1]
  2. stage: code With high webhook volume and many remote entities, repeated calls that retrieve all service definitions used up the code host API rate limit.[1]
  3. stage: third-party Onboarding a new region raised load on a global third-party dependency used by the regional authentication layer. The dependency throttled that layer.[2]

Public examples

  1. [1]Harness, 2026-02-20. A Harness release made webhook processing clear the GitX cache on every event, so entity fetches went straight to GitHub and exhausted GitHub API rate limits, blocking remote entities and pipelines for some enterprise customers until the release was rolled back. primary Git API rate limit errors
  2. [2]GitHub, 2023-11-13. Onboarding a new GitHub Codespaces region increased load on a global third-party dependency used by the regional authentication layer, which was throttled, so 67 percent of users could not create or start Codespaces for 2 hours and 49 minutes. primary GitHub Availability Report: October 2023

Practitioner sources cited for symptoms

  1. [3]Not named in the report, 2023-11-21. An engineering manager describes an unnamed company whose event-processing service started receiving HTTP 429 responses from a third-party API after breaching a project-wide limit that was not documented, which left thousands of events failed for about two weeks. practitioner How a 3rd Party API Can Ruin Your Weekend

Test classes that catch it

Test classStageLimitationBasis
Load and performance testspre-deployA load test with few remote entities and low webhook volume stays under the code host limit, because the call count grew with both webhook volume and entity count.by design analysis
Integration testspre-deployAn integration test can show that one webhook event clears the whole sync cache, but at test volume it does not exhaust a provider quota that only production traffic reaches.by design analysis
Canary and progressive delivery checkspost-deployA canary compares a new code version with the old one, and the region onboarding that throttled the authentication layer was not a code release, so the canary had no traffic to compare.by design analysis

Risk

  • Likelihood: low (2 public reports in 5 years)
  • Impact: medium (2 of 2 reports: availability)
Counts behind the grades
Accepted public reports2
Published in the last 5 years2
Impact stated: availability2

How the grades are computed

Pre-release checklist

  • PR-01
  • PR-02
  • PR-03
  • PR-04
  • PR-05
  • PR-06
  • PR-07

First 30 minutes

  • FM-01
  • FM-02
  • FM-03
  • FM-04
  • FM-05
  • FM-06
  • FM-07

Download the checklist

What catches this: catalogue entries

Listed because their own documentation or site names load and performance tests, integration tests, canary and progressive delivery checks on the verification date. Alphabetical. Not a recommendation.

Tools whose documentation names these test classes

Providers whose site names these test classes

Only on this page

In report [1], third-party rate limit hit came from a cache fault that a release introduced. Webhook event processing cleared the sync cache on every event [1]. The code host limit ran out for some enterprise customers with high webhook volume and many remote entities [1].

In report [2], the extra load came from onboarding a new region, not from a code release. The team reduced load on the external dependency by hand [2]. The follow-ups in report [1] and report [2] target the call pattern itself: alerting on cache hit rate and tests of cache behaviour in sanity and regression runs [1], and regional caching to cut calls to the dependency [2].

Questions

What causes third-party rate limit hit?

In report [1], a release made webhook processing clear the sync cache on every event, so fetches went straight to the code host API. In report [2], onboarding a new region throttled a global dependency of the authentication layer.

What are the symptoms of third-party rate limit hit?

Remote pipelines and templates failed to load with code host rate limit errors [1]. 67 percent of users could not create or start cloud development environments for 2 hours and 49 minutes [2].

Which test class catches third-party rate limit hit, and what does it miss?

Load tests, integration tests and canary checks are listed, all with the by-design label. A load test with few remote entities stays under the limit that real customers exhausted.

Which public reports document third-party rate limit hit?

Two primary reports from two organisations document it, published on 2023-11-13 [2] and 2026-02-20 [1]. Both classify the impact as availability.

What should be checked before a release to prevent third-party rate limit hit?

PR-01 confirms one webhook event leaves other cache entries in place. PR-03 replays production webhook volume and checks outbound calls against the provider's rate limit.

Should a team roll back or fix forward when third-party rate limit hit happens?

Report [1] rolled back the release, which restored normal caching. Report [2] had no release to roll back and reduced load on the external dependency by hand.

Release Incident Playbook (2026). Third-party rate limit hit: symptoms, cause, and the test that catches it, content version 1.0.0. https://releasefailures.com/incidents/third-party-rate-limit-hit/. Accessed 2026-10-21.