Release Incident Playbook

Provider API change breaks contract: symptoms, cause, and the test that catches it

A third-party API changes a field or behaviour and the integration fails.

Provider API change breaks contract is documented by 4 accepted public reports from 4 organisations, 4 of them published in the last 5 years.

  • Likelihood: medium (4 public reports in 5 years)
  • Impact: medium (2 of 4 reports: availability)

4 public reports · verified 2026-10-21

Symptoms

User-visible

  • Jobs that read contexts appeared to start in the UI and moved to Failed after a few minutes.[4]
  • Some customers' pull requests were never queued by the merge queue service.[3]
  • Draft pull requests that the merge queue service created were not always deleted.[3]
  • Customer servers with strict type validation failed to process post-call analysis events from the end of call webhook.[1]
  • Auth checks failed for customers who used a custom domain and validated the JWT iss field.[2]

Monitoring-visible

  • The jobs queue grew quickly while context permission checks failed.[4]
  • Requests to the moved upstream endpoint returned 404 responses with explanation text in the body.[4]
  • Logs showed unexpected HTTP errors saying the pull request the service tried to create already existed.[3]
  • Existing integration tests that used to pass started failing with no change in the consumer's code.[3]

Root causes

  1. stage: third-party The upstream provider moved a set of endpoints under the teams path and ran a brownout of the old path. The consumer read the migration notice as covering only the listed endpoints, and its code could not handle the 404 response format.[4]
  2. stage: third-party The upstream provider stopped closing a pull request when its branch was deleted, with no change to URLs, payloads or headers. The merge queue service relied on that behaviour when it recycled speculative branches.[3]
  3. stage: code A release changed call.analysis.successEvaluation from string or null to string, number, boolean or null. The change passed automated code tests and shipped.[1]
  4. stage: code An update made the JWT iss field match the custom domain in the request instead of the default issuer. The impact on existing iss validations was not fully considered, so the release became a breaking change without prior notice.[2]

Public examples

  1. [1]Vapi, 2025-06-19. A Vapi release that moved Success Evaluation to Gemini Structured Output changed the type of the call.analysis.successEvaluation field, so customer servers with strict type validation and users of Vapi Server SDKs failed to process post-call analysis events until a patch restored string-only output about 30 hours later. primary Breaking changes to Success Evaluation API Response
  2. [2]Logto, 2024-03-25. A Logto Cloud update made the JWT iss field reflect the custom domain used in the request instead of the default issuer, so customers who validated iss in their resource servers had auth checks fail until they changed their validation code. primary Postmortem: unexpected JWT `iss` change
  3. [3]Mergify, 2022-04-25. GitHub stopped automatically closing a pull request when its branch was deleted, without any change to API URLs, payloads or headers, so Mergify failed to recreate speculative merge queue pull requests and some customers' pull requests were not queued until a workaround shipped. primary Handling unexpected third party API changes
  4. [4]CircleCI, 2022-03-14. GitHub moved an API endpoint that CircleCI used to check context permissions and started a 12-hour brownout of the old path, so CircleCI could not handle the 404 responses and jobs that accessed contexts failed for about 110 minutes. primary Post Mortem: Workflows not running and jobs failing 01-03-2022

Test classes that catch it

Test classStageLimitationBasis
Integration testspre-deployAn integration suite that runs only on the consumer's own commits stays green when the provider changes behaviour between releases, and the merge queue service found the failing tests only after customer reports.cited in [3] [1] [2]
Contract testspre-mergeA contract test pins request and response shape, so it would not flag the branch deletion change where URLs, payloads and headers stayed the same.by design analysis
Synthetic monitoringpost-deployA scheduled probe that does not call through a custom domain with iss validation, or does not parse the webhook with strict types, stays green because only those customer segments failed.by design analysis

Risk

  • Likelihood: medium (4 public reports in 5 years)
  • Impact: medium (2 of 4 reports: availability)
Counts behind the grades
Accepted public reports4
Published in the last 5 years4
Impact stated: degraded2
Impact stated: availability2

How the grades are computed

Pre-release checklist

  • PR-01
  • PR-02
  • PR-03
  • PR-04
  • PR-05
  • PR-06
  • PR-07
  • PR-08

First 30 minutes

  • FM-01
  • FM-02
  • FM-03
  • FM-04
  • FM-05
  • FM-06
  • FM-07

Download the checklist

What catches this: catalogue entries

Listed because their own documentation or site names integration tests, contract tests, synthetic monitoring on the verification date. Alphabetical. Not a recommendation.

Tools whose documentation names these test classes

Providers whose site names these test classes

Only on this page

Provider API change breaks contract has a reported time to detect of 1311 minutes in report [1], about 22 hours after the release. The first signal was enterprise users reporting a breaking change in the end of call webhook message. The field type change had passed automated code tests [1]. Report [2] records 810 minutes, from a 10:00 UTC deployment to a first user report at 23:30 UTC.

In report [1] and report [2], the change shipped from the provider side and broke only consumers that validated strictly: typed servers in report [1] and iss validation behind a custom domain in report [2]. Neither report [1] nor report [2] records a monitoring signal before the first customer report. The fix in report [1] was a patch that restored string-only output about 30 hours after the release, and the action items name integration tests that catch response type changes.

Questions

What causes provider API change breaks contract?

Two reports trace it to the upstream provider: a brownout of a moved endpoint path [4] and a branch deletion that stopped closing pull requests [3]. Two reports trace it to a provider release that changed a payload field type [1] or the JWT iss value [2].

What are the symptoms of provider API change breaks contract?

Jobs that read contexts moved to Failed after a few minutes while the jobs queue grew [4]. Customer servers with strict type validation failed on post-call analysis events [1].

Which test class catches provider API change breaks contract, and what does it miss?

Integration tests carry the cited label, named by three reports [1][2][3]. A suite that runs only on the consumer's own commits misses a provider change shipped between releases. Contract tests and synthetic monitoring carry the by-design label.

Which public reports document provider API change breaks contract?

Four primary reports from four organisations document it, published between 2022-03-14 [4] and 2025-06-19 [1]. Two classify the impact as availability and two as degraded.

How long did it take to detect provider API change breaks contract in the public reports?

Report [1] records 1311 minutes and report [2] records 810 minutes. In both, the first signal was a customer report.

Should a team roll back or fix forward when provider API change breaks contract happens?

The provider in report [1] first kept the new field type, then shipped a patch that restored string-only output. The provider in report [2] did not roll back and emailed affected users to update their iss validation. FM-05 covers this decision.

Release Incident Playbook (2026). Provider API change breaks contract: symptoms, cause, and the test that catches it, content version 1.0.0. https://releasefailures.com/incidents/provider-api-change-breaks-contract/. Accessed 2026-10-21.