Release Incident Playbook

API key rotation breaks integrations: symptoms, cause, and the test that catches it

A key rotation invalidates partner integrations that were not notified.

API key rotation breaks integrations is documented by 2 accepted public reports from 2 organisations, 1 of them published in the last 5 years.

  • Likelihood: not published (1 public report in 5 years)
  • Impact: medium (2 of 2 reports: availability)

2 public reports · verified 2026-10-21

Symptoms

User-visible

  • Customers' API-based integrations failed with authorization errors.[1]
  • The product UI and core functions stayed available while API access failed.[1]
  • End users could not reach first-party and third-party applications that relied on the identity service for authentication.[2]
  • Some users had to restart services to obtain new tokens and key metadata.[2]

Monitoring-visible

  • Service telemetry identified the problem and engaged the engineering team automatically.[2]
  • Authorization errors appeared on API requests only, with the UI unaffected.[1]

Root causes

  1. stage: config A planned cleanup of old API keys disabled newly created keys as well.[1]
  2. stage: config Automation that removes unused signing keys ignored the retain state on one key and removed it.[2]
  3. stage: config The changed public key metadata led relying applications to reject tokens signed with the removed key.[2]

Public examples

  1. [1]Atlassian Statuspage, 2026-03-10. A planned cleanup of old Statuspage API keys also disabled new keys, and customers' API-based integrations failed with authorization errors until keys were restored from a database backup. primary Statuspage API Access Issues
  2. [2]Microsoft, 2021-03-16. An automated Azure AD key rotation removed a signing key that had been marked to retain, and Microsoft and third-party applications that relied on the published key metadata stopped trusting tokens signed with it. primary Notification: Authentication errors across multiple Microsoft services - Validating Mitigation

Test classes that catch it

Test classStageLimitationBasis
Integration testspre-deployA run of the key cleanup job against seeded test keys leaves out relying applications that cached the published key metadata and later needed restarts.by design analysis
Configuration and environment testspre-deployA check that the current key set resolves passes before the bulk job runs, so it misses selection logic that disables new keys or ignores a retain state.by design analysis
Synthetic monitoringpost-deployA synthetic probe fails only after the bulk job has already disabled keys, so it shortens the outage but does not prevent a restore from a database backup.by design analysis

Risk

  • Likelihood: not published (1 public report in 5 years)
  • Impact: medium (2 of 2 reports: availability)
Counts behind the grades
Accepted public reports2
Published in the last 5 years1
Impact stated: availability2

How the grades are computed

Pre-release checklist

  • PR-01
  • PR-02
  • PR-03
  • PR-04
  • PR-05
  • PR-06

First 30 minutes

  • FM-01
  • FM-02
  • FM-03
  • FM-04
  • FM-05
  • FM-06
  • FM-07

Download the checklist

What catches this: catalogue entries

Listed because their own documentation or site names integration tests, configuration and environment tests, synthetic monitoring on the verification date. Alphabetical. Not a recommendation.

Tools whose documentation names these test classes

Providers whose site names these test classes

Only on this page

API key rotation breaks integrations starts in the config stage in report [1] and report [2], not in application code. Neither fault shipped in an application release. One was a planned cleanup of old API keys that also disabled new keys [1]. The other was automation that removes unused signing keys and ignored a retain state [2]. Pre-merge tests on application code never execute these jobs, and canary checks watch a release rather than a scheduled or bulk job.

Report [1] and report [2] also differ in where the failure surfaced. In report [1], API-based integrations failed while the UI stayed up, so a UI check stays green. In report [2], relying applications held cached key metadata, and cached metadata for a subset of storage resources had to be invalidated after the key metadata rollback at 21:05 UTC.

Questions

What causes API key rotation breaks integrations?

In one report, a planned cleanup of old API keys also disabled new keys [1]. In the other, automation removed a signing key that was marked to retain [2]. Both causes sit in the config stage.

What are the symptoms of API key rotation breaks integrations?

API-based integrations fail with authorization errors while the UI keeps working [1]. Applications that rely on the published key metadata reject tokens, and some users must restart services to recover [2].

Which test class catches API key rotation breaks integrations, and what does it miss?

Integration tests, configuration and environment tests and synthetic monitoring are listed, all with the by-design label. Synthetic monitoring fires only after keys are already disabled.

Which public reports document API key rotation breaks integrations?

Two primary reports from two organisations document it, published in 2026 [1] and 2021 [2]. Both classify the impact as availability.

What should be checked before a release to prevent API key rotation breaks integrations?

PR-01 runs the cleanup job against seeded old, new and retained keys. PR-02 runs the removal automation in dry-run mode and checks its list for retained keys.

Should a team roll back or fix forward when API key rotation breaks integrations happens?

Both reports rolled back key state: one restored keys from a point-in-time database backup [1] and the other rolled back key metadata at 21:05 UTC [2]. Asking customers to create new keys was offered only as a workaround [1].

Release Incident Playbook (2026). API key rotation breaks integrations: symptoms, cause, and the test that catches it, content version 1.0.0. https://releasefailures.com/incidents/api-key-rotation-breaks-integrations/. Accessed 2026-10-21.