Release Incident Playbook

Canary and feature flag rollout checks drawn from public incident reports

Canary and progressive delivery checks (a small share of traffic runs the new version with automated metric comparison) catch regressions visible in traffic metrics, memory growth and error rate rises. The canary class cannot catch faults in traffic the canary never receives, batch jobs or scheduled tasks. Five published incidents show canary and feature flag rollouts letting faults through, and each check below traces to one of 8 cited reports.

8 cited public reports · verified 2026-10-21

What canary checks catch and what they cannot catch

Canary and progressive delivery checks run post-deploy.

The canary class catches regressions visible in traffic metrics.

The canary class catches memory growth.

The canary class catches a rise in error rate.

The canary class cannot catch faults in traffic the canary never receives.

The canary class cannot catch faults in batch jobs or scheduled tasks.

Across the 58 published incidents, the canary class appears in 26 caught-by tables.

Only memory leak in new release carries a cited canary row.

The other 25 canary rows carry the by design analysis label.

Seven tools in the current capability matrix document canary and progressive delivery checks.

No provider in the current capability matrix names canary and progressive delivery checks.

Worked example

Memory leak in new release: in a status page report from 2023, a canary ring did not send enough traffic to an authorization service to expose a memory leak [1].

That build graduated to production at 6:37 PM UTC [1].

Alerts in report [1] triggered at 6:44 PM UTC, seven minutes after graduation.

The rollback in report [1] was delayed because parts of the deployment infrastructure relied on the authorization service and needed manual intervention.

Environment mismatch, production points to staging: in a post-mortem from 2023, a progressive deployment API returned a list that mixed staging and production releases [2].

Promotion to production in report [2] referenced a script deployed to the staging account, and the storage service returned HTTP 401 errors.

Alerting in report [2] fired when the rollout routed a percentage of traffic to the wrong destination.

Config change without restart semantics: in a post-mortem from 2025, a database permissions change rolled out gradually across a cluster [3].

Updated nodes in report [3] generated a doubled feature file while other nodes generated a correct one, so 5xx errors fluctuated as good and bad files were distributed every five minutes.

The first automated test in report [3] detected the fault 26 minutes after the change.

In a post-mortem from 2022, alerts watched error rates but did not fire on a drop in traffic to a port forwarding backend after a secret rotation [4].

Some components in report [4] loaded the new secret only on process restart.

Rate limit misconfigured: in a post-mortem from 2025, a partial deployment of a feature flag (a runtime switch that turns a code path on or off without a deploy) meant for a subset of users put a global rate limiter into an invalid state [5].

The limiter in report [5] rejected 100 percent of requests with 403 errors for 25 minutes.

In a post-mortem from 2024, a proxy configuration change sent HTTP 429 responses to traffic that had bypassed rate limits for about four hours [6].

Detection in report [6] came from a customer contacting support, and the follow-up was an alert on increases in 429 responses.

Partial rollback leaves inconsistent state: in a regulator order from 2013, new code reached seven of eight servers, and a repurposed flag activated old code on the eighth server [7].

Uninstalling the new code from the seven servers in report [7] let incoming orders activate the old code on those servers as well.

In a status page report from 2024, an automatic rollback returned only a few services to the previous version while others could not initialise the new versions [8].

A manual downgrade at 07:20 UTC in report [8] restored full availability at 08:10 UTC.

Checks before a canary or flag rollout starts

Each check below traces to a cited report in the worked example.

  1. Confirm the canary receives traffic for every service the build changes, at a rate that exercises the changed code path [1].
  2. Compare the environment of every release in the promotion list with the environment of the target application before promotion [2].
  3. Confirm every server in the pool holds the new code before a flag that the new code reads is turned on [7].
  4. Confirm a repurposed flag no longer activates retired code on any server [7].
  5. List which processes load a changed secret or config only on restart, and schedule those restarts in the rollout plan [4].
  6. Run the flag change for a subset of users in pre-production and check the limiter state for all other users [5].

Checks while the rollout runs

The canary class compares metrics, so the choice of metric decides what the rollout can see.

  1. Alert on a drop in traffic to each changed backend as well as on error rate [4].
  2. Alert on 403 and 429 response rates during any flag or limiter change [5] [6].
  3. Compare canary memory with baseline memory over a fixed hold period before graduation [1].
  4. Watch for 5xx rates that alternate between normal and raised, which in report [3] matched a gradual rollout producing good and bad files.
  5. Stop propagation of a generated configuration file and redeploy a known-good file when the fault follows the file [3].

Checks before and after a rollback

A rollback (returning a system to the previous release) after a partial rollout can leave services on different versions [8].

  1. Confirm the rollback path does not depend on the service being rolled back [1].
  2. Check the running version on every service after an automatic rollback completes [8].
  3. Check whether removing new code re-exposes old code that a flag still activates [7].
  4. Rehearse switching the production route back to the last known good build before the progressive release starts [2].

Public reports cited

  1. [1]GitHub, 2023-11-03. A performance and resilience optimisation to GitHub's authorization microservice contained a memory leak exposed under high traffic; pods crashed repeatedly, authorization checks failed and pages returned 404s until the change was rolled back. primary Incident with Git Operations, Issues, Pull Requests, Actions, API Requests, Codespaces, Packages, Pages and Webhooks
  2. [2]Cloudflare, 2023-11-01. During a progressive Workers KV release, a deployment tool bug made production reference a script deployed to the staging account, so KV returned HTTP 401 errors and dependent Cloudflare services were unavailable for 37 minutes. primary Cloudflare incident on October 30, 2023
  3. [3]Cloudflare, 2025-11-18. A database permissions change was rolled out gradually across a ClickHouse cluster, so the Bot Management feature file was generated correctly on some nodes and wrongly on updated nodes, and the network alternated between good and bad files until every node produced the oversized file. primary Cloudflare outage on November 18, 2025
  4. [4]GitHub, 2022-11-02. After a Codespaces secret rotation with a missed step, some downstream components kept using the old secret because they only loaded the new one on process restart, and one service instance in West Europe was not rotated at all, so port forwarding and later codespace creation failed. primary GitHub Availability Report: October 2022
  5. [5]GitHub, 2025-10-08. A partial deployment of a feature flag meant to scale down rate limiting for a subset of Copilot users put the global rate limiter into an invalid state, so it limited 100 percent of requests with 403 errors for 25 minutes. primary GitHub Availability Report: September 2025
  6. [6]GitLab, 2024-06-20. A configuration change to HAProxy changed how application level rate limiting was handled, so traffic that had been bypassing rate limits, mainly allowlisted customers and Git over HTTP users, started receiving 429 errors for about four hours. primary Incident Review: Uptick in 429 errors, unexpected authentication errors
  7. [7]Knight Capital Americas LLC, 2013-10-16. Knight deployed new RLP code to seven of eight SMARS servers, the eighth kept defective Power Peg code that the repurposed flag activated, and when Knight uninstalled the new code from the seven servers during the incident, incoming orders activated the Power Peg code on those servers too. primary In the Matter of Knight Capital Americas LLC, Release No. 70694
  8. [8]Virtuoso, 2024-07-17. A regular platform update degraded the Virtuoso EU production environment, the automatic rollback returned only a few services to the previous version while others could not initialise the new versions, and a manual downgrade at 07:20 UTC restored full availability at 08:10 UTC. primary Increased error rate on our API

Questions

Does a staged rollout replace testing?

No. The canary class cannot catch faults in traffic the canary never receives, batch jobs or scheduled tasks. In report [1], a canary ring did not send enough traffic to an authorization service to expose a memory leak.

Why did alerts miss a fault during a gradual rollout?

In report [4], alerts watched error rates and did not fire on a drop in traffic to a port forwarding backend. In report [6], detection came from a customer contacting support, and the follow-up was an alert on 429 responses.

What can a feature flag do on servers that hold different code?

In report [7], new code reached seven of eight servers, and a repurposed flag activated old code on the eighth server. Removing the new code from the seven servers then let the old code run on those servers too.

How many tools in the catalogue document canary checks?

Seven tools in the current capability matrix document canary and progressive delivery checks. The What catches this block lists them alphabetically, and no provider in the matrix names that class, so the provider list prints its empty state.