Observability and rollback release incidents

Observability and rollback: 5 published incident types backed by 14 public reports.

The root-cause stage named most often in this group is release-process, in 4 of 5 incident types.

5 incident types published · 4 withheld · 14 public reports

Incident types in this group

Observability and rollback


Alerts silent after metric rename	Likelihood: low (2 public reports in 5 years)	Impact: unrated (fewer than 2 reports state impact)	2
Error budget burned by retry storm	Likelihood: low (2 public reports in 5 years)	Impact: medium (2 of 2 reports: availability)	2
Partial rollback leaves inconsistent state	Likelihood: low (2 public reports in 5 years)	Impact: low (2 of 3 reports: degraded)	3
Release marker missing in monitoring	Likelihood: not published (1 public report in 5 years)	Impact: medium (2 of 2 reports: availability)	3
Rollback fails due to forward-only change	Likelihood: low (2 public reports in 5 years)	Impact: medium (2 of 2 reports: availability)	4

Withheld in this group: Canary passed but full rollout failed, Log pipeline dropped during incident, Dashboard shows stale data, Status page not updated. Reasons per route

Risk grid

Risk grid: count of incident types per likelihood and impact cell, from data/risk.json
Likelihood \ Impact	Impact: high	Impact: medium	Impact: low
Likelihood: high	0	0	0
Likelihood: medium	0	0	0
Likelihood: low	0	2	1
Impact unrated or likelihood not graded	2

Table equivalent: incidents per cell

Incident	Likelihood	Impact
Alerts silent after metric rename	low	unrated
Error budget burned by retry storm	low	medium
Partial rollback leaves inconsistent state	low	low
Release marker missing in monitoring	not-published	medium
Rollback fails due to forward-only change	low	medium

Root-cause stages

Stage	Incident types naming it
release-process	4
code	3
config	2
migration	2
third-party	1

Fewer than 2 reports in this group state both the change time and the detection time; no median is printed.

Only on this page

Rollback fails due to forward-only change is the incident type in this group whose root causes do not include the release-process stage; its causes are introduced at migration.

Group checklist

The first pre-release item of each published incident type in the group.

GR-01Alerts silent after metric rename: Compare the collector's exported metric names with the previous release and fail the build on any rename or removal not listed in the changelog. contract
GR-02Error budget burned by retry storm: Drop a share of the feature flags service's database connections in staging while the writer database runs at peak ingestion load, and confirm retries back off instead of multiplying. chaos
GR-03Partial rollback leaves inconsistent state: Add a unit test that renders the table of contents with the previous release's code from a cache entry written by the new release. unit
GR-04Release marker missing in monitoring: Confirm the HTTP router deploy writes a deployment annotation to the monitoring dashboards in staging before the production deploy. config
GR-05Rollback fails due to forward-only change: Apply each column type change to a production-shaped copy, write rows with the new type, run its down migration, and fail when the down migration errors. migration

Questions

What causes Observability and rollback incidents?

Published incident types in this group name these root-cause stages: release-process (4), code (3), config (2), migration (2), third-party (1).

Which pipeline stage introduces most Observability and rollback incidents?

The release-process stage, named in 4 of 5 published incident types in the group.

Which test classes catch Observability and rollback incidents?

Test classes listed on the group's incident pages: configuration and environment tests (3), chaos and resilience tests (3), canary and progressive delivery checks (2), contract tests (1), end-to-end tests (1), load and performance tests (1), unit tests (1), synthetic monitoring (1), migration tests (1).

How quickly are Observability and rollback incidents detected in public reports?

Fewer than 2 reports in this group state both the change time and the detection time, so no median is published.

How likely are Observability and rollback incidents compared with each other?

Alerts silent after metric rename: low likelihood, unrated impact; Error budget burned by retry storm: low likelihood, medium impact; Partial rollback leaves inconsistent state: low likelihood, low impact; Release marker missing in monitoring: not-published likelihood, medium impact; Rollback fails due to forward-only change: low likelihood, medium impact.

What should a release checklist for Observability and rollback include?

The group checklist collects the first pre-release item of each of the 5 published incident types; each item names the test class that checks it.