How the playbook is built from public incident reports

Section 1 is the H1, in the form "{Incident name}: symptoms, cause, and the test that catches it".

Section 2 is a direct answer of 3 to 5 sentences with the definition, the likelihood and impact labels, the count of public reports and the verification date.

Section 3 lists 2 to 5 user-visible and 2 to 5 monitoring-visible symptoms, and each line cites an accepted report.

Section 4 lists 2 to 4 root causes, and each names the pipeline stage where the cause is introduced.

Section 5 lists every accepted primary and secondary report with organisation, date, a one-line summary, grade and link.

Section 6 names 1 to 3 test classes, each with a stage, one limitation sentence and a basis label.

Section 7 prints the full risk labels and the counts behind them.

Section 8 is a pre-release checklist of 5 to 10 items, and each item names its test class.

Section 9 is a first-30-minutes checklist of 5 to 8 items covering detection, containment, the rollback decision and customer communication.

Section 10 is the What catches this block, built only from the capability matrix.

Section 11 is a unique layer of 1 to 2 paragraphs that must not fit any other page.

Section 12 links 3 to 5 related incidents from data.

Section 13 holds 4 to 6 questions and the disclaimer.

The section order is frozen, and an audit script checks every page against it.

A primary report is the affected organisation's own post-mortem (a written review of an incident: timeline, cause, impact and follow-up actions), a status page (a public page where a service posts incident updates) record with a root cause, a regulator document, or a repository issue whose root cause a maintainer states.

A secondary report is an engineering analysis by another organisation that links to a primary source, or a recorded talk by the affected team.

A practitioner report describes the failure class without a specific dated incident.

Practitioner reports may back symptoms and checklists only, and never count toward the two-report rule.

Two reports are independent when they come from different organisations and do not describe the same incident.

Of the 189 accepted reports, 177 are primary and 12 are practitioner.

The 177 accepted primary reports split into 89 status page records, 48 own post-mortems, 39 repository issues and 1 regulator document.

Every candidate report enters the evidence file with an empty status.

The first step is mechanical: a script loads each URL with a plain HTTP request and checks that the recorded quote and the publication year appear on the page.

A report that fails the mechanical check, or that the script cannot load, goes to the second step for manual review.

The second step is a definition-fit review: the verifier opens the page and confirms it loads, is dated, names the organisation and contains the quoted fact.

The verifier then checks that the report describes the incident type as defined, and that a release or change caused the failure.

The verifier never changes a summary to fit a page, and rejects the report instead.

Each accepted report carries the date it was verified.

The evidence file holds 346 candidate reports across 81 incident types.

Of those 346 reports, 189 are accepted, 157 are rejected and none await review.

Of the 157 rejections, 126 state that the report does not match the incident definition.

Eleven rejections state that no release or change caused the failure.

Eleven rejections concern repository issues where no maintainer stated the cause.

Three rejections are secondary reports that do not link a primary source that still loads.

Two rejections name no specific organisation and date.

One rejection is a second page from the same organisation about the same incident.

Two rejections are pages where the quoted fact no longer loads.

One rejection is an undated page.

Of the 84 incident types, 58 pages are published and 26 are withheld.

Among the 26 withheld pages, 13 have 1 accepted report of the 2 required and 13 have none.

Seventeen withheld pages have no accepted report published in the last 5 years.

Three withheld pages have no candidate report recorded.

Every withheld route is listed with its reason in the not-published file.

Memory leak in new release is published with 3 accepted primary reports.

The first is a status page report from 2023, in which a memory leak in an authorization service passed a canary ring that did not send enough traffic to the service [1].

The second is a repository issue from 2023, in which a new adaptive limiter caused memory to build up on a file server until 3.35 percent of Git requests returned 500 errors [2].

The third is a repository issue from 2021, in which a goroutine leak appeared on a canary file server after a deploy [3].

Reports [1] and [2] come from different organisations and describe different incidents, so they meet the independence rule.

Reports [2] and [3] come from the same organisation, so they count as one organisation for the likelihood grade.

Report [3] was published in September 2021, outside the 5-year window of the build.

The page therefore prints "Likelihood: low (2 public reports in 5 years)".

Report [3] states no impact, so the impact label reads "Impact: medium (2 of 2 reports: availability)".

The canary row is labelled cited because report [1] names the canary ring.

Double charge after deploy is published with 2 accepted primary reports.

Five candidate reports were recorded for double charge after deploy.

Two of those candidates were secondary articles that did not link a primary source, and both were rejected.

A third candidate was rejected because it did not match the incident definition.

One accepted report for double charge after deploy was published in 2017 [4], and the other was published in 2025 [5].

Only 1 accepted report falls inside the 5-year window, so the double charge page prints "Likelihood: not published (1 public report in 5 years)".

The playbook does not publish composite examples or examples without an organisation, a date and a link.

The risk model does not weight by company size, industry or the author's opinion.

The risk model does not produce a single score.

Tools and providers appear only where their own documentation or site names a test class, in alphabetical order, with no ratings.

A caught-by row without a cited report carries the by design analysis label.