Evidence classes, grades, freshness and rejections
346 candidate reports were recorded for incident types; 189 were accepted and 157 rejected. Only primary and secondary reports count toward the rule of two independent organisations.
Grades and classes
| Grade | Class | What it is |
|---|---|---|
| primary | own-postmortem | the affected organisation's published post-mortem or incident write-up of a specific dated incident |
| primary | status-page | a status page incident record with a root-cause or what-happened update |
| primary | regulator | a regulator, court or ombudsman document naming a software failure and its effect |
| primary | repo-issue | a public repository issue whose maintainers mark the root cause of a release incident, with a date |
| secondary | third-party-analysis | an engineering analysis by another organisation that links a primary source |
| secondary | conference-talk | a recorded, dated talk by the affected team |
| practitioner | engineering-blog | a write-up of the failure class without a specific dated incident |
| practitioner | talk-or-podcast | practitioner discussion of the failure class |
Verification
Each candidate URL is loaded over plain HTTP and a verbatim passage recorded by the collector is matched in the page text, together with the publication year.
A second review reads each report against the incident definition and rejects reports that do not describe a change-caused instance of it.
Two pages from the same organisation about the same incident count as one source. The same outage reported by the affected company and by an analyst counts as one example.
Freshness
Incident reports have no age limit, but each published incident needs at least one report from the last 5 years. Tool and provider facts are re-verified when older than 12 months.
Rejection log summary
| Reason | Reports |
|---|---|
| does not match the incident definition | 126 |
| not caused by a release or change | 11 |
| repo-issue cause stated by the reporter, not a maintainer | 5 |
| repo-issue cause stated by a contributor, not a maintainer | 4 |
| no specific organisation and date | 2 |
| secondary entry does not link a primary source | 2 |
| secondary entry does not link a primary source that still loads | 1 |
| repo-issue cause not stated by a maintainer | 1 |
| second page from the same organisation about the same incident | 1 |
| repo-issue release cause stated by the reporter, not a maintainer | 1 |
| quoted fact absent on load | 1 |
| page no longer loads the fact on the original url | 1 |
| undated page | 1 |