How to set regression scope from published release incident history
Add an incident type to the regression scope in order of its impact grade, then its likelihood grade, and write each case around the data state named in the incident's caught-by limitation. Of the 58 published incident types, 11 carry impact high and none carries likelihood high. Of 158 caught-by entries, 17 are cited from public reports and 141 carry the by design analysis label.
Read the risk grid first
The likelihood grade counts accepted primary and secondary reports from distinct organisations published in the last 5 years.
The impact grade follows the majority impact class among reports that state an impact.
Of the 58 published incident types, 11 carry impact high, 40 carry impact medium, 5 carry impact low and 2 are unrated.
Of the 58 published incident types, 17 carry likelihood medium, 32 carry likelihood low and none carries likelihood high.
9 published incident types carry the likelihood label "not published", because only 1 of their accepted reports falls within the last 5 years.
Permission escalation via role change is the only published incident type in the medium likelihood and high impact cell.
- List every incident type with impact high first, because its reports state data, money or compliance consequences.
- Order the rest by likelihood grade, medium before low.
- Keep unrated and "not published" likelihood types in a separate list and review them on each refresh.
Write the case from the caught-by limitation
Each caught-by entry on an incident page carries one limitation sentence that says when the test class passes while the fault ships.
The limitation names the data state, code path or traffic the test needs in order to fail.
A cited entry rests on an accepted report that names the test class.
A by design entry rests on the author's reasoning and links to the methodology page.
- Copy the limitation sentence into the regression case description.
- Add the named data state, code path or traffic to the case fixtures.
- Confirm the case fails on the release that shipped the fault, where that release is still available.
Check what the test class cannot catch
Canary and progressive delivery checks (a small share of traffic runs the new version with automated metric comparison) cannot catch faults in batch jobs or scheduled tasks.
End-to-end tests (user flows through the real UI against a deployed environment) cannot catch rare data states or timing-dependent faults.
Unit tests (tests of one function or class in isolation) cannot catch integration faults, configuration or data shape.
Load and performance tests (traffic replay or synthetic load with latency and resource budgets) cannot catch functional defects or data correctness.
- Compare the incident's failure with the cannotCatch list of the chosen test class.
- Move the case to a different test class when the failure appears in that list.
Worked example
Permission escalation via role change carries "Likelihood: medium (3 public reports in 5 years)" and "Impact: high (4 of 4 reports: data)".
Both caught-by entries on the permission escalation page are cited from public reports [1] [2].
A post-mortem from 2026 states that tests covered a performance optimisation while it was active but not its disabled code path [1].
The regression case for that 2026 report therefore runs the access check with the optimisation switched off [1].
A post-mortem from 2015 states that security tests covering more access scenarios were added after an administration failure during permission synchronisation [2].
The security limitation on the permission escalation page says a pre-deploy security test never sees a grant made by a synchronisation run outside the release [2].
Missing index after migration carries "Likelihood: medium (3 public reports in 5 years)" and "Impact: medium (4 of 5 reports: availability)".
A status page report from 2025 names load tests of schema changes before production rollout, measuring P50 and P75 endpoint latency deviations, as a follow-up [3].
The load limitation on the missing index page says the test exposes the fault only if its traffic includes the few high traffic queries that ignored the replacement index [3].
Partial rollback leaves inconsistent state carries "Likelihood: low (2 public reports in 5 years)" and "Impact: low (2 of 3 reports: degraded)".
A post-mortem from 2021 states: "This issue could have been caught with a well written unit test." [4]
The unit entry on the partial rollback page carries the cited label from that 2021 post-mortem [4].
The same 2021 post-mortem lists tests for rollback behaviour of parser cache content changes among its actions [4].
Memory leak in new release carries "Likelihood: low (2 public reports in 5 years)" and "Impact: medium (2 of 2 reports: availability)".
A status page report from 2023 states that the canary ring did not send enough traffic to expose a memory leak [5].
A canary alone therefore does not cover the memory leak case in that 2023 report [5].
What the grades do not tell
The risk model does not weight incidents by company size, industry or the author's opinion.
The risk model produces no single score and ranks incidents only by grid cell.
The likelihood grade measures public reporting, so a failure type that organisations rarely publish can carry a low grade.
The regression scope still needs the team's own incident history next to the published grades.
Public reports cited
- [1]Linear, 2026-04-06. A deployed change to the access control layer skipped team-level permission filtering for all but one workspace, so for about one hour members and guests could access data from private teams in their own workspace. primary Post mortem on Linear security incident on March 24th, 2026
- [2]JetBrains, 2015-08-10. An administration failure during a permission synchronization process gave all external users and guest accounts permissions to every project on the company's production YouTrack instance, including internal ones, for about an hour. primary JetBrains YouTrack Instance Security Issue Post Mortem
- [3]Onfido, 2025-02-14. During a progressive schema migration that changed a primary key, Onfido inadvertently dropped an index, endpoint latency rose and Workflow API requests failed with database query timeouts until the index was re-created. primary Workflow errors and latency
- [4]Wikimedia Foundation, 2021-11-08. Wikimedia rolled the MediaWiki train back to the previous version because of a table of contents regression, but the parser cache still held output written by the newer version, which the older code could not handle, so pages lost their table of contents on all wikis until the train was rolled forward again. primary Incidents/2021-11-05 TOC language converter
- [5]GitHub, 2023-11-03. A performance and resilience optimisation to GitHub's authorization microservice contained a memory leak exposed under high traffic; pods crashed repeatedly, authorization checks failed and pages returned 404s until the change was rolled back. primary Incident with Git Operations, Issues, Pull Requests, Actions, API Requests, Codespaces, Packages, Pages and Webhooks
Questions
How do you decide whether an escaped defect belongs in the regression scope?
Order incident types by impact grade, then by likelihood grade, and write each case from the caught-by limitation. Of the 58 published incident types, 11 carry impact high.
Why did tests not catch the bug?
The caught-by limitation on each incident page answers this for its test class. A 2026 post-mortem states that tests covered an optimisation while it was active but not its disabled code path [1].
Which published incident types carry a high likelihood grade?
None of the 58 published incident types carries likelihood high, which needs 6 or more public reports in 5 years. 17 carry likelihood medium.
Should a by design caught-by entry enter the regression scope?
A by design entry rests on the author's reasoning rather than a cited report, and 141 of 158 caught-by entries carry that label. Its limitation sentence still names the data state the case needs.