Release Incident Playbook

Unit tests: what they catch and the release incidents they cover

Unit tests (tests of one function or class in isolation) run at the pre-merge stage.

9 published incident types list unit tests as a class that catches them; 1 of these links are cited in a public report and 8 are by design analysis.

stage: pre-merge · 9 published incident types · 1 cited

What unit tests catch and cannot catch

Catches

  • logic regressions in pure code
  • wrong calculations
  • edge cases in parsing

Cannot catch

  • integration faults
  • config
  • data shape
  • performance

Both lists come from the site's test class definitions.

Incident types this class catches

IncidentStageLimitation on that pageBasis
Partial rollback leaves inconsistent statepre-mergeA unit test of the table of contents patch does not cover a deploy that reached seven of eight order-routing servers, so it misses the old code still running on the eighth server.cited in [1]
Memory leak in new releasepre-mergeA goroutine leak check in unit tests covers only the code paths the tests run, so it misses a leak that appears only under high production traffic, as in report.by design analysis
Password reset link brokenpre-mergeA unit test of the reset link builder accepts whatever base URL it is given, so it misses an application URL left on the old host.by design analysis
Cache invalidation stormpre-mergeA unit test can show that the settings cache change expires every entry, yet it cannot show the replication delay that rewriting all entries causes in dependent services.by design analysis
Checkout blocked for segmentpre-mergeUnit tests on the postcode validator cover only the countries in their fixtures, so the 3.10.22 bump passes when Argentina is absent from them.by design analysis
Email or SMS provider template brokenpre-mergeA unit test that renders PASSWORD_RESET_CONFIRM against its variable list catches lostPasswordUrl, but it does not run the deployed email script version that sent the empty signing invitations.by design analysis
Soft-delete flag ignoredpre-mergeA unit test of the trial expiration escalation job starts only from the account states the test sets up, so it misses accounts that a separate deletion batch had already set to deactivated in production.by design analysis
Subscription renewal skippedpre-mergeA unit test of the renewal error handler covers only the gateway error responses its author feeds it, so it misses a response shape that reaches the null pointer path.by design analysis
Tax calculation regressionpre-mergeA unit test whose expected Estonian tax amount was written at the old 20% rate keeps passing after the rate change, so the stale 0.2 multiplier survives unless the expected value comes from the current rate table.by design analysis

For cited rows the number shows how many reports on that incident page name this class.

Only on this page

Unit tests are cited most often for Partial rollback leaves inconsistent state: 1 public report on that page name this class. The limitation stated there: A unit test of the table of contents patch does not cover a deploy that reached seven of eight order-routing servers, so it misses the old code still running on the eighth server.

Catalogue entries that name this class

From their own documentation or site, with the verification date. Alphabetical. Not a recommendation.

Tools whose documentation names these test classes

Providers whose site names these test classes

Questions

What is Unit tests?

Unit tests: tests of one function or class in isolation. They run at the pre-merge stage.

What does Unit tests catch?

Unit tests catch logic regressions in pure code, wrong calculations and edge cases in parsing. 9 published incident types list this class.

What can Unit tests not catch?

Unit tests cannot catch integration faults, config, data shape and performance.

How is Unit tests different from other test classes?

Unit tests share the pre-merge stage with contract tests; they differ in what each catches.

Can Unit tests run in a production environment?

Unit tests run at the pre-merge stage, before production traffic.

Which incidents is Unit tests used for?

cache invalidation storm, checkout blocked for segment, email or SMS provider template broken, memory leak in new release, partial rollback leaves inconsistent state, password reset link broken, soft-delete flag ignored, subscription renewal skipped and tax calculation regression. 1 of 9 links are cited in a public report; the rest are by design analysis.