Release Incident Playbook

Compatibility tests: what they catch and the release incidents they cover

Compatibility tests (the same flow across browsers, OS versions and device classes) run at the pre-deploy stage.

2 published incident types list compatibility tests as a class that catches them; 0 of these links are cited in a public report and 2 are by design analysis.

stage: pre-deploy · 2 published incident types · 0 cited

What compatibility tests catch and cannot catch

Catches

  • crashes on older OS
  • single-browser failures
  • hydration faults

Cannot catch

  • server-side faults
  • load

Both lists come from the site's test class definitions.

Incident types this class catches

IncidentStageLimitation on that pageBasis
Crash on older OS versionpre-deployA compatibility run on a binary from a different build passes, because one reported crash came from the build tooling rather than from app code.by design analysis
Critical path broken in one browserpre-deployA compatibility run on the current iOS browser passes, because the unsupported regular expression broke only versions below 16.4.by design analysis

For cited rows the number shows how many reports on that incident page name this class.

Only on this page

No public report on a published incident page names compatibility tests explicitly; every link on this page is by design analysis. The incident with the most public reports that lists this class is Crash on older OS version.

Catalogue entries that name this class

From their own documentation or site, with the verification date. Alphabetical. Not a recommendation.

Tools whose documentation names these test classes

Providers whose site names these test classes

Questions

What is Compatibility tests?

Compatibility tests: the same flow across browsers, OS versions and device classes. They run at the pre-deploy stage.

What does Compatibility tests catch?

Compatibility tests catch crashes on older OS, single-browser failures and hydration faults. 2 published incident types list this class.

What can Compatibility tests not catch?

Compatibility tests cannot catch server-side faults and load.

How is Compatibility tests different from other test classes?

Compatibility tests share the pre-deploy stage with contract tests, integration tests, end-to-end tests, migration tests, load and performance tests, security tests, configuration and environment tests and chaos and resilience tests; they differ in what each catches.

Can Compatibility tests run in a production environment?

Compatibility tests run at the pre-deploy stage, before production traffic.

Which incidents is Compatibility tests used for?

crash on older OS version and critical path broken in one browser. 0 of 2 links are cited in a public report; the rest are by design analysis.