Contract tests: what they catch and the release incidents they cover
Contract tests (consumer and provider agree on request and response shape before deploy) run at the pre-merge and pre-deploy stage.
9 published incident types list contract tests as a class that catches them; 0 of these links are cited in a public report and 9 are by design analysis.
What contract tests catch and cannot catch
Catches
- schema drift between services
- provider API field changes
- dropped API versions
Cannot catch
- runtime data values
- performance
- auth policy
Both lists come from the site's test class definitions.
Incident types this class catches
| Incident | Stage | Limitation on that page | Basis |
|---|---|---|---|
| Provider API change breaks contract | pre-merge | A contract test pins request and response shape, so it would not flag the branch deletion change where URLs, payloads and headers stayed the same. | by design analysis |
| Schema drift between services | pre-merge | Contract tests that pair the new consumer with the provider from the same commit miss the rolling deployment window, where the old provider still serves the new consumer. | by design analysis |
| SDK upgrade changes behaviour | pre-merge | A contract on outgoing request headers flags the new x-amz-checksum-mode header only when it is checked against each third-party provider's accepted headers, not only against the vendor's API. | by design analysis |
| Alerts silent after metric rename | pre-merge | A contract test on the list of exported metric names misses an added label that makes an alert query return nothing while the metric name stays the same. | by design analysis |
| Analytics event schema broken | pre-merge | A contract test runs on the sender's own release, so it does not run when a vendor changes the destination integration that added the event name prefix. | by design analysis |
| API version dropped while old app live | pre-deploy | A contract suite covers only the client versions registered in it, so a pinned infrastructure-as-code plugin or a customer's own automation that never published a contract still fails when its API version is retired. | by design analysis |
| Double charge after deploy | pre-merge | A contract test flags the removed eligible_for_card_updater response field, but the timing-dependent re-execution in the transaction workflow changed no request or response shape, so a contract test passes on it. | by design analysis |
| In-app purchase receipt validation fails | pre-merge | A contract on serverVerificationData flags the switch from a base64 receipt to a JWT, but an error from the purchase-type endpoint that keeps the expected shape and carries a wrong value passes it. | by design analysis |
| Rolling deploy mixed versions | pre-merge | A contract test on field shapes passes when a new pipeline state value is added, because the field is unchanged and only the runtime value is new. | by design analysis |
For cited rows the number shows how many reports on that incident page name this class.
Only on this page
No public report on a published incident page names contract tests explicitly; every link on this page is by design analysis. The incident with the most public reports that lists this class is Provider API change breaks contract.
Catalogue entries that name this class
From their own documentation or site, with the verification date. Alphabetical. Not a recommendation.
Tools whose documentation names these test classes
Providers whose site names these test classes
- Full Scale contract source verified 2026-10-21
- QA Madness contract source verified 2026-10-21
- QASource contract source verified 2026-10-21
- Quality Island contract source verified 2026-10-21
Questions
What is Contract tests?
Contract tests: consumer and provider agree on request and response shape before deploy. They run at the pre-merge and pre-deploy stage.
What does Contract tests catch?
Contract tests catch schema drift between services, provider API field changes and dropped API versions. 9 published incident types list this class.
What can Contract tests not catch?
Contract tests cannot catch runtime data values, performance and auth policy.
How is Contract tests different from other test classes?
Contract tests share the pre-merge and pre-deploy stage with unit tests, integration tests, end-to-end tests, migration tests, load and performance tests, security tests, configuration and environment tests, compatibility tests and chaos and resilience tests; they differ in what each catches.
Can Contract tests run in a production environment?
Contract tests run at the pre-merge and pre-deploy stage, before production traffic.
Which incidents is Contract tests used for?
alerts silent after metric rename, analytics event schema broken, aPI version dropped while old app live, double charge after deploy, in-app purchase receipt validation fails, provider API change breaks contract, rolling deploy mixed versions, schema drift between services and sDK upgrade changes behaviour. 0 of 9 links are cited in a public report; the rest are by design analysis.