Release Incident Playbook

Security tests: what they catch and the release incidents they cover

Security tests (automated and manual checks for auth, authorisation, secrets and injection) run at the pre-deploy stage.

3 published incident types list security tests as a class that catches them; 1 of these links are cited in a public report and 2 are by design analysis.

stage: pre-deploy · 3 published incident types · 1 cited

What security tests catch and cannot catch

Catches

  • permission escalation
  • MFA bypass
  • secrets in logs
  • CORS and CSP mistakes

Cannot catch

  • performance
  • data migration faults

Both lists come from the site's test class definitions.

Incident types this class catches

IncidentStageLimitation on that pageBasis
Permission escalation via role changepre-deployA security test covers only the access scenarios it lists, and guest and external accounts gaining every internal project after a synchronization run sat outside the scenarios that existed before the follow-up.cited in [1]
CORS policy blocks frontendpre-deployA CORS policy check validates the server's allowed origins and headers but not a new request header, such as baggage, that an upgraded browser component starts to send.by design analysis
CSP blocks required scriptpre-deployA check that reads only the Content-Security-Policy response header passes, because that header carried a valid nonce while the rendered HTML had lost it.by design analysis

For cited rows the number shows how many reports on that incident page name this class.

Only on this page

Security tests are cited most often for Permission escalation via role change: 1 public report on that page name this class. The limitation stated there: A security test covers only the access scenarios it lists, and guest and external accounts gaining every internal project after a synchronization run sat outside the scenarios that existed before the follow-up.

Catalogue entries that name this class

From their own documentation or site, with the verification date. Alphabetical. Not a recommendation.

Tools whose documentation names these test classes

Providers whose site names these test classes

Questions

What is Security tests?

Security tests: automated and manual checks for auth, authorisation, secrets and injection. They run at the pre-deploy stage.

What does Security tests catch?

Security tests catch permission escalation, MFA bypass, secrets in logs and CORS and CSP mistakes. 3 published incident types list this class.

What can Security tests not catch?

Security tests cannot catch performance and data migration faults.

How is Security tests different from other test classes?

Security tests share the pre-deploy stage with contract tests, integration tests, end-to-end tests, migration tests, load and performance tests, configuration and environment tests, compatibility tests and chaos and resilience tests; they differ in what each catches.

Can Security tests run in a production environment?

Security tests run at the pre-deploy stage, before production traffic.

Which incidents is Security tests used for?

cORS policy blocks frontend, cSP blocks required script and permission escalation via role change. 1 of 3 links are cited in a public report; the rest are by design analysis.