Performance and capacity release incidents

Performance and capacity: 6 published incident types backed by 17 public reports.

The root-cause stage named most often in this group is code, in 4 of 6 incident types.

6 incident types published · 3 withheld · 17 public reports

Incident types in this group

Performance and capacity


Background job queue backlog	Likelihood: medium (3 public reports in 5 years)	Impact: low (2 of 3 reports: degraded)	3
Cache invalidation storm	Likelihood: low (2 public reports in 5 years)	Impact: medium (2 of 2 reports: availability)	2
Cold start latency regression	Likelihood: low (2 public reports in 5 years)	Impact: low (2 of 2 reports: degraded)	2
Connection pool exhausted	Likelihood: medium (3 public reports in 5 years)	Impact: medium (3 of 4 reports: availability)	5
Memory leak in new release	Likelihood: low (2 public reports in 5 years)	Impact: medium (2 of 2 reports: availability)	3
Thundering herd on restart	Likelihood: low (2 public reports in 5 years)	Impact: medium (2 of 2 reports: availability)	2

Withheld in this group: N+1 queries after refactor, Unbounded query from new filter, CDN cache miss after header change. Reasons per route

Risk grid

Risk grid: count of incident types per likelihood and impact cell, from data/risk.json
Likelihood \ Impact	Impact: medium	Impact: low
Likelihood: high	0	0
Likelihood: medium	1	1
Likelihood: low	3	1
Impact unrated or likelihood not graded

Table equivalent: incidents per cell

Incident	Likelihood	Impact
Background job queue backlog	medium	low
Cache invalidation storm	low	medium
Cold start latency regression	low	low
Connection pool exhausted	medium	medium
Memory leak in new release	low	medium
Thundering herd on restart	low	medium

Root-cause stages

Stage	Incident types naming it
code	4
infra	3
release-process	2
third-party	1
config	1

Median time to detection across the group: 11 minutes from 2 reports that state both the change time and the detection time (values above the site-wide 95th percentile excluded).

Only on this page

Cold start latency regression is the incident type in this group whose root causes do not include the code stage; its causes are introduced at third-party.

Group checklist

The first pre-release item of each published incident type in the group.

GR-01Background job queue backlog: After a major-version upgrade of any shared library, start a background worker in an integration environment and confirm each scheduler thread is still running after startup. integration
GR-02Cache invalidation storm: Add a unit test that runs the changed settings cache write path against existing entries and asserts their expiry times do not change. unit
GR-03Cold start latency regression: Measure P90 cold start duration over 100 invokes of a minimal function on the new runtime version and on the current one before switching the runtime. load
GR-04Connection pool exhausted: Run each new background job in the integration suite and confirm the open database connection count returns to its starting value when the job ends. integration
GR-05Memory leak in new release: Confirm the canary receives at least the per-pod request rate of production peak before an authorization service build graduates. canary
GR-06Thundering herd on restart: Restart every backend pod at once in staging under peak-shaped traffic and confirm client reconnection attempts follow exponential backoff. chaos

Questions

What causes Performance and capacity incidents?

Published incident types in this group name these root-cause stages: code (4), infra (3), release-process (2), third-party (1), config (1).

Which pipeline stage introduces most Performance and capacity incidents?

The code stage, named in 4 of 6 published incident types in the group.

Which test classes catch Performance and capacity incidents?

Test classes listed on the group's incident pages: load and performance tests (6), canary and progressive delivery checks (5), integration tests (2), chaos and resilience tests (2), unit tests (2).

How quickly are Performance and capacity incidents detected in public reports?

The median time to detection is 11 minutes across 2 reports that state both the change time and the detection time.

How likely are Performance and capacity incidents compared with each other?

Background job queue backlog: medium likelihood, low impact; Cache invalidation storm: low likelihood, medium impact; Cold start latency regression: low likelihood, low impact; Connection pool exhausted: medium likelihood, medium impact; Memory leak in new release: low likelihood, medium impact; Thundering herd on restart: low likelihood, medium impact.

What should a release checklist for Performance and capacity include?

The group checklist collects the first pre-release item of each of the 6 published incident types; each item names the test class that checks it.