Infrastructure and deploy release incidents

Infrastructure and deploy: 6 published incident types backed by 19 public reports.

The root-cause stage named most often in this group is config, in 5 of 6 incident types.

6 incident types published · 2 withheld · 19 public reports

Incident types in this group

Infrastructure and deploy


Autoscaling policy misconfigured	Likelihood: medium (4 public reports in 5 years)	Impact: medium (5 of 5 reports: availability)	5
Container image base change	Likelihood: low (2 public reports in 5 years)	Impact: medium (3 of 3 reports: availability)	3
Disk full from new log volume	Likelihood: low (2 public reports in 5 years)	Impact: medium (2 of 3 reports: availability)	3
DNS change propagation outage	Likelihood: medium (3 public reports in 5 years)	Impact: medium (3 of 3 reports: availability)	3
IAM policy change blocks service	Likelihood: medium (3 public reports in 5 years)	Impact: medium (3 of 3 reports: availability)	3
Rolling deploy mixed versions	Likelihood: not published (1 public report in 5 years)	Impact: medium (1 of 2 reports: availability)	2

Withheld in this group: Health check passes on broken service, Cron timezone shift. Reasons per route

Risk grid

Risk grid: count of incident types per likelihood and impact cell, from data/risk.json
Likelihood \ Impact	Impact: high	Impact: medium	Impact: low
Likelihood: high	0	0	0
Likelihood: medium	0	3	0
Likelihood: low	0	2	0
Impact unrated or likelihood not graded	1

Table equivalent: incidents per cell

Incident	Likelihood	Impact
Autoscaling policy misconfigured	medium	medium
Container image base change	low	medium
Disk full from new log volume	low	medium
DNS change propagation outage	medium	medium
IAM policy change blocks service	medium	medium
Rolling deploy mixed versions	not-published	medium

Root-cause stages

Stage	Incident types naming it
config	5
release-process	3
infra	3
third-party	2
build	1
code	1

Median time to detection across the group: 92.5 minutes from 6 reports that state both the change time and the detection time (values above the site-wide 95th percentile excluded).

Only on this page

Container image base change is the incident type in this group whose root causes do not include the config stage; its causes are introduced at third-party and infra and build.

Group checklist

The first pre-release item of each published incident type in the group.

GR-01Autoscaling policy misconfigured: Run a pre-production load test at peak-hours request volume against every change to the API auto-scaling rule and confirm the instance count rises above the off-hours baseline. load
GR-02Container image base change: Start each rebuilt CI image on the same hosted runner infrastructure that customer jobs use and confirm a job inside it builds an image. integration
GR-03Disk full from new log volume: Measure log bytes written per request before and after the logging change and confirm the projected daily volume at peak traffic fits the free disk space. load
GR-04DNS change propagation outage: Read the infrastructure plan for the production apply and confirm no DNS record or TLS certificate is destroyed before its replacement exists. config
GR-05IAM policy change blocks service: Before enabling an account-level public access block, list every bucket that receives public-read uploads and confirm the block does not override their bucket-level ACL policies. config
GR-06Rolling deploy mixed versions: Run the pre-production suite on a cluster where old and new application servers share traffic, and fail on any request that ends a logged-in session. integration

Questions

What causes Infrastructure and deploy incidents?

Published incident types in this group name these root-cause stages: config (5), release-process (3), infra (3), third-party (2), build (1), code (1).

Which pipeline stage introduces most Infrastructure and deploy incidents?

The config stage, named in 5 of 6 published incident types in the group.

Which test classes catch Infrastructure and deploy incidents?

Test classes listed on the group's incident pages: canary and progressive delivery checks (5), configuration and environment tests (4), load and performance tests (2), integration tests (2), synthetic monitoring (2), chaos and resilience tests (1), contract tests (1).

How quickly are Infrastructure and deploy incidents detected in public reports?

The median time to detection is 92.5 minutes across 6 reports that state both the change time and the detection time.

How likely are Infrastructure and deploy incidents compared with each other?

Autoscaling policy misconfigured: medium likelihood, medium impact; Container image base change: low likelihood, medium impact; Disk full from new log volume: low likelihood, medium impact; DNS change propagation outage: medium likelihood, medium impact; IAM policy change blocks service: medium likelihood, medium impact; Rolling deploy mixed versions: not-published likelihood, medium impact.

What should a release checklist for Infrastructure and deploy include?

The group checklist collects the first pre-release item of each of the 6 published incident types; each item names the test class that checks it.