IAM policy change blocks service: symptoms, cause, and the test that catches it

A permission change removes a service's access to a queue, bucket or secret.

IAM policy change blocks service is documented by 3 accepted public reports from 3 organisations, 3 of them published in the last 5 years.

Likelihood: medium (3 public reports in 5 years)
Impact: medium (3 of 3 reports: availability)

3 public reports · verified 2026-10-21

Symptoms

User-visible

CI jobs queued and then timed out while waiting to acquire a hosted runner.[1]
Creating or resuming hosted development environments failed in all regions.[1]
Instructors could not use the test export and import functions.[2]
Customers saw 403 Forbidden or access denied errors when downloading operating system and Ruby packages from a hosted package registry, from about 01:53 to 04:24 UTC.[3]

Monitoring-visible

The cloud metrics service counted 967 4xx errors on the storage bucket used for test exports.[2]
Application logs showed the upload function failing with ERROR::FILE-003, failed to upload file to the export bucket.[2]
The service desk received 16 cases about the export failure from 15 instructors across 13 colleges and universities and 2 high schools.[2]
Command-line uploads with the same credentials succeeded from a developer workstation while the services failed.[2]
Customer reports of package download failures from 01:53 UTC were handled as separate per-customer investigations until the incident was declared at 03:40 UTC.[3]

Root causes

stage: third-party After a loss of telemetry, security policies were applied by mistake to backend storage accounts at the underlying compute provider, and those policies blocked access to VM metadata, so VM create, delete and reimage operations failed.[1]
stage: config A bug in an internal security tool meant only to run hardening checks applied an account-level public access block across the whole cloud account.[2]
stage: config The account-level public access block overrode bucket-level ACL policies, so public-read uploads from the scheduler, document and content extraction APIs failed.[2]
stage: config An infrastructure-as-code update to the bucket policy for customer packages overwrote the existing access statements instead of merging with them, which removed the permissions package downloads needed.[3]

Public examples

[1]GitHub, 2026-03-11. Security policies were mistakenly applied to backend storage accounts in GitHub's underlying compute provider, the policies blocked access to VM metadata, and VM operations failed so GitHub Actions hosted runners and Codespaces were unavailable in all regions until the policy changes were rolled back. primary GitHub availability report: February 2026
[2]W. W. Norton & Company, 2025-11-07. An internal security tool enabled an AWS account-level S3 public access block, which blocked the public-read S3 uploads that Testmaker export and import services relied on until the account-level configuration was rolled back about 15 hours later. primary Incident: Testmaker Export Outage | Norton Digital Product Guidebook
[3]Buildkite, 2025-11-07. A Terraform update to the S3 bucket policy for a bucket of customer packages overwrote the existing access policies instead of merging with them, so Debian and RubyGems package downloads returned 403 errors until the previous bucket policy was redeployed. primary Buildkite Status - Increased 403 Errors on Package Registries package access

Test classes that catch it

Test class	Stage	Limitation	Basis
Synthetic monitoring	post-deploy	A synthetic check detects the failed export or package download only after the account-level block or bucket policy is live, so it shortens detection but does not stop the change.	cited in [2]
Configuration and environment tests	pre-deploy	A configuration check runs only on changes that pass through the release pipeline, so it misses a security tool that wrote an account-level setting directly and a policy change made inside the compute provider.	by design analysis

Risk

Likelihood: medium (3 public reports in 5 years)
Impact: medium (3 of 3 reports: availability)

Counts behind the grades
Accepted public reports	3
Published in the last 5 years	3
Impact stated: availability	3

How the grades are computed

Pre-release checklist

PR-01Before enabling an account-level public access block, list every bucket that receives public-read uploads and confirm the block does not override their bucket-level ACL policies. config
PR-02Read the infrastructure plan for every bucket policy change and confirm each existing access statement still appears in the planned policy. config
PR-03After a bucket policy change in staging, evaluate the effective policy for the registry service role and confirm it still allows reads of every package format. config
PR-04Run a public-read upload from the scheduler, document and content extraction service roles in a staging account that carries the production account-level settings. config
PR-05Confirm the credentials of every security hardening tool are read-only and cannot change account-level settings in production. config
PR-06Add a synthetic check that runs the test export workflow against production on a schedule and alerts on a failed upload. synthetic
PR-07Add a synthetic check that downloads one package from each repository format and alerts on any 403 response. synthetic

First 30 minutes

FM-01Search service logs and bucket metrics for 403 or access denied responses and record the first timestamp.
FM-02List every policy and account-level permission change in the hours before the first error, including changes made by security tools.
FM-03Group repeated customer reports of the same 403 error into one incident instead of separate per-customer investigations.
FM-04Redeploy the previous bucket policy or revert the account-level setting if its change time precedes the first error.
FM-05Test the fix with the service role that failed, not with a personal workstation credential that may still succeed.
FM-06Post a status update naming the blocked function, such as exports or package downloads, and the time the errors began.
FM-07Freeze further permission changes on the affected account until uploads and downloads succeed again.

Download the checklist

Pre-release checklist (Markdown)1.7 KB · v1.0.0
First 30 minutes checklist (Markdown)1.2 KB · v1.0.0

What catches this: catalogue entries

Listed because their own documentation or site names synthetic monitoring, configuration and environment tests on the verification date. Alphabetical. Not a recommendation.

Tools whose documentation names these test classes

Checkly synthetic source verified 2026-10-21
Datadog Synthetic Monitoring synthetic source verified 2026-10-21
Flipt config source verified 2026-10-21
Grafana Cloud Synthetic Monitoring synthetic source verified 2026-10-21
Grafana k6 synthetic source verified 2026-10-21
GrowthBook config source verified 2026-10-21
LaunchDarkly config source verified 2026-10-21
PostHog config source verified 2026-10-21
Postman synthetic source verified 2026-10-21
Unleash config source verified 2026-10-21
Uptime Kuma synthetic source verified 2026-10-21

Providers whose site names these test classes

No provider in the catalogue names this test class on its site.

Only on this page

Report [2] records a time to detect of 145 minutes. The account-level public access block was applied at 3:00 pm EDT, and the first user reports arrived at 5:25 pm EDT [2]. Developers named the block as the root cause at 10:00 am the next morning, and the lack of an audit trail slowed identification [2]. Uploads with the same credentials kept working from a workstation while the services failed [2].

Report [3] records 40 minutes from the bucket policy update to the first customer reports. In report [3], those reports were handled as separate per-customer investigations until the incident was declared at 03:40 UTC. Report [2] lists synthetic monitoring of user workflows as an early warning and detection system among its action items.

DNS change propagation outage A DNS or load balancer change sends traffic to the wrong target or nowhere.
API key rotation breaks integrations A key rotation invalidates partner integrations that were not notified.
Autoscaling policy misconfigured A scaling change fails to add capacity or scales to zero under load.
Container image base change A base image update changes libraries or users and the service fails to start.
Dependency removed from registry A package or version is pulled from a registry and builds or deploys fail.

Questions

What causes IAM policy change blocks service?

In report [2], an internal security tool applied an account-level public access block that overrode bucket-level ACL policies. In report [3], a bucket policy update overwrote the existing access statements instead of merging with them.

What are the symptoms of IAM policy change blocks service?

Customers saw 403 Forbidden errors on package downloads from about 01:53 to 04:24 UTC [3]. In report [2], the cloud metrics service counted 967 4xx errors on the export bucket.

Which test class catches IAM policy change blocks service, and what does it miss?

Synthetic monitoring carries the cited label, because report [2] names synthetic monitoring of user workflows. It detects the blocked workflow only after the policy change is live.

How long did it take to detect IAM policy change blocks service in the public reports?

Report [2] records 145 minutes and report [3] records 40 minutes. Report [1] gives no time to detect.

What should be checked before a release to prevent IAM policy change blocks service?

Item PR-02 confirms each existing access statement still appears in the planned bucket policy. Item PR-05 confirms security hardening tools cannot change account-level settings in production.

Should a team roll back or fix forward when IAM policy change blocks service happens?

All 3 reports restored service by rolling back the policy change. In report [3], redeploying the previous bucket policy restored downloads at 04:24 UTC.

Release Incident Playbook (2026). IAM policy change blocks service: symptoms, cause, and the test that catches it, content version 1.0.0. https://releasefailures.com/incidents/iam-policy-change-blocks-service/. Accessed 2026-10-21.