Cache invalidation storm: pre-release and first 30 minutes checklists

A deploy flushes or invalidates caches at once and origin load spikes.

The pre-release checklist maps each item to a test class from the incident page. The first-30-minutes checklist covers detection, containment, the rollback decision and customer communication.

7 pre-release items · 7 first-30-minutes items · verified 2026-10-21

Pre-release checklist

PR-01Add a unit test that runs the changed settings cache write path against existing entries and asserts their expiry times do not change. unit
PR-02Add a unit test that asserts the killswitch for the caching mechanism restores the previous write path when switched off. unit
PR-03Expire all user settings cache entries in a production-shaped environment and measure replication delay on dependent services during the rewrite, and fail when replication delay exceeds the dependent services' timeout. load
PR-04Run each cache-fronted scatter query at stepped cache miss rates up to 100 percent and record read load on every shard at each step, and fail when any shard exceeds its read capacity. load
PR-05Confirm the database behind the channel membership cache serves full-miss read load from replicas within its query timeout. load
PR-06Restart the service discovery agent on a share of cache hosts in staging and confirm those cache nodes are neither replaced with empty nodes nor flushed. chaos
PR-07Throttle client boot requests in a drill and confirm the limit can be raised in steps as the cache refills. chaos

First 30 minutes

FM-01Check whether the cache hit rate fell or cache entries expired at the moment a deployment or agent rollout started.
FM-02Roll back a deployment that touched the cache write path at once, because every minute of rewrites adds replication delay.
FM-03Pause any agent upgrade rollout that is still restarting cache hosts.
FM-04Throttle client boot requests so read load on the database behind the cache stays within capacity.
FM-05Change the overloaded cache-fronted query to read only the missing data and to use replicas.
FM-06Raise the boot rate limit in steps as the cache hit rate recovers.
FM-07Post a status update that lists the failing surfaces and their error rates, such as web requests, API requests and Git over HTTP.

Download the checklist

Pre-release checklist (Markdown)1.6 KB · v1.0.0
First 30 minutes checklist (Markdown)1.0 KB · v1.0.0

Only in this checklist

Every test class in this checklist also appears in at least one sibling checklist in the group; the item texts differ: Add a unit test that runs the changed settings cache write path against existing entries and asserts their expiry times do not change.

Questions

What should be checked before a release to prevent Cache invalidation storm?

7 pre-release items, PR-01 to PR-07. PR-01: Add a unit test that runs the changed settings cache write path against existing entries and asserts their expiry times do not change.

Is there a rollback plan for Cache invalidation storm?

The first-30-minutes checklist has 7 items. FM-01: Check whether the cache hit rate fell or cache entries expired at the moment a deployment or agent rollout started.

What should a team do in the first 30 minutes after Cache invalidation storm is detected?

The first-30-minutes checklist has 7 items. FM-01: Check whether the cache hit rate fell or cache entries expired at the moment a deployment or agent rollout started.

Is a Cache invalidation storm checklist still needed if deployment is automated?

Yes. Automation runs the steps; the checklist names what the tests must cover: unit tests, load and performance tests and chaos and resilience tests.

Which pipeline stages does the Cache invalidation storm checklist cover?

The pre-release items map to the pre-merge, pre-deploy and post-deploy stages.

In what formats can the Cache invalidation storm checklist be downloaded?

Both checklists download as Markdown files: /downloads/cache-invalidation-storm-pre-release.md and /downloads/cache-invalidation-storm-first-30-minutes.md, licensed CC BY 4.0.