Release Incident Playbook

Cache invalidation storm: symptoms, cause, and the test that catches it

A deploy flushes or invalidates caches at once and origin load spikes.

Cache invalidation storm is documented by 2 accepted public reports from 2 organisations, 2 of them published in the last 5 years.

  • Likelihood: low (2 public reports in 5 years)
  • Impact: medium (2 of 2 reports: availability)

2 public reports · verified 2026-10-21

Symptoms

User-visible

  • Web request failures reached about 40% at peak after a deployment expired every user's settings cache.[1]
  • About 43% of API requests failed during the same cache invalidation storm.[1]
  • Git operations over HTTP failed at about 6%, while operations over SSH were not affected.[1]
  • Client boot requests took much longer than usual and often failed, so many users could not connect.[2]
  • Users without an already booted client could not connect while boot requests were throttled.[2]

Monitoring-visible

  • Replication delays cascaded to services that depend on the rewritten settings cache.[1]
  • The cache hit rate dropped as cache nodes were replaced or flushed.[2]
  • A database keyspace holding channel membership sharded by user carried more load than usual.[2]
  • Most queries to the channel-membership-by-user tables timed out.[2]
  • Pages, alerts, internal reports and user tickets arrived at about the same time just after 6 a.m. Pacific Time.[2]

Root causes

  1. stage: code A deployment meant to reduce writes to the user settings cache contained a bug that expired every user's entry, which was then recalculated and rewritten.[1]
  2. stage: infra A percentage-based rollout of a service discovery agent upgrade restarted agents on cache hosts, so departing cache nodes were replaced with empty ones and rejoining nodes were flushed.[2]
  3. stage: code A cache-fronted scatter query listing group message members had to query every shard once the hit rate fell, and database read load grew superlinearly with the cache miss rate.[2]

Public examples

  1. [1]GitHub, 2026-04-08. A GitHub deployment meant to reduce writes to the user settings cache contained a bug that expired every user's cache at once; recalculating and rewriting the entries caused replication delays across dependent services until the deployment was rolled back. primary GitHub availability report: March 2026
  2. [2]Slack, 2022-04-26. During a percentage-based rollout of a Consul agent upgrade at Slack, agent restarts on memcached hosts caused cache nodes to be replaced or flushed, the cache hit rate dropped and scatter queries for group DM membership overloaded a Vitess keyspace, so many users could not connect. primary Slack's Incident on 2-22-22

Test classes that catch it

Test classStageLimitationBasis
Load and performance testspre-deployA load test sets the cache miss rate itself, so it misses the functional bug in report [1] that expired every user's settings entry at the moment of deployment.by design analysis
Chaos and resilience testspre-deployA drill that restarts one cache host at a time keeps the hit rate near normal, while the rollout in report [2] restarted agents across a percentage of cache hosts at once.by design analysis
Unit testspre-mergeA unit test can show that the settings cache change expires every entry, yet it cannot show the replication delay that rewriting all entries causes in dependent services.by design analysis

Risk

  • Likelihood: low (2 public reports in 5 years)
  • Impact: medium (2 of 2 reports: availability)
Counts behind the grades
Accepted public reports2
Published in the last 5 years2
Impact stated: availability2

How the grades are computed

Pre-release checklist

  • PR-01
  • PR-02
  • PR-03
  • PR-04
  • PR-05
  • PR-06
  • PR-07

First 30 minutes

  • FM-01
  • FM-02
  • FM-03
  • FM-04
  • FM-05
  • FM-06
  • FM-07

Download the checklist

What catches this: catalogue entries

Listed because their own documentation or site names load and performance tests, chaos and resilience tests, unit tests on the verification date. Alphabetical. Not a recommendation.

Tools whose documentation names these test classes

Providers whose site names these test classes

Only on this page

Load tests are the class aimed at cache storms, yet the storm in report [1] began with a functional bug. A deployment meant to reduce writes to the user settings cache expired every user's entry instead. The recalculation and rewrite of every entry caused replication delays across dependent services. A load test that chooses its own miss rate never runs that write path, so a unit test on the expiry behaviour has to sit beside it.

In report [2], database read load grew superlinearly with the cache miss rate behind one scatter query. A load test run at a partial miss rate therefore understates the load of a full flush. Cache invalidation storm needs load measured up to a 100 percent miss rate, the case where every node is flushed.

Questions

What causes cache invalidation storm?

In one report, a deployment meant to reduce cache writes expired every user's settings entry at once [1]. In the other, agent restarts during an upgrade rollout replaced or flushed cache nodes [2].

What are the symptoms of cache invalidation storm?

Users see web request failures of about 40% and API failures of about 43% [1], or client boots that fail to connect [2]. Monitoring shows replication delays [1], a falling cache hit rate and timed-out shard queries [2].

Which test class catches cache invalidation storm, and what does it miss?

Load and performance tests, chaos and resilience tests and unit tests are listed, all with the by-design label. A load test misses the functional bug that expired every settings entry during deployment [1].

Which public reports document cache invalidation storm?

Two primary reports are accepted, published in April 2022 and April 2026. One names a code-stage cache expiry bug [1] and one names an infra-stage agent upgrade rollout [2].

How long did it take to detect cache invalidation storm in the public reports?

Neither report gives a time to detect in minutes. In report [2], pages, alerts and user tickets arrived at about the same time just after 6 a.m. Pacific Time.

Should a team roll back or fix forward when cache invalidation storm happens?

Report [1] rolled back the faulty deployment immediately. Report [2] fixed forward by throttling client boots and changing the query to read missing data from replicas.

Release Incident Playbook (2026). Cache invalidation storm: symptoms, cause, and the test that catches it, content version 1.0.0. https://releasefailures.com/incidents/cache-invalidation-storm/. Accessed 2026-10-21.