Release Incident Playbook

Config change without restart semantics: symptoms, cause, and the test that catches it

A config edit is applied to some instances and not others, producing inconsistent behaviour.

Config change without restart semantics is documented by 2 accepted public reports from 2 organisations, 2 of them published in the last 5 years.

  • Likelihood: low (2 public reports in 5 years)
  • Impact: medium (2 of 2 reports: availability)

2 public reports · verified 2026-10-21

Symptoms

User-visible

  • Port forwarding stopped working in the web client of the cloud development environment service after a secret rotation.[2]
  • Creation and start failures for cloud development environments increased in all regions.[2]
  • Two VM pools in West Europe kept failing after the service was restarted elsewhere.[2]
  • Internet users saw error pages indicating a failure inside the provider network.[1]
  • Core traffic recovered for periods and then failed again while nodes produced different feature files.[1]

Monitoring-visible

  • Error-rate alerts did not fire on the drop in traffic to the port forwarding backend.[2]
  • A monitor alerted on a separate downstream component at 17:18 UTC.[2]
  • VM pools were heavily drained.[2]
  • 5xx error volume spiked above baseline and fluctuated as good and bad configuration files were distributed every five minutes.[1]
  • The first symptom looked like a degraded response rate in the key-value service.[1]

Root causes

  1. stage: config Downstream components picked up a rotated secret only after a process restart, while the team expected the new secret to load without intervention.[2]
  2. stage: release-process A step in the secret rotation checklist was missed, and one service instance in West Europe was not rotated along with the rest.[2]
  3. stage: config A database permissions change was rolled out gradually across a cluster, so updated nodes generated the feature file differently from nodes not yet updated.[1]
  4. stage: code A query without a database name filter returned duplicate columns on updated nodes only, which doubled the feature file past the proxy's 200-feature limit.[1]

Public examples

  1. [1]Cloudflare, 2025-11-18. A database permissions change was rolled out gradually across a ClickHouse cluster, so the Bot Management feature file was generated correctly on some nodes and wrongly on updated nodes, and the network alternated between good and bad files until every node produced the oversized file. primary Cloudflare outage on November 18, 2025
  2. [2]GitHub, 2022-11-02. After a Codespaces secret rotation with a missed step, some downstream components kept using the old secret because they only loaded the new one on process restart, and one service instance in West Europe was not rotated at all, so port forwarding and later codespace creation failed. primary GitHub Availability Report: October 2022

Test classes that catch it

Test classStageLimitationBasis
Configuration and environment testspost-deployA configuration check confirms the new secret is stored, but it does not show which secret version each running process holds until that process restarts.by design analysis
Integration testspre-deployAn integration run against one database node sees either the old or the new permissions, never the mixed state that a gradual rollout creates across the cluster.by design analysis
Canary and progressive delivery checkspost-deployA canary that watches error rates misses a fault that shows as a traffic drop, which is how the port forwarding failure appeared.by design analysis

Risk

  • Likelihood: low (2 public reports in 5 years)
  • Impact: medium (2 of 2 reports: availability)
Counts behind the grades
Accepted public reports2
Published in the last 5 years2
Impact stated: availability2

How the grades are computed

Pre-release checklist

  • PR-01
  • PR-02
  • PR-03
  • PR-04
  • PR-05
  • PR-06
  • PR-07
  • PR-08
  • PR-09

First 30 minutes

  • FM-01
  • FM-02
  • FM-03
  • FM-04
  • FM-05
  • FM-06
  • FM-07
  • FM-08

Download the checklist

What catches this: catalogue entries

Listed because their own documentation or site names configuration and environment tests, integration tests, canary and progressive delivery checks on the verification date. Alphabetical. Not a recommendation.

Tools whose documentation names these test classes

Providers whose site names these test classes

Only on this page

In report [2], config change without restart semantics took 247 minutes to detect. The secret rotation ran at 23:14 UTC. An internal report flagged the fault at 03:21 UTC on 28 September. Error-rate alerts stayed quiet because the fault appeared as a drop in traffic to the port forwarding backend. The secret was exchanged for a token cached for up to 24 hours, and components loaded the new secret only after a process restart.

Report [1] detected its gradual permissions rollout 26 minutes after the change, through the first automated test. The follow-ups in report [2] add traffic-level anomaly alerts and monitoring of which secret versions are in use per component.

Questions

What causes config change without restart semantics?

In one report, components loaded a rotated secret only on process restart, and one West Europe instance was not rotated [2]. In another report, a permissions change rolled out gradually across a database cluster, so updated and old nodes generated different feature files [1].

What are the symptoms of config change without restart semantics?

Port forwarding and environment creation failed while some components still used the old secret [2]. Core traffic recovered and failed again as good and bad files were distributed every five minutes [1].

Why did tests not catch config change without restart semantics?

Error-rate alerts did not fire on the drop in traffic to the port forwarding backend [2]. An integration run against one database node never sees the mixed permission state of a gradual rollout.

How long did it take to detect config change without restart semantics in the public reports?

One report took 247 minutes from the 23:14 UTC rotation to an internal report at 03:21 UTC [2]. The other detected the fault through an automated test 26 minutes after the change [1].

What should be checked before a release to prevent config change without restart semantics?

PR-01 lists the secret version each process holds after a rotation. PR-05 runs the feature file query against a cluster with only part of the nodes on the new permissions.

Should a team roll back or fix forward when config change without restart semantics happens?

One report stopped propagation of new files and deployed a known-good file globally [1]. The other ran the missed rotation step and restarted the service in all regions [2].

Release Incident Playbook (2026). Config change without restart semantics: symptoms, cause, and the test that catches it, content version 1.0.0. https://releasefailures.com/incidents/config-change-without-restart-semantics/. Accessed 2026-10-21.