Config change without restart semantics: pre-release and first 30 minutes checklists

A config edit is applied to some instances and not others, producing inconsistent behaviour.

The pre-release checklist maps each item to a test class from the incident page. The first-30-minutes checklist covers detection, containment, the rollback decision and customer communication.

9 pre-release items · 8 first-30-minutes items · verified 2026-10-21

Pre-release checklist

PR-01After a secret rotation, list the secret version each component process holds and confirm none still uses the old version. config
PR-02Restart every process that loads the secret only at start, in every region, before the old secret is revoked. config
PR-03Compare the count of service instances with the count of rotated instances per region and confirm the two match. config
PR-04Record the operator and timestamp for each step of the secret rotation checklist and block closing the change while any step is unrecorded. config
PR-05Run the feature file generation query against a cluster with only part of the nodes on the new permissions and confirm every node returns the same column count. integration
PR-06Confirm the feature file generation query filters on the database name so it cannot return duplicate columns. integration
PR-07Confirm the generated feature file stays under the proxy's 200-feature limit before the file is distributed. integration
PR-08Halt the rotation rollout when traffic to the port forwarding backend drops below baseline. canary
PR-09Configure the canary to halt propagation of new feature files when 5xx volume rises above baseline in any five-minute distribution window. canary

First 30 minutes

FM-01Check whether errors alternate between recovery and failure on the same interval as config file distribution.
FM-02List the config or secret version each instance holds and group the failing instances by version.
FM-03Stop propagation of the newly generated config file to the remaining nodes.
FM-04Decide to roll back by deploying the last known-good config file to every node rather than waiting for the gradual rollout to converge.
FM-05If a secret rotation is involved, run the missed rotation step and restart the processes that load the secret only at start, in all regions.
FM-06Match the error fluctuation against the config distribution interval before escalating the event as an attack.
FM-07Publish a status update that names the affected functions and states that recovery may look partial while instances converge.
FM-08Close the incident only after every instance reports the same config version and error rates return to baseline.

Download the checklist

Pre-release checklist (Markdown)1.9 KB · v1.0.0
First 30 minutes checklist (Markdown)1.3 KB · v1.0.0

Only in this checklist

PR-05 is the only item among 5 published checklists in this group that relies on integration tests: Run the feature file generation query against a cluster with only part of the nodes on the new permissions and confirm every node returns the same column count.

Questions

What should be checked before a release to prevent Config change without restart semantics?

9 pre-release items, PR-01 to PR-09. PR-01: After a secret rotation, list the secret version each component process holds and confirm none still uses the old version.

Is there a rollback plan for Config change without restart semantics?

The first-30-minutes checklist has 8 items. FM-01: Check whether errors alternate between recovery and failure on the same interval as config file distribution.

What should a team do in the first 30 minutes after Config change without restart semantics is detected?

The first-30-minutes checklist has 8 items. FM-01: Check whether errors alternate between recovery and failure on the same interval as config file distribution.

Which configuration checks apply to Config change without restart semantics?

PR-01: After a secret rotation, list the secret version each component process holds and confirm none still uses the old version. PR-02: Restart every process that loads the secret only at start, in every region, before the old secret is revoked. PR-03: Compare the count of service instances with the count of rotated instances per region and confirm the two match. PR-04: Record the operator and timestamp for each step of the secret rotation checklist and block closing the change while any step is unrecorded.

Is a Config change without restart semantics checklist still needed if deployment is automated?

Yes. Automation runs the steps; the checklist names what the tests must cover: configuration and environment tests, integration tests and canary and progressive delivery checks.

Which pipeline stages does the Config change without restart semantics checklist cover?

The pre-release items map to the pre-deploy and post-deploy stages.