Release Incident Playbook

Partial rollback leaves inconsistent state: symptoms, cause, and the test that catches it

Only some services are rolled back and the system runs with incompatible versions.

Partial rollback leaves inconsistent state is documented by 3 accepted public reports from 3 organisations, 2 of them published in the last 5 years.

  • Likelihood: low (2 public reports in 5 years)
  • Impact: low (2 of 3 reports: degraded)

3 public reports · verified 2026-10-21

Symptoms

User-visible

  • The API error rate rose in the EU production environment after a regular platform update.[1]
  • On wikis with language variants, the table of contents showed in an incorrect or inconsistent variant.[2]
  • After the rollback, pages cached by the new release had no table of contents at all on all wikis.[2]
  • The order router sent millions of child orders to trading centres over about 45 minutes.[3]
  • The firm took on large unintended long and short positions in many stocks.[3]

Monitoring-visible

  • Alarms warned the team immediately that some services could not initialise the new versions.[1]
  • No automated monitoring detected the missing tables of contents after the rollback.[2]
  • The rollback set off unrelated spurious client error alerts.[2]
  • Engineers reported the missing tables of contents about two hours after the first user reports.[2]

Root causes

  1. stage: release-process The automatic rollback returned only a few services to the previous version while other services could not initialise the new versions.[1]
  2. stage: code Cached page output carried no version, so the rolled-back code could not handle the table of contents marker in cache entries left by the newer version.[2]
  3. stage: release-process New order-routing code reached seven of eight servers, and the eighth kept old code that a repurposed flag activated.[3]
  4. stage: release-process Uninstalling the new code from the seven correctly deployed servers let incoming orders activate the old code on those servers too.[3]

Public examples

  1. [1]Virtuoso, 2024-07-17. A regular platform update degraded the Virtuoso EU production environment, the automatic rollback returned only a few services to the previous version while others could not initialise the new versions, and a manual downgrade at 07:20 UTC restored full availability at 08:10 UTC. primary Increased error rate on our API
  2. [2]Wikimedia Foundation, 2021-11-08. Wikimedia rolled the MediaWiki train back to the previous version because of a table of contents regression, but the parser cache still held output written by the newer version, which the older code could not handle, so pages lost their table of contents on all wikis until the train was rolled forward again. primary Incidents/2021-11-05 TOC language converter
  3. [3]Knight Capital Americas LLC, 2013-10-16. Knight deployed new RLP code to seven of eight SMARS servers, the eighth kept defective Power Peg code that the repurposed flag activated, and when Knight uninstalled the new code from the seven servers during the incident, incoming orders activated the Power Peg code on those servers too. primary In the Matter of Knight Capital Americas LLC, Release No. 70694

Test classes that catch it

Test classStageLimitationBasis
Unit testspre-mergeA unit test of the table of contents patch does not cover a deploy that reached seven of eight order-routing servers, so it misses the old code still running on the eighth server.cited in [2]
Chaos and resilience testspre-deployA rollback drill shows whether every service returns to the previous version, but it does not check functional defects, so old code that a repurposed flag still activates passes the drill.by design analysis
Configuration and environment testspost-deployA version check across all eight servers finds the one that missed the new code, but it does not catch logic regressions such as the table of contents patch.by design analysis

Risk

  • Likelihood: low (2 public reports in 5 years)
  • Impact: low (2 of 3 reports: degraded)
Counts behind the grades
Accepted public reports3
Published in the last 5 years2
Impact stated: degraded2
Impact stated: money1

How the grades are computed

Pre-release checklist

  • PR-01
  • PR-02
  • PR-03
  • PR-04
  • PR-05
  • PR-06
  • PR-07

First 30 minutes

  • FM-01
  • FM-02
  • FM-03
  • FM-04
  • FM-05
  • FM-06
  • FM-07

Download the checklist

What catches this: catalogue entries

Listed because their own documentation or site names unit tests, chaos and resilience tests, configuration and environment tests on the verification date. Alphabetical. Not a recommendation.

Tools whose documentation names these test classes

Providers whose site names these test classes

Only on this page

In report [1], partial rollback leaves inconsistent state came from the rollback itself. The automatic rollback returned only a few services to the previous version. Other services could not initialise the new versions. A manual downgrade at 07:20 UTC restored full availability at 08:10 UTC.

In report [3], the mixed state existed before any rollback. The new code reached seven of eight servers, and the eighth kept old code that a repurposed flag activated. Uninstalling the new code from the seven servers spread the old behaviour to them. For partial rollback leaves inconsistent state, a drill that only checks version numbers misses old code that stays reachable through a flag.

Questions

What causes partial rollback leaves inconsistent state?

Two of the three accepted reports place the cause at the release-process stage. One describes an automatic rollback that returned only a few services, and one describes new code that reached seven of eight servers. The third describes cached output with no version that the rolled-back code could not read.

What are the symptoms of partial rollback leaves inconsistent state?

In one report, pages cached by the new release had no table of contents on all wikis after the rollback. No automated monitoring detected the missing tables of contents.

Why did tests not catch partial rollback leaves inconsistent state?

Report [2] states that a well written unit test could have caught the table of contents issue. Its actionables add tests for language conversion on the table of contents and tests for roll back behaviour [2].

Which public reports document partial rollback leaves inconsistent state?

Three primary reports are accepted, published between October 2013 and July 2024. One of them names unit testing as the class that could have caught the fault.

Should a team roll back or fix forward when partial rollback leaves inconsistent state happens?

One report rolled the release forward again after the rollback removed tables of contents. Another restored full availability at 08:10 UTC with a manual downgrade. FM-04 bases the choice on whether the old version can read data the new version wrote.

Release Incident Playbook (2026). Partial rollback leaves inconsistent state: symptoms, cause, and the test that catches it, content version 1.0.0. https://releasefailures.com/incidents/partial-rollback-leaves-inconsistent-state/. Accessed 2026-10-21.