Release Incident Playbook

Rollback fails due to forward-only change: symptoms, cause, and the test that catches it

A rollback is impossible because the data or schema has already moved forward.

Rollback fails due to forward-only change is documented by 4 accepted public reports from 3 organisations, 2 of them published in the last 5 years.

  • Likelihood: low (2 public reports in 5 years)
  • Impact: medium (2 of 2 reports: availability)

4 public reports · verified 2026-10-22

Symptoms

User-visible

  • Customer jobs stayed blocked in a not running state from about 18:50 to 19:43 UTC.[2]
  • Queue times stayed longer than usual, mainly for machine executors, until 20:19 UTC.[2]
  • After the rollback, pages cached by the new release showed no table of contents on any wiki until they were purged.[3]

Monitoring-visible

  • Errors and alerts tied to the job distribution failure appeared immediately after the 18:50 deploy.[2]
  • At the point of rollback the errors changed but did not go away, and incoming work kept decreasing.[2]
  • The production run migrations deploy task failed with a non-zero return code and a subclass-not-found error in the post-deploy migration.[4]
  • The daily deploy got stuck after the post-deploy migration job failed on production.[4]

Root causes

  1. stage: migration A deploy changed a field type in the job distribution database in a way that was not backwards compatible, so older work failed distribution.[2]
  2. stage: migration After the revert, rows written with the new field type between the two deploys could not be read by the old code.[2]
  3. stage: migration A post-deploy data migration branch that fetches a parent group's owner never ran on staging, because no staging record referenced a group without an owner.[4]
  4. stage: migration Some data and schema changes had already landed on production when the post-deploy migration failed, so a simple rollback was not possible.[4]

Public examples

  1. [1]Wikimedia Foundation, 2023-06-19. A scheduled Elasticsearch reindex for Wikidata and Commons de-duplicated analyzers and was promoted to production while the token_count_router query still referenced the removed text_search analyzer, so full-text search failed from June 17 11:30 to June 18 10:02 UTC; reverting was ruled out because it required a full re-index of more than 10 hours, and a one-line code fix was deployed instead. primary Incidents/2023-06-18 search broken on wikidata and commons
  2. [2]CircleCI, 2021-11-11. A deploy changed a field type in the job distribution database in a way that was not backwards compatible; after the team reverted it, the rows written between the two deploys could not be read by the old code, so distribution kept failing until a hand-built change made the distributor ignore the field. primary Incident Report: November 8, 2021 - Jobs stuck in a "not running" state
  3. [3]Wikimedia Foundation, 2021-11-08. A MediaWiki train release changed how the table of contents was stored in the ParserCache; when the train was rolled back to 1.38.0-wmf.6 on November 5, 2021, the older version could not handle the marker left in cached pages and tables of contents disappeared on all wikis, so the train was rolled forward again and a patch was deployed. primary Incidents/2021-11-05 TOC language converter
  4. [4]GitLab, 2020-09-22. A post-deploy data migration that had passed on staging failed on GitLab.com production because production held records staging lacked; the daily deploy was blocked and a simple rollback was not possible because some data and schema changes had already landed, so the team fixed the migration code and rolled forward. primary 2020-09-22: post-deployment migration failed on Production

Test classes that catch it

Test classStageLimitationBasis
Migration testspre-deployMigration tests check the schema and data change, not the application code paths that read it, so the old job distributor failing on rows with the new field type after the revert can still pass.cited in [4]
Chaos and resilience testspre-deployA rollback drill shows whether the previous version starts, but it does not check data correctness, so a drill run before any rows carry the new field type still passes.by design analysis

Risk

  • Likelihood: low (2 public reports in 5 years)
  • Impact: medium (2 of 2 reports: availability)
Counts behind the grades
Accepted public reports3
Published in the last 5 years2
Impact stated: availability2

How the grades are computed

Pre-release checklist

  • PR-01
  • PR-02
  • PR-03
  • PR-04
  • PR-05
  • PR-06
  • PR-07

First 30 minutes

  • FM-01
  • FM-02
  • FM-03
  • FM-04
  • FM-05
  • FM-06
  • FM-07

Download the checklist

What catches this: catalogue entries

Listed because their own documentation or site names migration tests, chaos and resilience tests on the verification date. Alphabetical. Not a recommendation.

Tools whose documentation names these test classes

Providers whose site names these test classes

Only on this page

In report [2], rollback fails due to forward-only change began with a field type change that older work could not pass through. The team reverted the deploy. The revert removed the new code but left the rows already written with the new field type. The old code could not read those rows, so job distribution kept failing after the rollback. Recovery came from a hand-built change that made the distributor ignore the field.

Report [4] states that spec coverage for the failing scenario in the data migration could have caught the error early. That test checks the forward step. It does not run the previous application version against rows the new version already wrote. For rollback fails due to forward-only change, the test needs both halves: data written by the new release, then read by the old one.

Questions

What causes rollback fails due to forward-only change?

Three of the four accepted reports place the cause at the migration stage, and report [3] places it in the release process. In report [2], a field type change was not backwards compatible. In report [3], the rolled-back version could not handle a marker the new release left in cached pages.

What are the symptoms of rollback fails due to forward-only change?

In report [2], customer jobs stayed in a not running state from about 18:50 to 19:43 UTC. After the rollback in report [2], the errors changed but did not go away. In report [3], pages cached by the new release lost their table of contents after the rollback.

Why did tests not catch rollback fails due to forward-only change?

Report [4] states that spec coverage for the scenario in the data migration could have caught the error early. Migration tests do not cover the application code paths that read the new data, which is where the reverted code failed in report [2].

Which public reports document rollback fails due to forward-only change?

Four primary reports are accepted, published between September 2020 and June 2023. Report [4] names migration testing as the class that could have caught the fault.

Should a team roll back or fix forward when rollback fails due to forward-only change happens?

All four accepted reports recovered by moving forward rather than by a plain rollback. In report [1], reverting was ruled out because it required a full re-index of more than 10 hours. FM-05 asks the team to record that a plain rollback is not available before choosing the fix.

Release Incident Playbook (2026). Rollback fails due to forward-only change: symptoms, cause, and the test that catches it, content version 1.0.0. https://releasefailures.com/incidents/rollback-fails-due-to-forward-only-change/. Accessed 2026-10-21.