Rollback fails due to forward-only change: pre-release and first 30 minutes checklists

A rollback is impossible because the data or schema has already moved forward.

The pre-release checklist maps each item to a test class from the incident page. The first-30-minutes checklist covers detection, containment, the rollback decision and customer communication.

7 pre-release items · 7 first-30-minutes items · verified 2026-10-22

Pre-release checklist

PR-01Apply each column type change to a production-shaped copy, write rows with the new type, run its down migration, and fail when the down migration errors. migration
PR-02Fail the migration check for any column type change that ships without a down migration. migration
PR-03Add a spec to each data migration that runs every branch, including the branch that fetches a parent group's owner when the group has no owner. migration
PR-04Run each post-deploy data migration against a production-shaped copy that holds the record types production holds and staging lacks. migration
PR-05Write rows with the new field type on staging, deploy the previous application version, and confirm the previous version reads those rows. chaos
PR-06Roll back the release on staging after its post-deploy migrations finish and confirm the previous version serves requests. chaos
PR-07Insert a row with an unexpected field type on staging and confirm the job distributor keeps distributing the other work. chaos

First 30 minutes

FM-01Compare the error pattern before and after the rollback, and treat errors that change but persist as a sign that data has moved forward.
FM-02Count the rows written in the new format between the forward deploy and the rollback.
FM-03Freeze the deploy pipeline so no further migration runs against production.
FM-04Take a database snapshot before any repair step.
FM-05Record that a plain rollback is not available and choose between a migration code fix and a code change that ignores the new-format field.
FM-06Scale worker capacity to drain the job backlog once distribution resumes.
FM-07Post a status update that names blocked jobs and longer queue times, with the start time of the fault.

Download the checklist

Pre-release checklist (Markdown)1.5 KB · v1.0.0
First 30 minutes checklist (Markdown)1.0 KB · v1.0.0

Only in this checklist

PR-01 is the only item among 5 published checklists in this group that relies on migration tests: Apply each column type change to a production-shaped copy, write rows with the new type, run its down migration, and fail when the down migration errors.

Questions

What should be checked before a release to prevent Rollback fails due to forward-only change?

7 pre-release items, PR-01 to PR-07. PR-01: Apply each column type change to a production-shaped copy, write rows with the new type, run its down migration, and fail when the down migration errors.

Is there a rollback plan for Rollback fails due to forward-only change?

The first-30-minutes checklist has 7 items. FM-01: Compare the error pattern before and after the rollback, and treat errors that change but persist as a sign that data has moved forward.

What should a team do in the first 30 minutes after Rollback fails due to forward-only change is detected?

The first-30-minutes checklist has 7 items. FM-01: Compare the error pattern before and after the rollback, and treat errors that change but persist as a sign that data has moved forward.

Is a Rollback fails due to forward-only change checklist still needed if deployment is automated?

Yes. Automation runs the steps; the checklist names what the tests must cover: migration tests and chaos and resilience tests.

Which pipeline stages does the Rollback fails due to forward-only change checklist cover?

The pre-release items map to the pre-deploy and post-deploy stages.

In what formats can the Rollback fails due to forward-only change checklist be downloaded?

Both checklists download as Markdown files: /downloads/rollback-fails-due-to-forward-only-change-pre-release.md and /downloads/rollback-fails-due-to-forward-only-change-first-30-minutes.md, licensed CC BY 4.0.