Release Incident Playbook

Long-running migration locks table: symptoms, cause, and the test that catches it

A migration takes a lock on a hot table and blocks writes during the deploy window.

Long-running migration locks table is documented by 3 accepted public reports from 3 organisations, 2 of them published in the last 5 years.

  • Likelihood: low (2 public reports in 5 years)
  • Impact: medium (3 of 3 reports: availability)

3 public reports · verified 2026-10-21

Symptoms

User-visible

  • Most API and web requests fail for the length of the lock, with one report counting 74.7% of requests failed over 24 minutes.[2]
  • Login and authentication calls fail because they query the locked table, with one report counting 75% of API requests rejected, mostly on the auth endpoint.[1]
  • Every inbound request times out across hosted pages, the management portal, the API and notifications.[3]
  • Background evaluations and notifications run late or not at all while queries on the locked table wait.[1]

Monitoring-visible

  • Service degradation alarms fire within minutes of the schema change reaching production.[2]
  • The database connection pool is exhausted by queries piled up behind the locked table.[2]
  • Database performance drops in step with the start of a schema migration, and one stuck query stays visible in the process list.[1]
  • Database load is high enough that operational tooling slows and privileged logins to the database fail.[2][1]

Root causes

  1. stage: migration A foreign key change expected to run in place triggered a full table copy that held the table lock until the copy finished.[2]
  2. stage: migration A column addition waited on a long-running SELECT, and the metadata lock it had already requested queued every other query on that table behind it.[1]
  3. stage: migration A migration dropping legacy columns took a table level lock, queued behind intermittent locks from worker jobs, and then blocked later calls to the same tables.[3]
  4. stage: release-process The migrated table was rated low risk because it carried under 0.2% of the busiest table's traffic, and that rating did not account for blocked writes taking the whole connection pool.[2]

Public examples

  1. [1]Honeycomb, 2024-08-07. A column addition to the teams table, applied during a deployment while a slow SELECT was running, took a metadata lock that queued all other queries on the table, including authentication, for about 20 minutes. primary Honeycomb Status - UI and API unavailable
  2. [2]Pulumi, 2023-10-18. Pulumi shipped a database migration modifying foreign keys on a table; the resulting table copy held a lock long enough to pile up queries and exhaust database connections, and Pulumi Cloud failed 74.7% of requests for about 24 minutes. primary Details of the Pulumi Outage on October 6, 2023
  3. [3]Atlassian Statuspage, 2019-09-06. An ad-hoc migration dropping legacy columns, part of an authentication change, took a table level lock that was queued behind worker job locks and blocked subsequent calls to those tables, causing a 7 minute outage of Statuspage. primary Atlassian Statuspage Status - Site unavailable due to errant database migration

Test classes that catch it

Test classStageLimitationBasis
Migration testspre-deployA migration test on a small or idle copy of the table shows neither the table copy duration nor the lock queue that forms behind a long-running SELECT or worker job lock on the production table.cited in [2]
Load and performance testspre-deployA load test catches connection pool exhaustion only if the schema change is applied while the test traffic is writing to the same table, and it says nothing about whether the migration itself is correct.by design analysis

Risk

  • Likelihood: low (2 public reports in 5 years)
  • Impact: medium (3 of 3 reports: availability)
Counts behind the grades
Accepted public reports3
Published in the last 5 years2
Impact stated: availability3

How the grades are computed

Pre-release checklist

  • PR-01
  • PR-02
  • PR-03
  • PR-04
  • PR-05
  • PR-06
  • PR-07

First 30 minutes

  • FM-01
  • FM-02
  • FM-03
  • FM-04
  • FM-05
  • FM-06
  • FM-07

Download the checklist

What catches this: catalogue entries

Listed because their own documentation or site names migration tests, load and performance tests on the verification date. Alphabetical. Not a recommendation.

Tools whose documentation names these test classes

Providers whose site names these test classes

Only on this page

In report [2], long-running migration locks table was detected when alarms reported degradation within 3 minutes of the foreign key migration reaching production. Detection was fast, yet requests kept failing for about 24 minutes, because the table copy had to finish or blocked writes had to be terminated before connections were freed [2].

The same report states that the migrated table had been rated low risk because it carried under 0.2% of the busiest table's traffic [2]. Blocked writes to that table still consumed the whole connection pool [2].

Questions

What causes long-running migration locks table?

All 3 public reports place the cause at the migration stage. One ALTER triggered an unexpected table copy, one waited behind a slow SELECT while holding a metadata lock request, and one column drop queued behind worker job locks.

What are the symptoms of long-running migration locks table?

Users see request timeouts and failed logins across the whole service, with one report counting 74.7% of requests failed for about 24 minutes [2]. Monitoring shows an exhausted database connection pool and a stuck query on the migrated table.

Why did tests not catch long-running migration locks table?

One report states that pre-production testing could not show how the migration behaved on the production dataset under full traffic load [2]. Report [2] rated the table low risk because it carried under 0.2% of the busiest table's traffic.

Which test class catches long-running migration locks table, and what does it miss?

Migration tests carry a cited basis from one report [2], and load tests are listed by design. A migration test on an idle table misses the lock queue that forms behind a concurrent long-running query.

How long did it take to detect long-running migration locks table in the public reports?

One report gives a measured time to detect of 3 minutes from the change reaching production to the first alarm [2]. The other 2 reports give no measured detection time.

Should a team roll back or fix forward when long-running migration locks table happens?

The reports show recovery by killing the blocking query or blocked writes, or by failing over the database, and then re-applying the migration. Checklist item FM-06 asks responders to choose between re-applying and reverting based on whether the ALTER completed.

Release Incident Playbook (2026). Long-running migration locks table: symptoms, cause, and the test that catches it, content version 1.0.0. https://releasefailures.com/incidents/long-running-migration-locks-table/. Accessed 2026-10-21.