Connection pool exhausted: symptoms, cause, and the test that catches it

Symptoms

Root causes

1. stage: code New code sent extra work to a primary database: an unexpectedly expensive query from a newly enabled code path in one report, and a newly deployed background job in the other. The added load exhausted the primary's connection pool.[1][2]
2. stage: code A feature flag made the database load balancer check out a fresh connection for each health check. The health checks ran several times inside every background job, so each job held one extra connection and pool usage roughly doubled.[3]
3. stage: code A condition in code shipped with a scheduled release opened a database connection and did not close it after the condition ended. The leak happened only under certain conditions.[4]
4. stage: code A code change made indexing processes keep database connections open where their call paths crossed into older session management code. The stale connections built up under rising load.[5]

Public examples

1. [1]GitHub, 2026-06-08. A recently enabled code path at GitHub issued an expensive query against a primary database, exhausting its connection pool so that an average of 31 percent of Git pushes failed for nine minutes. primary Elevated rate of Git push errors
2. [2]GitHub, 2025-02-28. A newly deployed background job at GitHub increased load on the primary database hosts and exhausted the connection pool, raising write latency and request timeouts until the job was halted. primary Elevated Request Latency for Write operations on github.com and api.github.com
3. [3]GitLab, 2024-11-05. GitLab rolled a feature flag out to 100 percent of pods that made the database load balancer check out an extra connection for health checks during every Sidekiq job, which saturated the urgent Sidekiq pgbouncer pool until the flag was disabled. primary Incident Review: The sidekiq_queueing SLI of the sidekiq service on shard low-urgency-cpu-bound has an apdex violating SLO
4. [4]tada, 2023-01-11. After a scheduled release at tada, a code condition opened database connections without closing them, and when real traffic arrived new database connections timed out, taking down the Integration API and Wallet API until the code was fixed and redeployed. primary Connection issue to database
5. [5]Close, 2020-11-25. A code change deployed at Close left database connections open where call paths crossed into older session management code, and stale connections accumulated until the application exhausted all connections to a backend database and all Close systems were disrupted for 11 minutes. primary Application loading issues

Test classes that catch it

Risk

Pre-release checklist

• PR-01Run each new background job in the integration suite and confirm the open database connection count returns to its starting value when the job ends. integration
• PR-02Exercise every branch of a code path that opens a database connection, including the branch where its condition ends, and assert the connection is closed afterwards. integration
• PR-03Count connections checked out per background job with the health-check feature flag on and off, and fail the test when the flag adds a connection. integration
• PR-04Run indexing processes through call paths that cross into the older session management code and assert every connection is released. integration
• PR-05Run each new query from the release against a production-sized copy of the primary database and fail the run when its execution time exceeds the agreed budget. load
• PR-06Run a load test with the new background job on its production schedule alongside user write traffic and confirm primary database pool usage stays below the pool limit. load
• PR-07Hold the load test at peak morning traffic until connection counts level off, and fail it when open connections keep rising. load
• PR-08Record connection pooler saturation at every feature flag rollout step, including the final step to 100 percent of pods, and halt the rollout when saturation rises above the pre-rollout level. canary
• PR-09Halt the rollout when database connections in use per canary pod exceed the baseline pods' count. canary

What catches this: catalogue entries

Listed because their own documentation or site names integration tests, load and performance tests, canary and progressive delivery checks on the verification date. Alphabetical. Not a recommendation.

Related incidents

• Background job queue backlog A job change slows processing and the queue grows faster than it drains.
• Memory leak in new release Memory grows per request or per screen until processes restart or the app is killed.
• Third-party rate limit hit A new call pattern exceeds a provider's quota and requests are rejected.
• Cache invalidation storm A deploy flushes or invalidates caches at once and origin load spikes.
• Cold start latency regression Serverless or container cold starts become slower after a dependency or image change.

Questions