Disk full from new log volume: pre-release and first 30 minutes checklists

A logging change fills disks and services stop writing.

The pre-release checklist maps each item to a test class from the incident page. The first-30-minutes checklist covers detection, containment, the rollback decision and customer communication.

7 pre-release items · 8 first-30-minutes items · verified 2026-10-22

Pre-release checklist

PR-01Measure log bytes written per request before and after the logging change and confirm the projected daily volume at peak traffic fits the free disk space. load
PR-02Run a soak test of the release candidate for at least 8 hours of production-shaped traffic and record root partition usage every hour. load
PR-03After an operating system update, compare base disk usage and kernel audit log volume on a soak test server against the previous image. load
PR-04Confirm during the soak run that log rotation keeps every root partition below the 90% threshold of the disk health check. load
PR-05Add disk usage growth rate per server to the canary metrics and fail the canary when growth exceeds the rate the free space can absorb in one day. canary
PR-06Fill the root partition on several staging servers at once and confirm the disk health check does not remove more servers than the remaining fleet can serve. chaos
PR-07Remove half of the staging application servers at peak load and confirm the remaining servers keep latency within budget. chaos

First 30 minutes

FM-01List root partition usage on every application server and sort the servers by percentage used.
FM-02Identify the log files that grew since the last release, checking application logs and kernel audit logs separately.
FM-03Scale out the application server stack if the disk health check has removed several servers at once.
FM-04Free disk space on servers near the 90% threshold by removing or compressing rotated log files, and record what was deleted.
FM-05Decide whether to revert the logging change or lower its log level, or to roll back the operating system update if kernel audit logs drive the growth.
FM-06Replace server instances whose disks are already full rather than waiting for them to recover.
FM-07Post a status update that names intermittent 500 errors and timeouts as the user-visible symptom.
FM-08Watch disk usage for at least one hour after the fix and confirm the growth rate has returned to the pre-release level.

Download the checklist

Pre-release checklist (Markdown)1.6 KB · v1.0.0
First 30 minutes checklist (Markdown)1.2 KB · v1.0.0

Only in this checklist

PR-06 is the only item among 6 published checklists in this group that relies on chaos and resilience tests: Fill the root partition on several staging servers at once and confirm the disk health check does not remove more servers than the remaining fleet can serve.

Questions

What should be checked before a release to prevent Disk full from new log volume?

7 pre-release items, PR-01 to PR-07. PR-01: Measure log bytes written per request before and after the logging change and confirm the projected daily volume at peak traffic fits the free disk space.

Is there a rollback plan for Disk full from new log volume?

The first-30-minutes checklist has 8 items. FM-01: List root partition usage on every application server and sort the servers by percentage used.

What should a team do in the first 30 minutes after Disk full from new log volume is detected?

The first-30-minutes checklist has 8 items. FM-01: List root partition usage on every application server and sort the servers by percentage used.

Is a Disk full from new log volume checklist still needed if deployment is automated?

Yes. Automation runs the steps; the checklist names what the tests must cover: load and performance tests, canary and progressive delivery checks and chaos and resilience tests.

Which pipeline stages does the Disk full from new log volume checklist cover?

The pre-release items map to the pre-deploy and post-deploy stages.

In what formats can the Disk full from new log volume checklist be downloaded?

Both checklists download as Markdown files: /downloads/disk-full-from-new-log-volume-pre-release.md and /downloads/disk-full-from-new-log-volume-first-30-minutes.md, licensed CC BY 4.0.