Alerts silent after metric rename: pre-release and first 30 minutes checklists

Renamed metrics or labels leave alerts pointing at nothing.

The pre-release checklist maps each item to a test class from the incident page. The first-30-minutes checklist covers detection, containment, the rollback decision and customer communication.

7 pre-release items · 7 first-30-minutes items · verified 2026-10-22

Pre-release checklist

PR-01Compare the collector's exported metric names with the previous release and fail the build on any rename or removal not listed in the changelog. contract
PR-02Compare the label set of each exported metric with the previous release and fail on any added or removed label that an alert query filters on. contract
PR-03Before upgrading the collector, run every alert rule's query against a staging metrics server on the new version and fail on any query that returns no series. config
PR-04Confirm every metric named in the bundled alert rules appears in the release's list of emitted metrics. config
PR-05Confirm the dashboards and the alert rules in the same bundle reference the same metric names. config
PR-06Deploy the v2 tracing backend in a test environment, load the bundled alert rules and confirm the metric behind each rule returns data. e2e
PR-07After each collector upgrade in a test environment, query the failed-spans counter by its documented name and fail when it returns no data. e2e

First 30 minutes

FM-01Run the query behind each alert rule and list the rules whose metric returns no series.
FM-02Match the first empty query to the latest collector or tracing backend upgrade.
FM-03Check whether the exporter's metric names now carry a doubled unit suffix or a new prefix.
FM-04Point the affected alert rules at the new metric names, or remove rules whose metrics are no longer emitted.
FM-05Decide whether to pin the collector to the prior release or move to the fixed patch release, based on whether the maintainers called the rename unintentional.
FM-06Check dashboards and logs for failed or dropped spans during the window in which the alerts were silent.
FM-07Tell on-call engineers which alerts were silent and for which window, so the silence is not read as health.

Download the checklist

Pre-release checklist (Markdown)1.5 KB · v1.0.0
First 30 minutes checklist (Markdown)1.1 KB · v1.0.0

Only in this checklist

PR-01 is the only item among 5 published checklists in this group that relies on contract tests: Compare the collector's exported metric names with the previous release and fail the build on any rename or removal not listed in the changelog.

Questions

What should be checked before a release to prevent Alerts silent after metric rename?

7 pre-release items, PR-01 to PR-07. PR-01: Compare the collector's exported metric names with the previous release and fail the build on any rename or removal not listed in the changelog.

Is there a rollback plan for Alerts silent after metric rename?

The first-30-minutes checklist has 7 items. FM-01: Run the query behind each alert rule and list the rules whose metric returns no series.

What should a team do in the first 30 minutes after Alerts silent after metric rename is detected?

The first-30-minutes checklist has 7 items. FM-01: Run the query behind each alert rule and list the rules whose metric returns no series.

Which configuration checks apply to Alerts silent after metric rename?

PR-03: Before upgrading the collector, run every alert rule's query against a staging metrics server on the new version and fail on any query that returns no series. PR-04: Confirm every metric named in the bundled alert rules appears in the release's list of emitted metrics. PR-05: Confirm the dashboards and the alert rules in the same bundle reference the same metric names.

Is a Alerts silent after metric rename checklist still needed if deployment is automated?

Yes. Automation runs the steps; the checklist names what the tests must cover: contract tests, configuration and environment tests and end-to-end tests.

Which pipeline stages does the Alerts silent after metric rename checklist cover?

The pre-release items map to the pre-merge, pre-deploy and post-deploy stages.