Release Incident Playbook

Alerts silent after metric rename: symptoms, cause, and the test that catches it

Renamed metrics or labels leave alerts pointing at nothing.

Alerts silent after metric rename is documented by 2 accepted public reports from 2 organisations, 2 of them published in the last 5 years.

  • Likelihood: low (2 public reports in 5 years)
  • Impact: unrated (fewer than 2 reports state impact)

2 public reports · verified 2026-10-22

Symptoms

User-visible

  • After collector upgrades, alert rules written against the documented collector metrics, such as the exporter's failed-spans counter, referenced names the collector no longer exposed.[2]
  • Operators who deployed the v2 tracing backend and loaded the project's bundled alert rules had alerts, such as the one for spans dropped at the collector, that could never fire.[1]
  • Organisations running the collector may not know that their collector alerts broke, according to the reporter.[2]

Monitoring-visible

  • The reporter's own rule that flags alert rules based on non-existent metrics fired after each rename.[2]
  • The failed-spans counter gained a doubled unit suffix in its name, reverted, then gained it again in version 0.131.0.[2]
  • Queries for the v1 metrics used in the bundled alerts, such as the dropped-spans counter, returned no data.[1]
  • Dashboards in the same bundle had already moved to the new metric names while the alert rules had not.[1]
  • Alert queries return no results after a metric is renamed or a label is added, and dashboard panels may go empty.[3]

Root causes

  1. stage: code Collector releases changed the names of the collector's own metrics, and a maintainer called the return of the doubled-suffix name unintentional and an undesired breaking change.[2]
  2. stage: third-party The collector's metrics endpoint is implemented by a language SDK module that is not yet considered stable. A follow-up commit downgraded the metrics exporter dependency.[2]
  3. stage: release-process The v2 release of the tracing backend replaced the v1 metric names, and the bundled dashboards were migrated while the alert rules were not.[1]

Public examples

  1. [1]Jaeger, 2026-05-04. Jaeger v2 emits otelcol_* metrics instead of the Jaeger v1 metric names, but 6 of 7 alert rules in the project's monitoring mixin still referenced the v1 names, so alerts such as JaegerCollectorDroppingSpans could never fire; the merged fix removed the obsolete v1 rules. primary [Bug]: Prometheus alert rules in monitoring mixin use v1 metrics that don't exist in Jaeger v2
  2. [2]OpenTelemetry, 2025-07-31. Collector releases changed the names of the collector's own Prometheus metrics three times in a few weeks, most recently in version 0.131.0, so alert rules that the documentation recommends pointed at metric names that no longer existed; a maintainer called the change unintentional and a patch release followed. primary Please, stop changing the names of important prometheus metrics

Practitioner sources cited for symptoms

  1. [3]Cloudflare, 2022-05-19. Cloudflare describes alerting rules that were left referencing renamed or removed metrics after exporter changes such as a fleet-wide node_exporter upgrade, and added label changes that made alert queries return nothing, and the linter it built to check rules against live Prometheus servers. practitioner Monitoring our monitoring: how we validate our Prometheus alert rules

Test classes that catch it

Test classStageLimitationBasis
Contract testspre-mergeA contract test on the list of exported metric names misses an added label that makes an alert query return nothing while the metric name stays the same.by design analysis
Configuration and environment testspre-deployA check of alert rules against a live metrics server finds the missing failed-spans counter only after a server it reads runs the renamed collector release.by design analysis
End-to-end testspre-deployAn end-to-end run in a healthy test deployment drops no spans, so the dropped-spans alert stays quiet whether or not its v1 metric still exists.by design analysis

Risk

  • Likelihood: low (2 public reports in 5 years)
  • Impact: unrated (fewer than 2 reports state impact)
Counts behind the grades
Accepted public reports2
Published in the last 5 years2
Impact stated: degraded1

How the grades are computed

Pre-release checklist

  • PR-01
  • PR-02
  • PR-03
  • PR-04
  • PR-05
  • PR-06
  • PR-07

First 30 minutes

  • FM-01
  • FM-02
  • FM-03
  • FM-04
  • FM-05
  • FM-06
  • FM-07

Download the checklist

What catches this: catalogue entries

Listed because their own documentation or site names contract tests, configuration and environment tests, end-to-end tests on the verification date. Alphabetical. Not a recommendation.

Tools whose documentation names these test classes

Providers whose site names these test classes

Only on this page

Alerts silent after metric rename produces no error when it happens. In report [2], only a rule that flags alert rules on non-existent metrics showed that each rename had broken the failed-spans alert. The reporter noted that other organisations may not know their collector alerts broke [2]. A test that waits for an alert to fire cannot tell a renamed metric from a healthy system.

In report [1], the bundled dashboards had already moved to the new metric names while the alert rules still used the v1 names. A dashboard check passed in report [1], and the dropped-spans alert could never fire. The guard for alerts silent after metric rename queries each alert rule's metric for series, rather than checking the dashboard or waiting for the alert.

Questions

What causes alerts silent after metric rename?

In report [2], collector releases renamed the collector's own metrics, and a maintainer called the change unintentional. In report [1], a v2 release replaced the v1 metric names, and the bundled alert rules were not migrated.

What are the symptoms of alerts silent after metric rename?

Alert rules referenced metric names the collector no longer exposed [2]. Queries for the v1 metrics in the bundled alerts returned no data, so the dropped-spans alert could never fire [1].

Why did tests not catch alerts silent after metric rename?

The renamed metrics produced no error, and the only signal in report [2] came from a rule that flags alert rules on non-existent metrics. Maintainers then linked an end-to-end test to stop such changes going unnoticed [2].

Which test class catches alerts silent after metric rename, and what does it miss?

Contract tests, configuration tests and end-to-end tests are listed, all with the by-design label. A contract test on metric names misses an added label that makes an alert query return nothing.

Which public reports document alerts silent after metric rename?

Report [2] is an issue on the collector's metric renames, published on 2025-07-31. Report [1] is an issue on the bundled v1 alert rules, published on 2026-05-04.

Should a team roll back or fix forward when alerts silent after metric rename happens?

Report [2] fixed forward with a patch release, version 0.131.1. Report [1] fixed forward by removing the obsolete v1 alert rules.

Release Incident Playbook (2026). Alerts silent after metric rename: symptoms, cause, and the test that catches it, content version 1.0.0. https://releasefailures.com/incidents/alerts-silent-after-metric-rename/. Accessed 2026-10-21.