Release marker missing in monitoring: symptoms, cause, and the test that catches it

Responders cannot see which deploy correlates with the change in metrics.

Release marker missing in monitoring is documented by 3 accepted public reports from 2 organisations, 1 of them published in the last 5 years.

Likelihood: not published (1 public report in 5 years)
Impact: medium (2 of 2 reports: availability)

3 public reports · verified 2026-10-21

Symptoms

User-visible

CI pipelines failed to upload artifacts with 502 Bad Gateway errors, mainly for large files.[1]
At least 12 customers were affected, with an error rate slightly under 5% on artifact uploads.[1]
Merge requests were slow and pipeline triggers were delayed while the primary database was overwhelmed.[2]
All queued CI workflow jobs failed for a period of time.[3]

Monitoring-visible

No alerts or obvious dashboard indicators pointed at the HTTP router.[1]
Monitoring dashboards showed no deployment annotations for the HTTP router.[1]
Failed upload requests never reached internal application or proxy logs, and the 502 errors became visible only after heavy filtering in the edge network dashboards.[1]
A sharp increase in in-flight background jobs coincided with the recent canary deployment.[2]
Service monitors detected a high error rate on workflow runs for new and in-progress jobs.[3]

Root causes

stage: release-process The HTTP router deploys separately from the standard deployment process, and its deployment left no annotation in the monitoring dashboards.[1]
stage: migration A post-deployment migration removed a required index on the ssh_signatures table. The rise in in-flight jobs coincided with the canary deployment, so responders drained the canary first.[2]
stage: config A bad service record pushed to the service discovery system broke service-to-service calls.[3]

Public examples

[1]GitLab, 2026-01-16. A deployment of a Cloudflare Worker in the GitLab.com HTTP router made large CI artifact uploads fail with 502 errors for about 10 hours, and because the router deploys outside the standard process with no deployment annotations in monitoring dashboards, responders found it hard to correlate the deployment with the incident. primary Incident Review: INC-6665 Pipelines are failing to upload artifacts due to 502 Bad Gateway
[2]GitLab, 2026-01-06. A post-deployment migration on GitLab.com removed a required index on the ssh_signatures table, responders first drained the recent canary deployment without improvement before turning to the post-deployment migrations, and a follow-up was opened to add post-deployment migration annotations to Grafana. primary Incident Review: INC-6458: Sidekiq queueing SLO violation on urgent-cpu-bound shard
[3]GitHub, 2021-09-01. A bad service record pushed while setting up a new Actions microservice broke service-to-service calls in GitHub Actions, responders first reverted recent Actions deployments, and GitHub afterwards added broader visibility into what had changed recently across all Actions microservices. primary GitHub Availability Report: August 2021

Test classes that catch it

Test class	Stage	Limitation	Basis
Canary and progressive delivery checks	post-deploy	Canary checks judge only the traffic the canary receives, and the post-deployment migration that removed the index ran outside the canary, so draining the canary did not improve the metrics.	by design analysis
Synthetic monitoring	post-deploy	A scripted upload through the HTTP router shows the 502 errors only if its file exceeds the edge worker memory buffer, because smaller uploads are a segment that passed.	by design analysis
Configuration and environment tests	pre-deploy	Configuration checks can reject the bad service record before it is pushed, but they do not catch load, and the missing index surfaced only as slow queries under production traffic.	by design analysis

Risk

Likelihood: not published (1 public report in 5 years)
Impact: medium (2 of 2 reports: availability)

Counts behind the grades
Accepted public reports	2
Published in the last 5 years	1
Impact stated: availability	2

How the grades are computed

Pre-release checklist

PR-01Confirm the HTTP router deploy writes a deployment annotation to the monitoring dashboards in staging before the production deploy. config
PR-02Confirm each post-deployment migration run writes its own dashboard annotation, separate from the canary deployment annotation. config
PR-03Confirm a push to the service discovery system writes a change event that appears on the same dashboards as deploy annotations. config
PR-04Validate every new service record against the service discovery schema before it is pushed. config
PR-05After each HTTP router deploy, run a scripted artifact upload larger than the edge worker memory buffer and confirm it returns success. synthetic
PR-06Route failures of the scripted artifact upload check to an alert that names the HTTP router. synthetic
PR-07List the post-deployment migrations pending for the release in the canary report. canary
PR-08Chart in-flight background jobs for canary and baseline on a dashboard that shows every annotated change. canary

First 30 minutes

FM-01List every change from the last 24 hours across all deploy paths: application deploys, HTTP router deploys, post-deployment migrations and service discovery pushes.
FM-02Check the edge network's request logs for the failing path when internal application and proxy logs show no errors.
FM-03Check for indexes dropped by post-deployment migrations when database load climbs without a matching application deploy.
FM-04If draining the canary does not improve the metrics, move on to the next change in the list.
FM-05Roll back the change whose time matches the start of impact, through the rollback control of its own deploy path.
FM-06Add a manual dashboard annotation for each change found and each action taken, with its time.
FM-07Post a status update that names the failing workflow, such as artifact uploads, and the share of failing requests.

Download the checklist

Pre-release checklist (Markdown)1.6 KB · v1.0.0
First 30 minutes checklist (Markdown)1.2 KB · v1.0.0

What catches this: catalogue entries

Listed because their own documentation or site names canary and progressive delivery checks, synthetic monitoring, configuration and environment tests on the verification date. Alphabetical. Not a recommendation.

Tools whose documentation names these test classes

Argo Rollouts canary source verified 2026-10-21
Checkly synthetic source verified 2026-10-21
Datadog Synthetic Monitoring synthetic source verified 2026-10-21
Flagger canary source verified 2026-10-21
Flipt config source verified 2026-10-21
Grafana Cloud Synthetic Monitoring synthetic source verified 2026-10-21
Grafana k6 synthetic source verified 2026-10-21
GrowthBook config source verified 2026-10-21 canary source verified 2026-10-21
Kayenta canary source verified 2026-10-21
LaunchDarkly config source verified 2026-10-21 canary source verified 2026-10-21
PostHog config source verified 2026-10-21
Postman synthetic source verified 2026-10-21
Spinnaker canary source verified 2026-10-21
Unleash config source verified 2026-10-21 canary source verified 2026-10-21
Uptime Kuma synthetic source verified 2026-10-21

Providers whose site names these test classes

No provider in the catalogue names this test class on its site.

Only on this page

Release marker missing in monitoring appears in report [1] as a deploy path with no marker at all. The HTTP router deploys outside the standard process. No alerts pointed at it, and its deployment carried no annotation on the dashboards. Failed requests never reached internal logs, so the only trace sat in the edge network dashboards after heavy filtering.

In report [2], a marker existed but pointed at the wrong change. The rise in in-flight background jobs coincided with the canary deployment. Responders drained the canary without improvement before turning to the post-deployment migrations. In report [3], responders first reverted recent deployments of the workflow microservices. The follow-up in report [3] added broader visibility into what had changed recently across those microservices. For release marker missing in monitoring, a canary check cannot help when the faulty change travels on a path the canary does not carry.

DNS change propagation outage A DNS or load balancer change sends traffic to the wrong target or nowhere.
Autoscaling policy misconfigured A scaling change fails to add capacity or scales to zero under load.
Error budget burned by retry storm Client retries amplify a partial failure into a full outage.
Partial rollback leaves inconsistent state Only some services are rolled back and the system runs with incompatible versions.
Rate limit misconfigured A rate limit change blocks legitimate traffic or stops protecting an endpoint.

Questions

What causes release marker missing in monitoring?

The three accepted reports name three deploy paths outside the usual marker. They are an HTTP router deployed outside the standard process, a post-deployment migration that ran after the canary, and a service discovery record push.

What are the symptoms of release marker missing in monitoring?

In one report, no alerts or dashboard indicators pointed at the HTTP router while artifact uploads failed with 502 errors. In another, a rise in in-flight background jobs coincided with a canary deployment that was not the change at fault.

Which test class catches release marker missing in monitoring, and what does it miss?

The page lists canary checks, synthetic monitoring and configuration tests, all by design. Canary checks miss changes that run outside the canary, such as the post-deployment migration that removed an index.

Which public reports document release marker missing in monitoring?

Three primary reports are accepted, published between September 2021 and January 2026. None of them names a test class.

Should a team roll back or fix forward when release marker missing in monitoring happens?

One report rolled back the HTTP router deployment with its automated rollback control. Another recreated the missing index manually. FM-05 asks for the rollback of the change whose time matches the start of impact.

Release Incident Playbook (2026). Release marker missing in monitoring: symptoms, cause, and the test that catches it, content version 1.0.0. https://releasefailures.com/incidents/release-marker-missing-in-monitoring/. Accessed 2026-10-21.