My deploy pipeline looked solid. Push to main, GitHub Actions builds a Docker image, and Watchtower on my server pulls it live within a minute. Every step reported success.
Then Watchtower froze, because its ancient Docker client was getting rejected on every poll, and the pipeline's blind spot stopped being theoretical.
Builds kept passing. Images kept pushing. The site kept serving a version from days earlier. Nothing anywhere was red.
The reason is worth stating plainly, because it is a property of most pipelines and not a quirk of mine: every stage only vouched for itself. The build vouched for the build. The push vouched for the push. Nobody's job was to confirm the site had actually changed.
Two small parts
The site carries its own version tag. The build bakes the git commit ID into
a build-commit meta tag in the page's HTML. Invisible to visitors, but any
script can read it and know exactly which commit production is serving.
<meta name="build-commit" content="742d3a7f4b298eca04c99cad57b27d6011ec2a4b" />
CI's last job interrogates production. After the image is pushed, a
verify-deploy job polls the live site every 30 seconds for up to 10 minutes,
comparing that tag against the commit it just built. Match, and the workflow goes
green. No match after 10 minutes, and it fails loudly, attached to the exact
commit that never arrived.
The asymmetry is the point. The build job answers "did I produce an artifact?" The verify job answers "is that artifact what the internet is serving?" Only the second question is the one I actually care about, and until this existed nothing was asking it.
What it does not fix
A frozen Watchtower can still freeze. I cannot reach into that box from CI, and this job does not try to.
What it changes is the detection time and the attribution. The failure now shows up minutes after the push, pinned to the right commit, instead of surfacing days later as a vague "why is the site old?" that costs an hour to trace back to its cause.