A loud failure is a gift. A crash points at itself: stack trace, timestamp, an obvious place to start reading. You may not enjoy it, but you are never confused about where to look.
The failures that have cost me real time never did that. Every one of them passed every check that existed, because the checks were measuring the wrong thing.
The gallery
Each of these was live for days or months before anyone noticed.
The deploys that stopped happening. My auto-deploy watcher's Docker client was too old for the server, so every poll errored before doing anything at all. The container stayed "healthy" throughout.
The padlock that vanished mid-visit. A one-line server default was redirecting visitors from the secure site to the insecure one. Browsers follow that without complaint.
The pages that were 404s wearing a 200. After a framework upgrade, every project page rendered "not found" while reporting success. One bug hiding a second bug behind it.
The menu link that did nothing. A component quietly swallowed clicks on an external URL. No error. The click simply went nowhere.
The settings that were never applied. Security rules sat in a config file my setup reads, accepts, and then completely ignores. Next to a manifest promising icons that did not exist.
Why software fails this way
Three causes keep recurring, and they are worth naming separately because they need different answers.
Errors get absorbed. Somewhere between the failure and you, a layer catches the problem and carries on. The browser follows the insecure redirect. The poller retries next minute. The missing icon just does not render. Every one of those layers is doing its job, which is exactly why the failure never reaches you.
Health checks measure effort, not outcome. "The container is running" and "the build passed" are claims about machinery. Neither is a claim about the result you actually care about. A process can run perfectly while accomplishing nothing.
Absence does not alarm. Monitoring notices events that happen. A deploy that does not happen, a header that is not sent, a file that is not there: non-events trigger nothing. There is no log line for the thing that failed to occur.
The habit that catches it
Do not ask the machinery whether it ran. Ask the outcome whether it exists.
Curl the live site and read the headers that actually came back. Push a trivial visible change and watch it arrive. Request the file the manifest promises. The question is always the same shape, and it is rude on purpose: show me, live, right now.
The strongest version is automated. My CI now refuses to go green until the live site proves it is serving the new version, and on its very first real run it caught ten days of deploys that had silently gone nowhere.