Debug a Git auto-deploy that quietly died

My site auto-deploys from Git (a push to the deploy branch should reach
production), but a change that is merged is not live. Find out exactly where the
chain is broken, fix what you can reach, and prove the fix end-to-end. Work in
the site's repository; if I have host API access or MCP tools, use them.

Do not guess at the cause. Every silent deploy failure I have met looked like
three other failure modes until it was measured.

## Step 0 — establish the two truths

Before touching anything, pin down two facts independently:

1. **What Git believes.** Fetch the deploy branch from the remote and confirm
   the missing change is actually merged into it — commit hash, date, files
   changed. Do not trust the merge from memory or from a UI screenshot; run
   `git log` against the fetched remote branch. A "merged" PR whose merge never
   reached the remote is a real failure mode.
2. **What production serves.** Fetch the live URL with a cache-buster query
   string and `Cache-Control: no-cache`. A 404 alone is not enough — also fetch
   a page that *should* be unchanged, so you know the site itself is up and
   you are not looking at a DNS or outage problem.

If direct fetches from your environment are blocked, use the host's file API
(if available) to read files from the docroot instead. Never report "site
unreachable" from a single blocked request.

## Step 1 — find the exact last-deployed commit

Fingerprint production against history. Walk recent commits and pick, for each,
one file with a small distinguishing change — a sentence added, an image with
an exact byte size. Fetch each from production. The newest commit whose
fingerprint matches is the last deploy; everything after it never shipped.

State it precisely: "production is at <hash> (<date>); <n> commits since have
not deployed." This single line usually reveals the shape of the problem —
*deploys stopped at a date* points to an expiry; *only some files stale*
points to caching, not deployment.

## Step 2 — walk the chain in order

Test each link, cheapest first, and stop at the first broken one:

1. **Credentials.** Expired access tokens are the most common cause and fail
   silently — the host's clone simply stops fetching. Check the deploy log in
   the host panel for auth errors. Timing evidence: if deploys worked until
   date X and the token was issued ~90 days before X, you have your answer.
2. **Webhook.** In the repo host's settings, check the webhook's recent
   deliveries for the exact pushes that should have deployed. A webhook that
   points at an old/regenerated integration URL delivers successfully into a
   void, or 404s. If the deploy integration was ever disconnected and
   reconnected, assume the webhook URL changed and the one in GitHub is stale.
3. **Stale server-side clone.** Deploys can fire and still ship an old commit:
   the host's clone deploys its cached HEAD when its fetch fails. Symptom:
   the site *reverts* or redeploys old content after a push. Fingerprint the
   docroot again right after a deploy to catch this.
4. **The deploy itself failing.** Read the actual deploy log. A classic:
   `untracked working tree files would be overwritten by merge` — files placed
   in the docroot by hand (uploads, hotfixes) that the incoming commit now
   tracks. Fix: delete those exact untracked copies from the docroot (they are
   about to be recreated as tracked files, byte-identical) and re-deploy.

## Step 3 — fix, then prove with a marker

Apply the fix for the broken link you found. Then never declare success from
the fix alone — prove the chain:

1. Add an invisible marker to a production-served file:
   `<!-- deploy-check-<UTC-timestamp>-<short-tag> -->` near the top of
   `index.html`. Zero visual, zero functional impact.
2. Commit and push to the deploy branch. This push *is* the webhook test.
3. Poll the live URL (cache-buster + no-cache) until the marker appears.
   Measure the latency and report it — knowing "this host deploys in ~60s" or
   "~8 min" is what stops the next person from declaring a working deploy dead.
4. Cross-check the `Last-Modified` header, and confirm the *other* undeployed
   changes arrived with the marker — the marker proves the pipeline, the full
   diff proves the content.

## Rules

- **Never hand-upload files and call the problem fixed.** Uploading the missing
  files through a file manager makes the symptom disappear while the pipeline
  stays broken — and worse, those hand-placed files are untracked in the
  host's clone and will abort the next real `git pull`. If you must hot-patch
  to end a user-facing outage, say so explicitly, and treat the pipeline as
  still broken until a marker push proves otherwise.
- **Never wipe or force-overwrite the docroot.** Production may hold files
  that exist nowhere else — generated dashboards, uploads, `.well-known`
  material. List and compare before any operation that overwrites, and treat
  anything not in the repo as irreplaceable until proven otherwise.
- **Distinguish "not deployed" from "cached".** Same symptom, different organ.
  If the docroot has the new file but the URL serves the old one, you have a
  caching problem, and the fix is a purge, not a redeploy.
- **One change per test.** If you fix the token and the webhook in one pass,
  you will not know which was broken, and neither will the next session.

## Traps that cost time

- Unauthenticated requests to a private repo return 404, not 403 — a real,
  accessible repo looks nonexistent. Never conclude "repo missing" from
  a raw request; use authenticated tooling.
- Reconfiguring a Git integration often does **not** trigger an initial
  deploy. Silence after reconnecting means nothing; push a marker.
- Deploy latency varies wildly between hosts and even between configs on one
  host (I have measured 60 seconds and 8 minutes on the same site in one
  day). Poll for at least 10 minutes before declaring a deploy dead.
- A merge done on a phone UI can fail silently or hit a different branch —
  re-verify the remote branch head after every merge you did not script.
- If the site serves `.md` or config files it shouldn't, that is a separate
  finding — report it, but do not widen this task into fixing it.

## Report back

1. Last-deployed commit vs deploy-branch head, with the fingerprint evidence
2. The broken link in the chain, with the log line or delivery record that
   proves it — not a theory
3. What was fixed, what was hot-patched (if anything), and what remains
4. The marker push: timestamp, deploy latency, and the live URL check
5. If credentials were involved: when the new ones expire, and a suggestion
   to set a reminder a week before that date