GeezerVison

--check-external threw without --diff, so the external links already on main could not be scanned.

What changes

• Without --diff, --check-external requests every external link in docs/. With --diff it still requests only the links on added lines.
• stderr counts, by host and reason, the URLs that could be confirmed neither live nor dead (rate limits, bot blocks, DNS failures). They are still not findings; a full scan now shows what it did not verify. In CI this goes to the job log, not the PR comment.
• A 429 with a Retry-After header (seconds or an HTTP date) is retried after that wait. Every request to that host waits, not only the refused one. The scan stops waiting on a host 5 minutes after its first wait, and never takes a wait that would end past that; those 429s stay unverified.
• SRC_ACCESS_TOKEN, when set, is sent as Authorization: token … to SRC_ENDPOINT (default https://sourcegraph.com, as in src) and to no other host.
• A https://sourcegraph.com/search?q=… link is no longer requested as a page, which answers 200 whatever the query. Its query is run through https://sourcegraph.com/.api/graphql, with the link's patternType (default keyword, as on the search page). A search that returns the alert "No repositories found" is a finding. A search without matches, another alert, or an API error is counted on stderr as unverified.
• A https://sourcegraph.com/github.com/… link to a repository, directory, or file page (with or without @revision, /-/blob/, /-/tree/) is looked up through the same API. It is a finding when sourcegraph.com does not have the repository, the revision, or the path, and the finding says which. Other pages under a repository (/-/commit/, /-/badge.svg) are still requested.

This changes the PR check: a PR that adds a search link whose repo: filters match no repository on sourcegraph.com now fails Broken links introduced by this PR. A dead repository, directory, or file link failed it before as an HTTP 404, when the page was not behind the challenge; it now reads sourcegraph.com has no repository github.com/sourcegraph/jetbrains.

⚠️ 25 of 1392 external URL(s) could not be verified, and are not reported as dead:
   2 × gitlab.com: HTTP 403
   1 × phabricator.org: ENOTFOUND
   6 × sourcegraph.com: search has no matches
   1 × sourcegraph.com: search alert: Error during ownership search
   ...

Test plan

• Full scan of this branch on origin/main (45d23e43): node dev/check-links.mjs --check-anchors --check-self-links --check-external --format json requested 1,392 URLs in 72 seconds. It reported 232 internal findings and 111 dead external links (91 URLs, 45 files): 50 that return HTTP 404, 41 search links that match no repository, 19 sourcegraph.com links to a repository it does not have, and 1 to a path that is gone (docker-library/postgres@master/-/blob/12/alpine/Dockerfile), which no earlier scan could verify.
• Every finding of the scan before the repository-link change is still there: 19 changed from External link returns HTTP 404 to the new text, and the path above is the only new one.
• For the 35 repository, directory, and file links in docs/, the GraphQL lookup agrees with the status of the page itself, requested while the challenge was down: 19 live (200), 15 without a repository (404), 1 without the path (404).
• The same scan with and without a valid SRC_ACCESS_TOKEN gives byte-identical findings.
• With a --diff that adds one line holding a dead github.com link, the JSON output is byte-identical to main's, and stderr is empty.
• With a --diff that adds a line with a search link for a repository that is not on sourcegraph.com and a line with a live one, only the first is a finding.
• On a scratch page: repository links with a made-up repository, revision, and path are three findings, each naming what is missing; a repository page with and without a trailing slash, a file at a tag with ?L10-20, a directory linked as /-/blob/, a directory at a branch with a trailing slash, and a percent-encoded path are silent.
• On a scratch page: a search link with a misspelled repository name is a finding; one with a real repository and no matches, and one with an invalid patternType, are counted on stderr; one with matches is silent.
• Against local servers: a host that refuses its first three requests with Retry-After: 2 is asked again at +2 s and +4 s and its links verify; other requests to that host wait too. An HTTP-date Retry-After is honored. A 429 with Retry-After: 400, an unparsable Retry-After, or none is not retried and is counted as unverified.
• Against the same servers with SRC_ENDPOINT set to one of them: that server receives the token, including after a redirect to itself; a second host does not, neither directly nor after a redirect from the first. With SRC_ENDPOINT unset, neither receives it.
• Against sourcegraph.com with a made-up token, a dead repository link is counted as GraphQL API returns HTTP 401 instead of reported: the header arrives, and a rejected token turns that host's links into unverified 401s.

What the token does not do

sourcegraph.com's 429 is a Cloudflare challenge (cf-mitigated: challenge, no Retry-After). It started on the 26th /search request in 4 seconds and was still up 44 minutes later. A valid token does not get past it: the same 19 repository and file pages (/github.com/…) returned the challenge with and without one. The GraphQL API is not behind it, and ran all 98 search queries anonymously in 35 seconds while the challenge was up.

Not in this PR

• A scan while the challenge is up, since the repository-link change: it had lifted by then. Those links no longer request a page, so the challenge has nothing to refuse.
• Catching every dead search link: two links with rev:at.time(…) on a repository that is not on sourcegraph.com return no matches and no alert, so they are counted as unverified, not reported. The count of searches without matches varies between scans (6 to 8), as a slow search sometimes times out.
• Fixing the dead links the scan found.