Extract JavaScript references from HTML pages, print their URLs, and optionally save external scripts and inline code. This is bp0lr's fork of 003random/getJS.

Requires Go 1.26.0 or newer. Use the latest patch of a supported Go release. The preferred development toolchain is Go 1.27.1; Go can download it automatically with GOTOOLCHAIN=auto.

From this checkout:

go build -o getJS .
./getJS --help
go install .

On Windows:

go build -o getJS.exe .
.\getJS.exe --help

go install . installs into your Go binary directory, which must be on PATH.

Install the release with:

go install github.com/bp0lr/[email protected]

Use @latest for the newest tagged release, or @master for the development branch. Download packaged Linux, Windows, and macOS binaries (amd64 and arm64) from Releases. Each archive includes licenses; verify its SHA-256 against SHA256SUMS. CI also keeps development binaries in successful GitHub Actions runs.

Extract absolute references without requesting the scripts:

getJS --url https://example.com --resolve=false

Check script URLs, following page and script redirects:

getJS --url https://example.com --follow-redirect

Print the original HTML attribute values:

getJS --url https://example.com --complete=false --resolve=false

Both --complete and --resolve default to true. URL completion respects the final page URL after redirects and the first usable HTML <base href>.

Use --flag=false to disable a boolean option. Long options use two hyphens.

Combine stdin, a file, and a single URL:

cat pages.txt | getJS --url https://example.org --resolve=false
getJS --input pages.txt --output scripts.txt

PowerShell:

Get-Content pages.txt | getJS --resolve=false

Blank input lines are ignored; surrounding whitespace is trimmed. Input lines are limited to 1 MiB. URLs must use HTTP or HTTPS and cannot contain embedded credentials.

Pages are consumed progressively and duplicate page URLs are skipped. Continuous script workers start new tasks as soon as capacity is available, while output retains document order. Scheduling advances at most four times the worker count beyond the next output position to bound lookahead. Duplicate references share work without occupying extra workers; their metadata remains separate in JSONL.

Plain output contains each absolute resource once; query strings remain significant. Checks are cached across pages, with separate caches for header origins. Downloads are deduplicated within each source page so page directories remain independent. Input and output must be different files.

The HTTP client reuses connections when response bodies permit it. Checks use GET and drain at most 64 KiB; a larger body may require closing the connection. Deduplication and check caches consume memory proportional to unique URLs.

Results go to stdout; progress and errors go to stderr. --output also prints results. To suppress stdout, redirect to /dev/null in a POSIX shell or $null in PowerShell.

getJS --html-file page.html --base-url https://example.com/docs/page --jsonl

Offline mode makes no HTTP requests. It disables resolution automatically and rejects explicit --resolve=true. It cannot be combined with --url or --input; stdin is ignored. The base URL resolves relative references, and HTML <base href> is honored. JSONL records identify the input with an absolute file URI.

With --save, offline mode saves inline code only. External script URLs are listed without checks or downloads. Size limits, URL filters, and --same-origin still apply. The HTML input and output file must be different.

getJS -u https://example.com -H "Authorization: Bearer example" -H "X-Context: value:with:colons"
getJS -u https://example.com --save --follow-redirect

Custom headers are sent to each supplied page and its same-origin scripts. They are removed when a redirect chain changes origin and are not restored later in that chain. Same origin means equal scheme, hostname, and effective port. Header values are not logged.

--same-origin also filters output when checks are disabled. The final page URL defines the origin; an HTML base can resolve references but cannot expand it. Inline code remains included. Page redirects may establish a new page origin; a script redirect leaving that origin is refused before contacting the destination and is reported as a script error.

Select declared script URLs by pattern:

getJS -u https://example.com --include 'app|worker' --exclude 'vendor' --resolve=false

Patterns use Go's regexp syntax and are compiled once. They match the full resolved reference URL, including its query, even with --complete=false. Matching happens before script requests; it is not a redirect-destination policy. Inline code has no URL and is unaffected by these patterns.

Downloads go under <output-dir>/<page-host>/<page-url-hash>/. External names include a stable URL hash, including query strings. Names are sanitized for Windows and existing files are preserved using numbered suffixes.

Files are written to a temporary path, checked against the size limit, flushed, and published without replacing existing files. Hard links publish complete files atomically where supported. Other filesystems use exclusive creation and copying: the destination is visible during that copy, and reported only after completion. Paths are confined to the selected download root. Temporary files and incomplete fallback copies are cleaned up on handled failures. HTTP 304 is accepted for checks but cannot produce a download without a cached body.

The body limit applies to parsed HTML and saved scripts after automatic HTTP decompression. Checks reject a known oversized Content-Length but may stop after 64 KiB when the length is unknown; they do not certify the full body size. --max-body-size accepts an integer from 1 byte to 1 TiB.

getJS --input pages.txt --jsonl --save --output-dir scripts --output results.jsonl

Each object includes page and kind (script, inline, preload, modulepreload, or page for a page error). External discoveries include absolute url and the original reference. Optional fields include status, final_url, path, size, and error.

Completed downloads also include sha256, calculated while writing their content. Incremental runs include reused: true when an existing verified file is retained; status still describes the actual HTTP response (200 or 304).

JSONL preserves individual discoveries, including repeated references and their source pages, while checks and downloads remain cached. It includes inline records without embedding inline source text. Failed scripts have an error record; failed page fetches have a page error record. Diagnostics still go to stderr and partial failure still returns code 2.

The attributes object records type, async, defer, nomodule, integrity, crossorigin, referrerpolicy, and link rel/as when applicable. Boolean HTML attributes use presence semantics: async="false" is still present. Empty crossorigin="" remains distinguishable from an absent attribute.

attributes.index is the one-based position among script and link-with-href candidates in the parsed document, before filtering. Inline filenames use this position. Metadata describes HTML declarations; getJS does not execute scripts or verify Subresource Integrity.

--complete=false changes plain-text output only; JSONL always keeps the resolved URL alongside the original reference.

getJS --input pages.txt --save --manifest manifest.jsonl --output-dir scripts

The manifest contains one object per distinct completed file: page, optional external url, kind, path, size, and sha256. External files also record final_url and, when available, etag, last_modified, and a request_context fingerprint. Inline files record their document index. Repeated tags that share a cached download produce one manifest entry. Failed downloads produce no entry. Paths are absolute when --output-dir is absolute; otherwise they are relative to the process working directory.

Compare hashes to detect content changes or identical content at different URLs. A hash records content identity, not trust or authenticity. The manifest is replaced on each run and must differ from input and result files. An interrupted run can leave a partial manifest; already completed script files remain available.

Run the same command again to update a saved collection:

getJS --input pages.txt --save --incremental --manifest manifest.jsonl --output-dir scripts

A missing manifest starts a new collection. Existing entries are read before any replacement. Files must remain inside the selected output directory and match the recorded size and SHA-256. Missing, changed, or oversized cached files trigger a regular download. Keep the same working directory when using relative paths.

External scripts are always requested. For a verified file, getJS sends If-None-Match (ETag), or If-Modified-Since when no ETag is available. Validators follow only the previously recorded final URL, so a changed redirect target receives an unconditional request. Changes to configured headers, page origin, proxy, or TLS/origin settings invalidate conditional reuse. The manifest stores a configuration fingerprint, without header values.

HTTP 304 reuses the verified local body; if that body disappears or changes during the request, getJS retries once without validators. A 200 response with identical content retains the old file, even if the server ignored validators. Changed content gets a numbered filename and preserves the previous version. Responses with Cache-Control: no-store or Vary: * do not enable subsequent conditional reuse. Explicit conditional request headers cannot be combined with --incremental.

Inline code is reused by source page, document position, and content hash. Offline mode continues to make no network requests. Older manifests without validators still allow content comparison after a normal download.

Incremental manifests are staged and replace the previous manifest only when the entire run succeeds. A failed or interrupted run preserves the old manifest; files already completed remain on disk. A later run may create another copy of files completed during a failed run because they are absent from that preserved manifest. Invalid manifests cause an error before processing. No separate cache database is created.

• Reads src and data-src from script elements. Inline text is saved with --save.
• Also reads link[rel~=modulepreload] with a script-like destination and link[rel~=preload][as=script], in document order. Explicit non-script destinations are excluded. A preload is a declared resource hint, not evidence of execution.
• Parses returned HTML only; does not execute JavaScript or observe browser-injected scripts.
• Ignores non-HTTP references for network operations.
• Requires HTTP 200 for HTML and HTTP 200 or 304 for script checks. Status alone does not prove the body is JavaScript.

1. Automatic identification of installed versions and local Git builds.
2. Continuous script workers with ordered output and bounded lookahead.
3. Portable file publication for destinations without hard-link support.
4. Verified incremental downloads with persistent HTTP validators.
5. Upstream license attribution and a release workflow that tests all supported platforms before publishing binaries and checksums.

Changes are kept in separate commits. This README is the only Markdown document in the repository.

go test ./...
go vet ./...
go build ./...

Tests use local HTTP servers, TLS fixtures, and temporary directories. CI tests Go 1.26.0 and 1.27.1 on Linux, Windows, and macOS, and runs the race detector on Linux.

Pushing a vMAJOR.MINOR.PATCH tag runs the same test matrix and race detector, then publishes six archives and SHA256SUMS. Release builds use CGO_ENABLED=0 and include the project, Go runtime, and dependency licenses. Version and commit are embedded during compilation. Only the publishing job has repository write permission.

Build with version information:

go build -ldflags="-X main.version=dev -X main.commit=local" .

--version uses module build information for go install, Git revision information for local builds, and explicit linker values when supplied. Local builds with uncommitted changes show [modified]. Installed release tags identify the version even when Go does not embed a Git revision.

go test -run '^$' -bench 'Benchmark(Pipeline|CheckMethod)' -benchmem ./...

A local synthetic sample on Go 1.27.1, Windows amd64 and Ryzen 9 3900X used a page with 16 distinct 1 KiB scripts, each referenced twice, and a 2 ms response delay. Three repetitions of three iterations produced these median batch times:

The per-host limit matched concurrency for the experiment; the normal default is 2. These are small samples from two revisions of the same fixture, not Internet speed guarantees. Continuous workers allocated about 279 to 285 KB/op at concurrency 1 and 497 to 521 KB/op at concurrency 8; allocated bytes are not peak memory.

For a separate 1 KiB fixture with 1 ms delay, HEAD with fallback took about 3.06 ms when HEAD returned 405, versus 1.56 ms for GET. GET remains the default. DOM replacement and browser execution are outside this update.

• TLS certificates are verified by default; --insecure is explicit.
• Errors are visible on stderr and may return a nonzero exit code.
• Plain output is deduplicated; JSONL preserves discoveries and their origin pages.
• Downloads use portable names and preserve existing files.
• Headers apply to same-origin scripts and are removed on origin changes.
• --save --resolve=false still downloads external scripts in online mode.
• --nocolors remains accepted; output uses plain text.

Use this fork's issue tracker. Include a minimal example and remove credentials from logs.

Licensed under the MIT License, preserving the original copyright notice from 003random/getJS.

Thanks to 003random for the original implementation and pczajkowski for credited improvements and ideas.