An Awesome List for getting started with web archiving

awesome awesome-list webarchiving

• Updated Nov 6, 2026

Wayback Machine API interface & a command-line tool

osint internet-archive web-archiving wayback-machine webarchiving cdx-api internet-archiving savepagenow archive-webpage archive-webpages wayback-machine-api wayback-machine-python

• Updated Feb 26, 2024
• Python

WARC + AI - Experimental Retrieval Augmented Generation Pipeline for Web Archive Collections.

ai warc webarchiving rag

• Updated Feb 11, 2025
• Python

Squidwarc is a high fidelity, user scriptable, archival crawler that uses Chrome or Chromium with or without a head

crawler chrome crawling chrome-headless browser-automation headless-chrome webarchiving webarchives high-fidelity-preservation puppeteer

• Updated May 19, 2020
• JavaScript

Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.

crawler scraper downloader spider lua ftp scraping crawling archiving wget crawl zstd crawlers warc webarchiving archiveteam wget-lua

• Updated Oct 26, 2026
• C

A list of things related to software, literature, and other content for 🕣 Memento

awesome memento awesome-list webarchiving memento-rfc

• Updated May 22, 2026

Parse And Create Web ARChive (WARC) files with node.js

warc web-archiving webarchive web-archives webarchiving warc-files chrome-remote-interface pupeteer

• Updated Jan 29, 2025
• JavaScript

Various Jupyter notebooks about Common Crawl data

jupyter-notebook aws-athena commoncrawl common-crawl webarchiving webgraph-framework

• Updated Sep 8, 2026
• Jupyter Notebook

A dockerized, queued high fidelity web archiver based on Squidwarc

archiving preservation webarchiving webarchives high-fidelity-preservation

• Updated Jul 9, 2024
• Python

Awesome list dedicated to digital and data preservation tools, sources, services and so on.

crawler list awesome awesome-list warc digital-preservation archival webarchiving

• Updated Oct 22, 2026

metawarc: a command-line tool for metadata extraction from files from WARC (Web ARChive)

metadata osint warc webarchiving warc-files osint-python

• Updated Nov 8, 2026
• Python

Quick Cache and Archive search buttons

webarchive webarchiving google-cache yandex-cache baidu-cache

• Updated May 11, 2024
• JavaScript

A social media open post web archiving tool

archiving preservation webarchiving high-fidelity-preservation

• Updated Feb 4, 2026
• JavaScript

Digital Preservation of HTTP in documentary heritage.

archives code4lib wayback internetarchive digipres webarchiving digitalpreservation documentary-heritage digital-repositories waybackmachine

• Updated Jul 5, 2026
• Go

Decentralized web archiving

python crawler web decentralized archiving archiver warc webarchiving

• Updated Aug 7, 2018
• Python

A tool for detecting viruses and NSFW material in WARC files

antivirus warc webarchiving nsfw-classifier warc-safe

• Updated Nov 6, 2026
• Python

🗄 File-Based Reference Filing System.

storage workflow-engine webdav fuse-filesystem webarchiving gtd-workflow

• Updated Aug 30, 2026
• Go

Seeder - Czech webarchive curating tool and public site

government django tools czech czech-republic archive webarchive webarchiving webarchives

• Updated Nov 7, 2026
• Python

A javascript for fighting link rot and content drift using link decoration and web archives.

linkrot webarchiving

• Updated Oct 31, 2024
• HTML

Parser for WARC (aka WebArchive) files

warc webarchive webarchiving warc-files webarchives warc-format warc-reader warc-record

• Updated Mar 26, 2026
• C#