Archive git repositories to the Internet Archive.
iagitup clones a repository, packs it into a git bundle, and uploads it to archive.org with metadata.
archive-watchlist does it in bulk: every repo of a user or organization, or the most-starred GitHub repos.
- Any HTTPS git host -- GitHub, GitLab, Bitbucket, Codeberg, self-hosted Gitea/Forgejo, or a plain URL.
- Complete snapshots -- every branch, tag and ref in one verified bundle.
- Wikis, LFS, submodules -- archived alongside the main bundle.
- Releases, issues, pull requests -- optional GitHub exports, assets included.
- Rich metadata -- description, README, avatar, stars and forks from the platform API.
- No duplicates -- a snapshot is uploaded once; a new push creates a new item.
- Resumable -- keep the clone and retry an interrupted upload without re-cloning.
- Bulk mode -- every repo of a user or org, or the top 1000 GitHub repos. Parallel workers, JSON summaries.
pip install iagitup
Installs two commands: iagitup and archive-watchlist.
Requirements:
- Python 3.10+
giton$PATH(git-lfstoo, if you want LFS objects)- An Internet Archive account
- HTTPS repository URLs (SSH syntax is not supported)
iagitup https://github.com/torvalds/linux iagitup https://gitlab.com/inkscape/inkscape iagitup https://codeberg.org/forgejo/forgejo
:: Downloading https://github.com/torvalds/linux ...
:: Uploading bundle: torvalds-linux_-_2026-02-28_10-00-00.bundle
:: Upload FINISHED.
Archived repository: https://archive.org/details/github.com-torvalds-linux_-_2026-02-28_10-00-00
Git bundle: https://archive.org/download/github.com-torvalds-linux_-_2026-02-28_10-00-00/torvalds-linux_-_2026-02-28_10-00-00.bundle
Bulk mode:
archive-watchlist --dry-run --top 10 # preview archive-watchlist --workers 8 # top 100, all-time archive-watchlist --days 7 # trending this week archive-watchlist --since 2025-01-01 --until 2025-06-30 archive-watchlist --owner torvalds --skip-archived --max-repos 20 # all repos of a user/org
| Flag | Description |
|---|---|
-m, --metadata |
Extra metadata as key:value,key2:value2 |
--dry-run |
Print identifier and metadata, upload nothing. No credentials needed |
--submodules |
Clone recursively and upload one bundle per submodule |
--releases |
Archive release notes and assets (GitHub only) |
--issues |
Archive issues and pull requests with comments (GitHub only) |
--work-dir DIR |
Where to put temporary clones (default: system temp) |
--keep-work-dir |
Keep the clone after the run and print its path |
--resume PATH |
Upload from a kept clone instead of cloning again |
--include-forks |
Owner URL: also archive forks |
--skip-archived |
Owner URL: skip archived repos |
--max-repos N |
Owner URL: only the N most recently pushed repos |
-v, --version |
Print version |
Archive every repo of a user or organization by passing its URL:
iagitup https://github.com/torvalds iagitup --skip-archived --max-repos 20 https://gitlab.com/gnome
Resume an interrupted upload:
iagitup --keep-work-dir https://github.com/user/repo
# ... fails; stderr shows ":: Work directory kept: /tmp/tmpabc123/repo"
iagitup --resume /tmp/tmpabc123/repo https://github.com/user/repo| Flag | Default | Description |
|---|---|---|
--top N |
100 | Repos to check, up to 1000 |
--days N |
all-time | Only repos created in the last N days |
--since DATE |
Only repos created on or after YYYY-MM-DD |
|
--until DATE |
Only repos created on or before YYYY-MM-DD |
|
--owner URL_OR_NAME |
Archive all repos of a user/org (GitHub, GitLab, Gitea/Codeberg) instead of the top-starred; excludes --top/--days/--since/--until |
|
--include-forks |
With --owner: include forks |
|
--skip-archived |
With --owner: skip archived repos |
|
--max-repos N |
With --owner: only the N most recently pushed repos |
|
--releases |
Archive release notes and assets (GitHub only) | |
--issues |
Archive issues and pull requests (GitHub only) | |
--submodules |
Upload one bundle per submodule | |
--no-state |
Do not read or write the state file (IA duplicate check still applies) | |
--max-size MB |
no limit | Skip repos larger than this |
--workers N |
4 | Parallel uploads |
--dry-run |
Preview only | |
--state-file PATH |
./watchlist_state.json |
Where the last-seen snapshots are recorded |
--json-summary PATH |
Write a run summary (counts, timings, failures) as JSON | |
--max-failures N |
Exit with code 2 if more than N repos fail | |
--work-dir DIR |
system temp | Where to put temporary clones |
--keep-work-dir |
Keep clones for iagitup --resume |
Cron, daily at 03:00:
GITHUB_TOKEN=ghp_your_token_here 0 3 * * * archive-watchlist --json-summary /var/log/watchlist.json >> watchlist.log 2>&1
Internet Archive credentials. On first run you are asked to run ia configure. Keys are read from ~/.config/internetarchive/ia.ini, ~/.ia or ~/.config/ia.ini. Get them at https://archive.org/account/s3.php.
GITHUB_TOKEN raises the GitHub API limit from 60 to 5,000 requests per hour. No scopes needed for public repos. Get one at https://github.com/settings/tokens.
GITLAB_TOKEN authenticates GitLab API calls for private projects and higher limits.
Each snapshot is one IA item named {host}-{owner}-{repo}_-_{YYYY-MM-DD_HH-MM-SS}, dated by the last push.
| File | When |
|---|---|
<name>.bundle |
always |
cover.jpg |
the platform reports an avatar |
<name>_wiki.bundle |
the repo has a wiki |
<name>_lfs.tar.gz |
the repo uses Git LFS |
submodules/<name>_submodule_<path>.bundle and <name>_gitmodules.txt |
--submodules |
releases/releases.json, releases/<tag>/<asset> |
--releases |
issues.json, pulls.json |
--issues |
Item metadata: title, date, year, creator, originalurl, pushed_date, subject (e.g. GitHub;code;software;git), stars_count, forks_count, has_lfs, and an HTML description built from the repo description, README and restore instructions. archive-watchlist adds language, topics and watchers_count, plus github_rank for top-starred runs. Fields passed with --metadata override these.
- Fetch metadata from the platform API, or from git history if there is none.
- Skip if that snapshot already exists on archive.org. For GitHub this happens before cloning.
- Clone, fetch the wiki, avatar and LFS objects, and any requested extras.
- Create and verify the bundles.
- Upload everything to one item.
In owner mode, archive-watchlist first lists the repos through the platform API. It also keeps a local state file of pushed_at per repo, so unchanged repos are skipped with no network calls. Delete a repo's entry to force a re-archive.
wget https://archive.org/download/<identifier>/<name>.bundle git clone <name>.bundle my-repo
LFS objects:
cd my-repo wget https://archive.org/download/<identifier>/<name>_lfs.tar.gz tar -xzf <name>_lfs.tar.gz -C .git/ git lfs install && git lfs checkout
Submodules: clone each submodules/*.bundle into the path listed in <name>_gitmodules.txt.
Issues and pull requests are welcome at https://github.com/gdamdam/iagitup. Tests run on Python 3.10 to 3.13.
GPLv3. Copyright (C) 2018-2026 Giovanni Damiola. See LICENSE.