Wordlists for web fuzzing: curated micro, categorized short/long, and combined final lists.

• onelistforallmicro.txt: curated list (maintained manually).
• dict/<category>_short.txt: per-category curated wordlist (small/quality sources).
• dict/<category>_long.txt: per-category comprehensive wordlist (all sources).
• onelistforall.txt: micro + all *_short.txt, deduplicated.
• onelistforall_big.txt: everything combined, deduplicated.

1. Sync (update): Clones ~36 wordlist repos into sources/.
2. Classify (classify): Walks every .txt file in sources/, classifies each into categories using path structure, filename keywords, and content sampling. Produces classification_index.json.
3. Build (build --all-categories): For each category, merges classified source files into dict/{cat}_short.txt and dict/{cat}_long.txt with filtering and deduplication.
4. Assemble (assemble): Combines micro + all shorts into onelistforall.txt, and everything into onelistforall_big.txt.
5. Package (package): Creates 7z archives with checksums.
6. Publish (publish): Auto-commits dict/ changes and pushes to remote.

• Go 1.22+
• git (for update)
• 7z (only for package)

# Check dependencies
go run ./cmd/olfa check

# List sources and categories
go run ./cmd/olfa list
go run ./cmd/olfa list --categories

# Sync source repos
go run ./cmd/olfa update
go run ./cmd/olfa update --source SecLists

# Classify source files
go run ./cmd/olfa classify
go run ./cmd/olfa classify --format json

# Build all category wordlists
go run ./cmd/olfa build --all-categories
go run ./cmd/olfa build --category wordpress --variant short

# Assemble final combined lists
go run ./cmd/olfa assemble

# Validate and package
go run ./cmd/olfa validate-categories
go run ./cmd/olfa stats
go run ./cmd/olfa package

# Full pipeline (all steps in one command)
go run ./cmd/olfa pipeline
go run ./cmd/olfa pipeline --dry-run
go run ./cmd/olfa pipeline --skip-publish       # skip git commit+push
go run ./cmd/olfa pipeline --skip-update        # skip git fetch
go run ./cmd/olfa pipeline --commit-msg "my custom message"

# Option A: single command
go run ./cmd/olfa pipeline

# Option B: step by step
go run ./cmd/olfa check
go run ./cmd/olfa update
go run ./cmd/olfa classify
go run ./cmd/olfa build --all-categories
go run ./cmd/olfa assemble
go run ./cmd/olfa validate-categories
go run ./cmd/olfa package
git add dict/ && git commit -m "chore: update dict/ wordlists" && git push

The repo includes ~418 category wordlists in dict/ (~930 MB). However, 6 files exceed GitHub's 100 MB file size limit and are not included in the repository:

To generate them locally, run:

go run ./cmd/olfa pipeline

Running the full pipeline (syncing sources + building all categories) requires ~15 GB of disk space.

Everything is controlled by two files:

• configs/pipeline.yml — sources, filters, classification rules, dedup settings, release config.
• configs/taxonomy.json — category taxonomy (236 categories with aliases).

Edit configs/pipeline.yml and add an entry to the sources array:

{
  "name": "my-wordlists",
  "repo": "username/repo-name",
  "branch": "main",
  "paths": ["all"],
  "tags": ["directories", "api"],
  "priority": "medium"
}

After adding a source, run the pipeline to pull and classify it:

go run ./cmd/olfa pipeline

Or sync just the new source:

go run ./cmd/olfa update --source my-wordlists

Global filters in pipeline.yml control what lines are kept or dropped:

• regex_denylist — lines matching any pattern are removed (e.g., URLs, UUIDs, image extensions)
• max_line_len — lines longer than this are dropped (default: 100 chars)
• trim — strip leading/trailing whitespace
• drop_empty — remove blank lines

Per-category filters can be set in category_filters (e.g., subdomains only allow valid hostname characters).

Each category produces two wordlists:

• *_short.txt — only from high priority sources, or files with fewer than 5000 lines, or filenames containing keywords like common, short, top, default.
• *_long.txt — all sources combined.

Thresholds and keywords are configurable in classification.short_line_threshold, classification.short_keywords, and classification.long_keywords.

Categories are defined in configs/taxonomy.json. Each has a canonical name and aliases. Source files are matched to categories by:

1. Explicit path rules (e.g., Discovery/DNS/* → subdomains)
2. Directory path keywords matched against taxonomy
3. Filename keywords matched against taxonomy
4. Content sampling with regex patterns (fallback)
5. Source-level tags (last resort)

To list all available categories:

go run ./cmd/olfa list --categories
go run ./cmd/olfa list --categories --format json