Dingo: A Comprehensive AI Data, Model and Application Quality Evaluation Tool

spark data-validation datascience openai data-quality vlm common-crawl data-evaluation dataquality hallucination data-quality-report data-quality-assessment llm qwen data-agent deepseek hallucination-detection llm-as-a-judge opencompass agent-as-a-judge

• Updated Nov 10, 2026
• Python

Process Common Crawl data with Python and Spark

spark pyspark sparksql wet commoncrawl common-crawl warc-files wat-files

• Updated Sep 17, 2026
• Python

News crawling with StormCrawler - stores content as WARC

crawler news web-crawler apache-storm warc commoncrawl common-crawl storm-crawler

• Updated Nov 7, 2026
• Java

Statistics of Common Crawl monthly archives mined from URL index files

statistics commoncrawl common-crawl

• Updated Nov 3, 2026
• HTML

🕷️ The pipeline for the OSCAR corpus

nlp crawler corpus-linguistics fasttext oscar commoncrawl common-crawl language-classification

• Updated Nov 9, 2025
• Rust

Drill into WARC web archives

security internet-archive command-line-tool warc security-tools common-crawl

• Updated Oct 16, 2024
• Go

Tools to construct and process Common Crawl webgraphs

pagerank webgraph commoncrawl common-crawl centrality-measures webgraph-framework

• Updated Nov 3, 2026
• Java

An asynchronous concurrent pipeline for classifying Common Crawl based on fastText's pipeline.

nlp corpus-linguistics fasttext common-crawl language-classification

• Updated Apr 21, 2021
• Go

The largest open corpus of classified docx documents

nlp machine-learning typescript corpus dataset docx word-documents bun common-crawl document-processing

• Updated Nov 11, 2026
• TypeScript

Various Jupyter notebooks about Common Crawl data

jupyter-notebook aws-athena commoncrawl common-crawl webarchiving webgraph-framework

• Updated Sep 8, 2026
• Jupyter Notebook

DuckDB extension to fetch pages from Wayback Machine & Common Crawl

website sql archived wayback-machine common-crawl cdx-api duckdb duckdb-extension

• Updated Oct 22, 2026
• C++

[NeurIPS 2024] 🕸 GlotCC Dataset and Pipline

crawler multlingual corpus-linguistics glot language-identification commoncrawl common-crawl glotcc multilingual-dataset glotlid

• Updated Apr 6, 2025
• Jupyter Notebook

German small and large versions of GPT2.

nlp machine-learning german language-model common-crawl gpt2

• Updated May 11, 2022
• Python

Lightweight Python utility for retrieving individual pages from the Common Crawl archives.

commoncrawl common-crawl common-crawl-with-python common-crawl-python common-crawl-data

• Updated Feb 21, 2026
• Python

The website of the Oscar Project

nlp website machine-learning hugo language-model common-crawl

• Updated Mar 27, 2025
• TeX

A fast, friendly command line for Common Crawl: URL index search, WARC fetch, Parquet columnar queries, and dataset building.

cli golang scraper open-data dataset web-scraping warc common-crawl web-archive duckdb

• Updated Aug 24, 2026
• Go

Common Crawl's processing tools

warc wat wet commoncrawl common-crawl warc-files wat-files common-crawl-data wet-files

• Updated Mar 26, 2026
• C#

Reproducible measurement of how many company job boards you can reach through the public APIs of Greenhouse, Ashby, Lever, Recruitee, Teamtailor and Breezy. 16,361 boards and 399,398 open postings counted directly, enumerated from Common Crawl. No scraping; robots.txt checked live and against the archive.

open-data dataset web-scraping robots-txt hiring job-board greenhouse ats recruiting job-search breezy common-crawl lever job-scraper jobs-api teamtailor recruitee ashby

• Updated Sep 5, 2026
• JavaScript

Sample code to grep Common Crawl WARC files in Go, Java, Node and Python.

warc commoncrawl common-crawl

• Updated Apr 30, 2021
• Shell

MCP server for the CrawlGraph backlink-intelligence API (Common Crawl webgraph). Backlink lookups + competitor gap analysis for Claude, Cursor, Cline, and any MCP client.

mcp seo cursor backlinks webgraph claude common-crawl gap-analysis model-context-protocol mcp-server agent-skills backlink-analysis cursor-plugin claude-code-plugin gemini-cli-extension ahrefs-alternative hyperlink-graph

• Updated Nov 12, 2026
• JavaScript