🌐 Guide and tools to run a full offline mirror of Wikipedia.org with three different approaches: Nginx caching proxy, Kiwix + ZIM dump, and MediaWiki/XOWA + XML dump

html docker nginx wiki docker-compose mediawiki wikipedia archiving datascience kiwix zim wikipedia-dump wikipedia-mirror openzim xowa internet-archiving mwdumper kiwix-offline-wikipedia

• Updated Mar 25, 2025
• PLpgSQL

A command-line tool to extract plain text from Wikipedia dumps with category and section filtering

ruby nlp machine-learning wikipedia corpus wikipedia-dump

• Updated Oct 28, 2026
• Ruby

An F/OSS solution combining AI with Wikipedia knowledge via a RAG pipeline

natural-language-processing ai wikipedia wikipedia-dump language-processing rag retrieval-augmented-generation

• Updated Jan 12, 2025
• Python

Corpus creator for Chinese Wikipedia

nlp wikipedia-dump wikipedia-corpus chinese-corpus

• Updated Jun 30, 2021
• Python

Reading the data from OPIEC - an Open Information Extraction corpus

nlp natural-language-processing wiki wikipedia corpus information-extraction dataset corpora corpus-data nlp-resources wikipedia-dump corpus-tools natural-language-understanding open-information-extraction dataset-interface wikipedia-corpus corpus-processing nlp-datasets

• Updated Jun 12, 2019
• Java

Wikipedia-based Explicit Semantic Analysis, as described by Gabrilovich and Markovitch

java vector java-8 concept lucene esa wikipedia-dump semantic-analysis explicit-semantic-analysis

• Updated May 13, 2020
• Java

Downloads and imports Wikipedia page histories to a git repository

git mediawiki wikipedia-dump

• Updated Oct 6, 2026
• Python

Extracting useful metadata from Wikipedia dumps in any language.

multilingual redirects wikipedia python3 disambiguation wikipedia-dump metadata-extraction wikiextractor

• Updated Sep 20, 2019
• Python

Python package for working with MediaWiki XML content dumps

python wikipedia wikia wikipedia-dump fandom python3-library wikipedia-corpus xml-dump mediawiki-dump

• Updated Nov 2, 2026
• Python

A simple utility to index wikipedia dumps using Lucene.

java lucene wikipedia-dump

• Updated Oct 13, 2020
• Java

Network Visualizer for the 'Geschichten aus der Geschichte' Podcast

javascript python data-science database podcast wikipedia data-visualization sqlite3 network-analysis wikipedia-dump

• Updated Sep 6, 2025
• Python

Collects a multimodal dataset of Wikipedia articles and their images

database wikipedia wikipedia-api wikipedia-bot data-collection wikipedia-page wikipedia-viewer data-processing multimodality wikipedia-dump data-cleaning wikipedia-search wikipedia-corpus wikipedia-scraper wikipedia-entries multimodal-learning multimodal multimodal-datasets multimodal-data multimodal-representation

• Updated Mar 25, 2023
• Python

Simple Python script to download the entirety of Wikipedia on a weekly basis.

wikipedia wikipedia-bot wikipedia-dump

• Updated May 2, 2025
• Python

A library that assists in traversing and downloading from Wikimedia Data Dumps and their mirrors.

wikipedia wikidata wikimedia wikipedia-dump

• Updated May 12, 2026
• Python

A Python toolkit to generate a tokenized dump of Wikipedia for NLP

multilingual nlp wikipedia dump wikipedia-dump tokenize

• Updated May 3, 2024
• Python

Node.js module for parsing the content of wikipedia articles into javascript objects

nodejs javascript parser json wikipedia wikipedia-dump

• Updated Oct 24, 2021
• JavaScript

Convert Wikipedia XML dump files to JSON or Text files

wikipedia wikipedia-api wikipedia-page wikipedia-dump wikipedia-search wikipedia-corpus wikipedia-entries

• Updated Sep 23, 2020
• C#

Convert WIKI dumped XML (Chinese) to human readable documents in markdown and txt.

wikipedia wikipedia-dump wikipedia-corpus

• Updated Mar 25, 2020
• Python

Scripts to download the Wikipedia dumps (available at https://dumps.wikimedia.org/ )

wikipedia wikipedia-dump

• Updated Apr 19, 2024
• Shell

Contains code to build a search engine by creating an index and perform search over Wikipedia data.

search-engine information-retrieval information-extraction xml-parser wikipedia-dump

• Updated Oct 11, 2021
• Python