GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing wikitech.wikimedia.org — Dumps - Wikitech

See Help:Shared storage#Dumps for information on using Dumps data from Toolforge.

Please note that these dumps are now running under Airflow as Kubernetes workloads, rather than on the bare-metal Dump servers.

This documentation is currently being updated to reflect the new configuration.

Documentation about Dumps is stored in a few different wikis:

These Wikitech docs are for maintainers of the various dumps.
- Information about the clouddumps servers serving mirrors to various clients can be found on Portal:Data Services/Admin/Dumps.
Information for users of the dumps can be found at Meta-wiki's m:Data dumps page.
Information for developers can be found at MediaWiki-wiki's mw:SQL/XML Dumps page.

Daily checks

These daily checks are now integrated with the Data Platform Engineering/Ops week rota, so engineers should include monitoring of Dumps with other routine data pipeline checks.

emails to the data-engineering-alerts internal mailing list.
xmldatadumps-l public mailing list
Phabricator Dumps-Generation workboard
https://dumps.wikimedia.org/ (mentions the current run, unless idle)

Dumps types

We produce several types of dumps. For information about deployment of updates, architecture of the dumps, and troubleshooting each dump type, check the appropriate entry below.

xml/sql dumps which contain revision metadata and content for public Wikimedia projects, along with contents of select sql tables
adds/changes dumps which contain a daily xml dump of new pages or pages with new revisions since the previous run, for public Wikimedia projects
Wikidata entity dumps which contain dumps of 'entities' (Qxxx) in various formats, and a dump of lexemes, run once a week.
category dumps which contain weekly full and daily incremental category lists for public Wikimedia projects, in rdf format
other miscellaneous dumps including content translation dumps, cirrus search dumps, and global block information.

Other datasets are also provided for download, such as page view counts; these datasets are managed by other folks and are not documented here.

Service

Airflow

Please see Dumps/Airflow for more information on how the dumps are now executed and published.

Hardware

Dumps servers that provide the dumps to the public, to our mirrors, and via nfs to Wikimedia Cloud Services and stats host users

The snapshot and dumpsdata servers below are now legacy and will be decommissioned soon, following the migration to Airflow.

Dumps snapshot hosts that run scripts to generate the dumps
Dumps datastores where the snapshot hosts write intermediate and final dump output files, which are later published to our web servers

Adding new dumps

If you are interested in adding a new dumpset, please check the guidelines (still in draft form).

If you are working with wikibase dumps of some sort, you might want to look at a code walkthrough; see Dumps/Wikibase dumps overview.

Not an SLO but...

Dumps have never had an SLO. But current dumps maintainers have a set of unofficial standards for responsiveness and reliability.

We try to reply to incoming tasks filed, emails from folks interested in hosting mirrors, and requests for information, within 2 business days. This may be extended if the dumps maintainers are ill or out of the office for other reasons.

When the SQL/XML dumps for one or more wikis are broken, we do our best to respond to the breakage within 24 hours; this usually includes the filing of a task in Phabricator and some investigation of the problem. If changes to MediaWiki code are required, we will coordinate that work even when we do not write the patch, also arranging for a timely backport and deployment of the patch.

We do our best to ensure that all jobs for the SQL/XML dumps for every wiki are complete before the start of the next run. So for the run starting on the 1st of the month, all jobs on all wikis must be complete before the 20th of the month, and for the run starting on the 20th, all jobs for all wikis must be complete before the end of the month. This sometimes requires work on days off, or beyond the regular workday, in which case future workdays might be shortened to compensate.

Testing changes to the dumps or new scripts

See Dumps/Testing for more about this.

Mirrors

If you are adding a mirror, see Dumps Mirror setup.

Source code

operations/dumps.git

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Dumps&oldid=2361574"

Category:

Dumps

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.