GeezerVisonThe World Wide Web, restored to 800 pixels of glory. |
| Home|About|Gaming cabinet|Archives|How it works|Starter sites |
GeezerVison is showing wikitech.wikimedia.org — Debugging in production - Wikitech See also: WikimediaDebug
Debugging a web requestExternallyUse X-Wikimedia-Debug to make a request bypass the Varnish cache and route to a specific debug server. DirectlySee also: Performance/Guides/Measure backend performance#Benchmarking and load testing in production You can bypass the CDN frontend and make a web request directly to a MediaWiki backend server, by using Curl from a Deployment server: krinkle@deploy1003:~$ curl -i --connect-to ::mwdebug.discovery.wmnet:4444 'https://test.wikipedia.org/w/load.php' HTTP/1.1 200 OK server: mw-debug.eqiad.pinkunicorn-XXX-XXX … /* This file is the Web entry point for MediaWiki's ResourceLoader: … */ Directly by server groupTo target a different server groups in WikimediaDebug, you can find the appropriate hostname and port number in puppet:/trafficserver/x-wikimedia-debug-routing.lua: krinkle@deploy1003:~$ curl -i --connect-to ::mwdebug-next.svc.codfw.wmnet:4453 'https://test.wikipedia.org/w/load.php' HTTP/1.1 200 OK server: mw-debug.codfw.next-XXX-XXX … /* This file is the Web entry point for MediaWiki's ResourceLoader: … */ While you can target mw-experimental in the same way, it is recommended to instead run Curl from the mw-experimental host itself, as this keeps your work contained there and avoids mistakes (such as mistakenly editing files on the deployment server; or testing the wrong data center and not seeing your changes). When you're ssh-ed to mw-experimental you can use Curl like so: krinkle@wikikube-worker-exp1001:~$ curl -i --connect-to ::$HOSTNAME:4456 'https://test.wikipedia.org/w/load.php' HTTP/1.1 200 OK server: mw-experimental.eqiad.pinkllama-XXX-XXX … /* This file is the Web entry point for MediaWiki's ResourceLoader: … */
Directly by wiki domain nameYou can use any URL as-is: krinkle@deploy1003:~$ curl -i --connect-to ::mwdebug.discovery.wmnet:4444 'https://www.wikimedia.org/' HTTP/1.1 200 OK server: mw-debug.eqiad.pinkunicorn-XXX-XXX … <!DOCTYPE html> <html lang="mul" dir="ltr"> <head> <meta charset="utf-8"> <title>Wikimedia</title> <meta name="description" content="Wikimedia is a global movement whose mission is to bring free educational content to the world."> … Directly in Beta ClusterThis is currently the only way to debug in Beta Cluster, since internal HTTPS is not available there: user@deployment-mediawiki11:~$ curl -i --connect-to ::$HOSTNAME -H 'X-Forwarded-Proto: https' 'http://en.wikipedia.beta.wmflabs.org/wiki/Main_Page' HTTP/1.1 200 OK Server: deployment-mediawiki11.deployment-prep.eqiad1.wikimedia.cloud … <!DOCTYPE html> … Directly by layerTargetting Apache or php-fpm directly is currently unsupported since 2025 with the move to Kubernetes (phab:T397498). See phab:T405688 and phab:T276994 for the latest conversations.
The above example does still go through the Kubernetes host networking layer, and the local Envoy proxy in front of Apache inside each MediaWiki On Kubernetes pod. There is currently no way to make plain HTTP requests that target Apache directly.[citation needed] There is also currently no way to target php-fpm directly (i.e. bypass Kubernetes/Envoy/Apache together). You can get close from a deployment server, by using P84273 to open a shell inside the mw-experimental pod (see also T405688). But there is no curl or netcat available within those pods, and the php-fpm socket isn't exposed to the host.[citation needed] etv and only limitation here is that it means you can't debug Apache rewrite rules, and are limited to mw-experimental (going inside the pod for mw-debug or mw-web is for SRE/root only):
user@mwdebug1002:~$ curl -i --connect-to ::$HOSTNAME 'http://test.wikipedia.org/wiki/Main_Page' HTTP/1.1 302 Found Server: mwdebug1002.eqiad.wmnet Location: https://test.wikipedia.org/wiki/Main_Page Notes about Curl methodology
Pushing code to a debug serverDevelopers can debug code changes using Mw-experimental Conditional codeAny changes you make by debugging in this way will be overwritten by the next MediaWiki deployment (e.g. train or backport window). If changes are meant to last more than an hour, please commit them to Git instead and deploy them the normal way. Use conditionals such as Testing itSee also: X-Wikimedia-Debug#Staging changes Use X-Wikimedia-Debug in a browser to route one of your regular web requests to the debug server you have staged code on. Debugging databasesSee also: How to run queries on live data, MariaDB#Sections and shards, and Maintenance_scripts#SQL_queries
From a deployment server, use the Run Note that some DB clusters have a different name inside MediaWiki. For example "x1" is known as "extension1", for the purposes of the Examples: # Connect to s3.test2wiki database on a live replica in production. $ sql test2wiki # Connect to s7.centralauth $ sql centralauth $ mwscript mysql.php --wiki metawiki --wikidb centralauth # Connect to x1.wikishared $ sql wikishared $ mwscript mysql.php --wiki metawiki --cluster extension1 --wikidb wikishared # Connect to x3.wikidatawiki $ sql wikidatawiki --cluster extension3 $ mwscript mysql.php --wiki metawiki --cluster extension3 --wikidb wikidatawiki # Connect to an arbitrary host, such as for: # * a depooled host # * a MainStash or ParserCache host (which do not use $wgLBFactoryConf) # # MainStash (ms#) and ParserCache (pc#) hosts can be found at # https://noc.wikimedia.org/dbconfig/eqiad.json # https://noc.wikimedia.org/dbconfig/codfw.json # $ sql mainstash --raw-host db1152 $ mwscript mysql.php --wiki metawiki --raw-host db1152 --wikidb mainstash $ sql parsercache --raw-host 10.64.0.57 $ mwscript mysql.php --wiki metawiki --raw-host 10.64.0.5 --wikidb parsercache # List pooled hosts (s1) $ sql enwiki --list-hosts $ mwscript mysql.php --wiki enwiki --list-hosts db0001 db0002 db0003 # List pooled hosts (centralauth) $ sql centralauth --list-hosts $ mwscript mysql.php --wiki enwiki --wikidb centralauth --list-hosts … # List pooled hosts (external cluster) $ mwscript mysql.php --wiki metawiki --cluster extension1 --list-hosts … Debugging a maintenance script
Uncommitted maintenance scripts can be executed as follows: [urbanecm@deploy1003 ~]$ cat test.php <?php declare( strict_types = 1 ); namespace Urbanecm; use MediaWiki\Maintenance\Maintenance; $IP = getenv( 'MW_INSTALL_PATH' ); require_once "$IP/maintenance/Maintenance.php"; class TestScript extends Maintenance { public function execute() { $this->output("Hello from the k8s side\n"); } } $maintClass = TestScript::class; require_once RUN_MAINTENANCE_IF_MAIN; [urbanecm@deploy1003 ~]$ mwscript-k8s --file=test.php --follow -- /data/test.php --wiki=testwiki ⏳ Starting /data/test.php on Kubernetes as job mw-script.eqiad.8k82ojao ... 🚀 Job is running. 📜 Streaming logs: Hello from the k8s side [urbanecm@deploy1003 ~]$ More details are available in a Phabricator discussion at phab:T341553#11424364. An alternative is shelling out to mw-experimental, which is not officially supported yet (phab:T405688), but there is a bash script making it possible. However, using that route currently requires detailed knowledge of how maintenance scripts are executed in production. As of April 2026, the mwscript-k8s route described above is preferrable for running uncommitted maintenance scripts. Debugging logsTo inspect how log messages are sent over the network from MediaWiki PHP to Logstash, read Application servers/Runbook#Logging. To enable verbose logging in production, it is recommended to work on an mwdebug host and then use WikimediaDebug (either via your browser, or via curl from a shell on the mwdebug host). If you are investigating problems with a specific appserver or otherwise can't rely on Logstash, follow WikimediaDebug#Debug logging (CLI) to temporarily read logs on the same server. Debugging in shellTo open a command-line shell to PHP, log in to a deployment server and run eval.php or shell.php as maintenance script. On Kubernetes: deploy1002:~$ mwscript-k8s --attach -- shell.php --wiki=testwiki Or directly on deployment server: deploy1002:~$ mwscript shell.php testwiki Where testwiki can be any wiki dbname. You can execute arbitrary MW code from here. Ad-hoc log messagesThe recommended approach to ad-hoc logging in production is If the data will be logged from a mwdebug host via CLI or via WikimediaDebug, then the message will show up at Logstash dashboard: mwdebug If the data is expected to come from a different host (e.g. only reproducible there, or waiting for the condition to be hit organically), then the message will show up at Logstash dashboard: mediawiki where you can query for Ad-hoc command line loggingTo reproduce an issue programmatically, it is recommended to follow #Debugging in shell instead without modifying source code on disk or running modified programs. If an issue is difficult to reproduce and you need to modify a maintenance script to log some information quickly you can use the
error_log Type 4 corresponds to STDERR in CLI. For web requests via Apache, STDERR is not defined and these go to syslog instead. For such web requests, these will end up in Logstash as Syslog will end up on disk, readable via |
| You are visitor 01402503 — GeezerVison does not store the pages you look up. Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required. |