GeezerVison

This page is about the Memcached infrastructure (clusters, pools and servers) that backs MediaWiki and Thumbor in production.

For how MediaWiki itself uses this cluster (WANObjectCache, ParserCache, MicroStash), see Memcached for MediaWiki. For the proxy that MediaWiki talks to in order to reach these servers, see mw-mcrouter.

This page is not about other Memcached clusters in production, such as, Wikimedia Cloud Services, and Swift.

There are three logical pools of memcached servers for MediaWiki:

• Main: 18 shards per DC, running on the mc10XX hosts (in Eqiad) and mc20XX hosts (in Codfw).
• Gutter: 3 shards per DC, hosted on mc-gp100x and mc-gp200x hosts. This is a fallback pool for the Main pool, and should not generally receive traffic. See mcrouter routes for how failover to the Gutterpool works.
• mc-wf (Wikifunctions): a separate Memcached cluster dedicated to the Wikifunctions service. Unlike the Main/Gutter pools above, mw-wikifunctions does not go through the shared mw-mcrouter daemonset, but rather talks to its own in-pod mcrouter container.

Cross-DC traffic between the MediaWiki and Main pools (e.g. WANCache purges relayed by mcrouter's WAN route) is encrypted using TLS.

The list of Memcached servers for the Main and Gutter pools is defined in Puppet, under profile::mediawiki::mcrouter_wancache::shards. This list is passed to mw-mcrouter after being parsed by some wonderful code.

After a change to this list has been merged and propagated to the deployment server, mw-mcrouter needs to be redeployed to pick up the change.

For how to replace a host in a pool, see mw-mcrouter.

See Performance/Runbooks/Analyze memcached for how to use memkeys or cachedump to inspect what a memcached instance is storing, for example to find hot keys or an unexpectedly large key.

See SRE/Service_Operations/Documentation/Reboots#Datastores

• Memcached server failure
• Performance/Runbooks/Analyze memcached (how to use memkeys or cachedump)

• Memcached Dashboard
• Memcached Gutterpool status
• mw-mcrouter

• list of mediawiki memcached production servers

• Memcached for MediaWiki
• mw-mcrouter
• Redis