GeezerVison

GeezerVison is showing wikitech.wikimedia.org (truncated) — Server Admin Log - Wikitech

2026-11-12

08:24 kharlan@deploy2003: dreamyjazz, kharlan: Continuing with deployment
08:23 kharlan@deploy2003: dreamyjazz, kharlan: Backport for hCaptcha: Enable on French Wikipedia for API editing (T444509) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
08:21 kharlan@deploy2003: Started scap sync-world: Backport for hCaptcha: Enable on French Wikipedia for API editing (T444509)
08:19 dcausse@deploy2003: Finished scap sync-world: Backport for cirrus: Provide short cluster names (T440376) (duration: 11m 06s)
08:13 dcausse@deploy2003: ebernhardson, dcausse: Continuing with deployment
08:10 dcausse@deploy2003: ebernhardson, dcausse: Backport for cirrus: Provide short cluster names (T440376) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
08:08 dcausse@deploy2003: Started scap sync-world: Backport for cirrus: Provide short cluster names (T440376)
02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 49s)
02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync

2026-11-11

23:41 brett@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp3074.esams.wmnet with reason: troubleshooting cmos failure
21:16 matthiasmullie: UTC late backport window done
21:15 mlitn@deploy2003: Finished scap sync-world: Backport for Do not Implement UserThankedListener (1.48.0-wmf.4), Squashed diff to master (1.48.0-wmf.4) (duration: 09m 15s)
21:09 mlitn@deploy2003: mlitn: Continuing with deployment
21:08 mlitn@deploy2003: mlitn: Backport for Do not Implement UserThankedListener (1.48.0-wmf.4), Squashed diff to master (1.48.0-wmf.4) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
21:06 mlitn@deploy2003: Started scap sync-world: Backport for Do not Implement UserThankedListener (1.48.0-wmf.4), Squashed diff to master (1.48.0-wmf.4)
19:47 brouberol@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/echoserver: apply
19:47 brouberol@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/echoserver: apply
19:46 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/analytics-test: apply
19:46 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/analytics-test: apply
19:46 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply
19:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply
19:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply
19:44 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply
19:44 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
19:44 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
19:44 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply
19:43 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply
19:43 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply
19:42 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply
19:42 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply
19:41 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply
19:41 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply
19:40 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply
19:40 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
19:39 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
19:39 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
19:38 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
19:38 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply
19:37 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply
19:37 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-experiment-platform: apply
19:36 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-experiment-platform: apply
19:36 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
19:36 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
19:36 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply
19:35 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply
19:35 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply
19:33 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply
19:33 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply
19:32 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply
19:32 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply
19:31 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply
19:31 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply
19:30 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply
19:30 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset: apply
19:30 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset: apply
19:30 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset-next: apply
19:29 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset-next: apply
19:28 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/spark-history: apply
19:28 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/spark-history: apply
19:28 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/spark-history: apply
19:28 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/spark-history: apply
19:27 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/spark-history: apply
19:27 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/spark-history: apply
19:27 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/spark-history: apply
19:26 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/spark-history: apply
19:21 sfaci@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply
19:20 sfaci@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply
19:14 brouberol@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
19:14 brouberol@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
19:12 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
19:10 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
16:34 inflatador: bking@deploy2003 `apt-get install python3-swiftclient` T434970
15:33 urbanecm@deploy2003: Finished scap sync-world: Backport for cswiki: Remove celebration logo (T442387), GrowthExperiments: Remove the $wgGECampaignTopics setting (T444124) (duration: 09m 26s)
15:27 urbanecm@deploy2003: urbanecm: Continuing with deployment
15:26 urbanecm@deploy2003: urbanecm: Backport for cswiki: Remove celebration logo (T442387), GrowthExperiments: Remove the $wgGECampaignTopics setting (T444124) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
15:23 urbanecm@deploy2003: Started scap sync-world: Backport for cswiki: Remove celebration logo (T442387), GrowthExperiments: Remove the $wgGECampaignTopics setting (T444124)
15:18 jforrester@deploy2003: Finished scap deploy-service wikifunctions: Deploy evaluators config to silence logs from incoming calls (1363475) (duration: 02m 20s)
15:18 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions
15:17 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions
15:16 jforrester@deploy2003: scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions
15:16 jforrester@deploy2003: Started scap deploy-service wikifunctions: Deploy evaluators config to silence logs from incoming calls (1363475)
15:13 jforrester@deploy2003: Finished scap deploy-service wikifunctions: Weekly service deploy for orchestrator (1368639) (duration: 02m 05s)
15:12 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions
15:11 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions
15:11 urbanecm@deploy2003: Finished scap sync-world: Backport for Revert^2 "Growth: enable home beta on pilot and innovation wikis" (T443176) (duration: 12m 51s)
15:11 jforrester@deploy2003: scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions
15:11 jforrester@deploy2003: Started scap deploy-service wikifunctions: Weekly service deploy for orchestrator (1368639)
15:07 jforrester@deploy2003: Finished scap deploy-service wikifunctions: Weekly service deploy for evaluators (1368638) (duration: 03m 05s)
15:06 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions
15:05 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions
15:05 urbanecm@deploy2003: urbanecm: Continuing with deployment
15:04 jforrester@deploy2003: scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions
15:04 jforrester@deploy2003: Started scap deploy-service wikifunctions: Weekly service deploy for evaluators (1368638)
15:02 gkyziridis@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
15:01 gkyziridis@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
15:00 urbanecm@deploy2003: urbanecm: Backport for Revert^2 "Growth: enable home beta on pilot and innovation wikis" (T443176) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
14:58 urbanecm@deploy2003: Started scap sync-world: Backport for Revert^2 "Growth: enable home beta on pilot and innovation wikis" (T443176)
14:57 dzahn@dns1004: END - running authdns-update
14:57 urbanecm@deploy2003: Finished scap sync-world: Backport for Hooks: Do not disable exempt beta features when auto-enroll is on (T444361) (1.48.0-wmf.4), Handle multiple updates in case creation atomic section (T444117) (1.48.0-wmf.3, 1.48.0-wmf.4) (duration: 14m 34s)
14:55 dzahn@dns1004: START - running authdns-update
14:51 urbanecm@deploy2003: stran, urbanecm: Continuing with deployment
14:45 urbanecm@deploy2003: stran, urbanecm: Backport for Hooks: Do not disable exempt beta features when auto-enroll is on (T444361) (1.48.0-wmf.4), Handle multiple updates in case creation atomic section (T444117) (1.48.0-wmf.3, 1.48.0-wmf.4) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
14:42 urbanecm@deploy2003: Started scap sync-world: Backport for Hooks: Do not disable exempt beta features when auto-enroll is on (T444361) (1.48.0-wmf.4), Handle multiple updates in case creation atomic section (T444117) (1.48.0-wmf.3, 1.48.0-wmf.4)
14:31 stran@deploy2003: Finished scap sync-world: Backport for Enable Special:SuggestedInvestigations on ptwiki (T442582) (duration: 09m 38s)
14:25 stran@deploy2003: stran: Continuing with deployment
14:24 stran@deploy2003: stran: Backport for Enable Special:SuggestedInvestigations on ptwiki (T442582) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
14:21 stran@deploy2003: Started scap sync-world: Backport for Enable Special:SuggestedInvestigations on ptwiki (T442582)
14:19 Tran: Deployed patch for T442582, T443646
13:59 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
13:59 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
13:54 trueg@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
13:53 trueg@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
13:52 trueg@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
13:51 trueg@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
13:14 dreamyjazz@deploy2003: Finished scap sync-world: Backport for hCaptcha: Stop collecting risk scores for IP block notices (T444473) (duration: 09m 24s)
13:07 dreamyjazz@deploy2003: dreamyjazz: Continuing with deployment
13:06 dreamyjazz@deploy2003: dreamyjazz: Backport for hCaptcha: Stop collecting risk scores for IP block notices (T444473) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
13:04 dreamyjazz@deploy2003: Started scap sync-world: Backport for hCaptcha: Stop collecting risk scores for IP block notices (T444473)
13:03 mvolz@deploy2003: Finished scap deploy-service citoid: PDF fallback (duration: 21m 04s)
12:47 mvolz@deploy2003: scap deploy-service citoid: [PRODUCTION/DEFAULT] codfw Deploying citoid
12:46 mvolz@deploy2003: scap deploy-service citoid: [PRODUCTION/DEFAULT] eqiad Deploying citoid
12:42 mvolz@deploy2003: scap deploy-service citoid: [STAGING/DEFAULT] staging-eqiad Deploying citoid
12:42 mvolz@deploy2003: Started scap deploy-service citoid: PDF fallback
12:26 dreamyjazz@deploy2003: Finished scap sync-world: Backport for hCaptcha: Drop CommunityRequests exemption (T426897) (duration: 11m 00s)
12:20 dreamyjazz@deploy2003: dreamyjazz: Continuing with deployment
12:17 dreamyjazz@deploy2003: dreamyjazz: Backport for hCaptcha: Drop CommunityRequests exemption (T426897) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
12:15 dreamyjazz@deploy2003: Started scap sync-world: Backport for hCaptcha: Drop CommunityRequests exemption (T426897)
12:12 dreamyjazz@deploy2003: Finished scap sync-world: Backport for hCaptcha: Require hCaptcha for all API edits on testwiki (T444470) (duration: 11m 30s)
12:06 dreamyjazz@deploy2003: dreamyjazz: Continuing with deployment
12:04 dreamyjazz@deploy2003: dreamyjazz: Backport for hCaptcha: Require hCaptcha for all API edits on testwiki (T444470) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
12:01 dreamyjazz@deploy2003: Started scap sync-world: Backport for hCaptcha: Require hCaptcha for all API edits on testwiki (T444470)
12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
11:36 zabe@deploy2003: Finished scap sync-world: Backport for Add script to remove own user page watches for users without edits (T442779) (1.48.0-wmf.4) (duration: 11m 05s)
11:30 zabe@deploy2003: zabe: Continuing with deployment
11:29 zabe@deploy2003: zabe: Backport for Add script to remove own user page watches for users without edits (T442779) (1.48.0-wmf.4) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
11:25 zabe@deploy2003: Started scap sync-world: Backport for Add script to remove own user page watches for users without edits (T442779) (1.48.0-wmf.4)
11:01 kevinbazira@deploy2003: helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' .
11:00 kevinbazira@deploy2003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' .
10:56 kevinbazira@deploy2003: helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' .
10:52 jmm@cumin2003: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging CDobbins out of all services on: 2439 hosts
10:43 dpogorzelski@deploy2003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
10:42 dpogorzelski@deploy2003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
10:41 ammarpad@deploy2003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki mediawikiwiki 'Language Converter' 'Language converter' Ammarpad --reason 'Requested at T444079'
10:40 dpogorzelski@deploy2003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'.
10:38 dpogorzelski@deploy2003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'.
10:36 dpogorzelski@deploy2003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
10:35 dpogorzelski@deploy2003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
10:30 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'.
10:29 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'.
10:03 trueg@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
09:58 trueg@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
09:40 jelto@cumin1004: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: version upgrade
09:19 aklapper@deploy2003: rebuilt and synchronized wikiversions files: group1 to 1.48.0-wmf.4 refs T442222
09:10 jelto@cumin1004: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: version upgrade
09:09 jelto@cumin1004: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: version upgrade
08:59 jelto@cumin1004: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: version upgrade
08:58 jelto@cumin1004: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: version upgrade
08:47 jelto@cumin1004: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: version upgrade
08:31 wmde-fisch@deploy2003: Finished scap sync-world: Backport for Drop TemplateData EventStreams/EventLogging config (T258917) (duration: 11m 08s)
08:25 wmde-fisch@deploy2003: wmde-fisch, reedy: Continuing with deployment
08:24 wmde-fisch@deploy2003: wmde-fisch, reedy: Backport for Drop TemplateData EventStreams/EventLogging config (T258917) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
08:20 wmde-fisch@deploy2003: Started scap sync-world: Backport for Drop TemplateData EventStreams/EventLogging config (T258917)
08:15 wmde-fisch@deploy2003: Finished scap sync-world: Backport for Enable Citoid PDF fallback on testwiki (T443469) (duration: 10m 59s)
08:09 wmde-fisch@deploy2003: wmde-fisch: Continuing with deployment
08:07 wmde-fisch@deploy2003: wmde-fisch: Backport for Enable Citoid PDF fallback on testwiki (T443469) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
08:04 wmde-fisch@deploy2003: Started scap sync-world: Backport for Enable Citoid PDF fallback on testwiki (T443469)
00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync

2026-11-10

23:52 dreamyjazz@deploy2003: Finished scap sync-world: Backport for Set $wgGlobalBlockingCentralWikiContentLanguage to 'en' (T444367) (duration: 10m 02s)
23:46 dreamyjazz@deploy2003: dreamyjazz: Continuing with deployment
23:45 dreamyjazz@deploy2003: dreamyjazz: Backport for Set $wgGlobalBlockingCentralWikiContentLanguage to 'en' (T444367) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
23:42 dreamyjazz@deploy2003: Started scap sync-world: Backport for Set $wgGlobalBlockingCentralWikiContentLanguage to 'en' (T444367)
23:38 zabe@deploy2003: Finished scap sync-world: Backport for Move reads/writes of wbc_entity_usage to x1 for wikidatawiki (T439753) (duration: 10m 18s)
23:32 zabe@deploy2003: zabe: Continuing with deployment
23:30 zabe@deploy2003: zabe: Backport for Move reads/writes of wbc_entity_usage to x1 for wikidatawiki (T439753) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
23:28 zabe@deploy2003: Started scap sync-world: Backport for Move reads/writes of wbc_entity_usage to x1 for wikidatawiki (T439753)
23:18 cooltey@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
23:17 cooltey@deploy2003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
23:17 cooltey@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
23:17 cooltey@deploy2003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
23:16 cooltey@deploy2003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
23:16 cooltey@deploy2003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
23:03 inflatador: bking@cumin2003 `curl -XDELETE "https://search.svc.codfw.wmnet:9242/enwiki_general_1794311678" T444363
22:26 cjming@deploy2003: Finished scap sync-world: Backport for Force absent request headers to null (T444205) (1.48.0-wmf.3, 1.48.0-wmf.4) (duration: 09m 03s)
22:20 cjming@deploy2003: cjming: Continuing with deployment
22:19 cjming@deploy2003: cjming: Backport for Force absent request headers to null (T444205) (1.48.0-wmf.3, 1.48.0-wmf.4) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
22:17 cjming@deploy2003: Started scap sync-world: Backport for Force absent request headers to null (T444205) (1.48.0-wmf.3, 1.48.0-wmf.4)
21:37 tchin@deploy2003: Finished deploy [analytics/refinery@23a3975]: Regular analytics weekly train [analytics/refinery@23a39754] (duration: 10m 33s)
21:26 tchin@deploy2003: Started deploy [analytics/refinery@23a3975]: Regular analytics weekly train [analytics/refinery@23a39754]
21:26 tchin@deploy2003: Finished deploy [analytics/refinery@23a3975] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@23a39754] (duration: 00m 45s)
21:25 jsn@deploy2003: Finished scap sync-world: Backport for Enable PersonalDashboard on Home pilot wikis (T444355), Enable DBSC report-only mode on all Wikimedia wikis (T393212), VisualEditor: Precompute edit suggestion counts on 100% of enwiki, jawiki, arwiki (T442767), Implement multiple proposals for Wikimedia Outreach (T444368) (duration: 16m 5
21:25 tchin@deploy2003: Started deploy [analytics/refinery@23a3975] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@23a39754]
21:19 jsn@deploy2003: jsn, sbassett, codenamenoreste, kemayo: Continuing with deployment
21:11 jsn@deploy2003: jsn, sbassett, codenamenoreste, kemayo: Backport for Enable PersonalDashboard on Home pilot wikis (T444355), Enable DBSC report-only mode on all Wikimedia wikis (T393212), VisualEditor: Precompute edit suggestion counts on 100% of enwiki, jawiki, arwiki (T442767), Implement multiple proposals for Wikimedia Outreach (T444368) sy
21:09 jsn@deploy2003: Started scap sync-world: Backport for Enable PersonalDashboard on Home pilot wikis (T444355), Enable DBSC report-only mode on all Wikimedia wikis (T393212), VisualEditor: Precompute edit suggestion counts on 100% of enwiki, jawiki, arwiki (T442767), Implement multiple proposals for Wikimedia Outreach (T444368)
20:11 kemayo@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
20:10 kemayo@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
20:09 kemayo@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
20:08 kemayo@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
20:08 kemayo@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
20:08 kemayo@deploy2003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
20:01 kemayo@deploy2003: helmfile [codfw] DONE helmfile.d/services/editcheck-headless: apply
20:01 kemayo@deploy2003: helmfile [codfw] START helmfile.d/services/editcheck-headless: apply
19:58 kemayo@deploy2003: helmfile [eqiad] DONE helmfile.d/services/editcheck-headless: apply
19:57 kemayo@deploy2003: helmfile [eqiad] START helmfile.d/services/editcheck-headless: apply
19:57 kemayo@deploy2003: helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply
19:56 kemayo@deploy2003: helmfile [staging] START helmfile.d/services/editcheck-headless: apply
19:52 kemayo@deploy2003: helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply
19:52 kemayo@deploy2003: helmfile [staging] START helmfile.d/services/editcheck-headless: apply
18:44 tchin@deploy2003: Finished scap sync-world: Backport for EventStreamConfig: Refine app_base as medium job (T444095) (duration: 13m 41s)
18:38 tchin@deploy2003: tchin, aqu: Continuing with deployment
18:32 tchin@deploy2003: tchin, aqu: Backport for EventStreamConfig: Refine app_base as medium job (T444095) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
18:30 tchin@deploy2003: Started scap sync-world: Backport for EventStreamConfig: Refine app_base as medium job (T444095)
17:31 urbanecm@deploy2003: Finished scap sync-world: Backport for beta: Enable PersonalDashboard on all wikis (T444353), [Growth] beta: Enable Special:Home as a beta feature on all beta wikis (T443176) (duration: 11m 08s)
17:25 urbanecm@deploy2003: urbanecm: Continuing with deployment
17:25 urbanecm@deploy2003: urbanecm: Backport for beta: Enable PersonalDashboard on all wikis (T444353), [Growth] beta: Enable Special:Home as a beta feature on all beta wikis (T443176) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
17:20 urbanecm@deploy2003: Started scap sync-world: Backport for beta: Enable PersonalDashboard on all wikis (T444353), [Growth] beta: Enable Special:Home as a beta feature on all beta wikis (T443176)
16:35 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns3.*
16:33 sukhe@dns1004: END - running authdns-update
16:31 sukhe@dns1004: START - running authdns-update
16:31 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns3.*,service=authdns-update
16:22 trueg@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
16:19 trueg@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
16:17 trueg@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
16:13 trueg@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
16:11 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool esams [reason: network upgrade, T440349]
16:11 ayounsi@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool esams [reason: network upgrade, T440349]
15:30 gmodena@deploy2003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
15:29 gmodena@deploy2003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
15:24 gmodena@deploy2003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
15:23 gmodena@deploy2003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
14:40 Lucas_WMDE: UTC afternoon backport+config window done
14:40 lucaswerkmeister-wmde@deploy2003: Finished scap sync-world: Backport for CommonSettings: set $wgCampaignEventsArticleQualitySupportedWikis (T440359) (duration: 14m 22s)
14:34 lucaswerkmeister-wmde@deploy2003: lucaswerkmeister-wmde, daimona: Continuing with deployment
14:32 lucaswerkmeister-wmde@deploy2003: lucaswerkmeister-wmde, daimona: Backport for CommonSettings: set $wgCampaignEventsArticleQualitySupportedWikis (T440359) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
14:25 lucaswerkmeister-wmde@deploy2003: Started scap sync-world: Backport for CommonSettings: set $wgCampaignEventsArticleQualitySupportedWikis (T440359)
14:20 dreamyjazz@deploy2003: Finished scap sync-world: Backport for Follow-ups for CheckUser AbuseFilter variables (T234155) (1.48.0-wmf.4), Enable CheckUser AbuseFilter variables on testwiki (T234155) (duration: 13m 37s)
14:14 dreamyjazz@deploy2003: dreamyjazz: Continuing with deployment
14:11 dreamyjazz@deploy2003: dreamyjazz: Backport for Follow-ups for CheckUser AbuseFilter variables (T234155) (1.48.0-wmf.4), Enable CheckUser AbuseFilter variables on testwiki (T234155) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
14:06 dreamyjazz@deploy2003: Started scap sync-world: Backport for Follow-ups for CheckUser AbuseFilter variables (T234155) (1.48.0-wmf.4), Enable CheckUser AbuseFilter variables on testwiki (T234155)
13:46 ladsgroup@deploy2003: Finished scap sync-world: Backport for DASH: Add "auto" resolution as default and still show other resolutions (T443728) (1.48.0-wmf.4, 1.48.0-wmf.3) (duration: 18m 18s)
13:40 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
13:32 ladsgroup@deploy2003: ladsgroup: Backport for DASH: Add "auto" resolution as default and still show other resolutions (T443728) (1.48.0-wmf.4, 1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
13:28 ladsgroup@deploy2003: Started scap sync-world: Backport for DASH: Add "auto" resolution as default and still show other resolutions (T443728) (1.48.0-wmf.4, 1.48.0-wmf.3)
12:58 kharlan@deploy2003: Finished scap sync-world: Backport for AbuseReview: Delay the link spam model check job (T442170) (1.48.0-wmf.3) (duration: 12m 21s)
12:52 kharlan@deploy2003: kharlan: Continuing with deployment
12:50 kharlan@deploy2003: kharlan: Backport for AbuseReview: Delay the link spam model check job (T442170) (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
12:46 kharlan@deploy2003: Started scap sync-world: Backport for AbuseReview: Delay the link spam model check job (T442170) (1.48.0-wmf.3)
12:05 gmodena@deploy2003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
12:05 gmodena@deploy2003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
12:05 gmodena@deploy2003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
12:05 gmodena@deploy2003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
12:04 gmodena@deploy2003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
12:04 gmodena@deploy2003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
12:03 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp3074.mgmt.esams.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
11:55 gmodena@deploy2003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
11:55 gmodena@deploy2003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
11:49 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host cp3074.mgmt.esams.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
11:42 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
11:42 ayounsi@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: new cr1-esams - ayounsi@cumin1004"
11:41 ayounsi@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: new cr1-esams - ayounsi@cumin1004"
11:35 ayounsi@cumin1004: START - Cookbook sre.dns.netbox
10:57 ayounsi@cumin1004: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
10:54 ayounsi@cumin1004: START - Cookbook sre.dns.netbox
10:37 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on asw1-bw27-esams,asw1-bw27-esams IPv6,asw1-bw27-esams.mgmt with reason: Switch maintenance
10:35 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 39 hosts with reason: Switch maintenance
09:28 aklapper@deploy2003: rebuilt and synchronized wikiversions files: group0 to 1.48.0-wmf.4 refs T442222
09:22 XioNoX: set graceful shutdown on cr1-esams transit/ix bgp - T440349
09:02 matthiasmullie: UTC morning backport window done
09:01 mlitn@deploy2003: Finished scap sync-world: Backport for Squashed diff to master (1.48.0-wmf.3) (duration: 13m 06s)
08:55 mlitn@deploy2003: mlitn: Continuing with deployment
08:55 mlitn@deploy2003: mlitn: Backport for Squashed diff to master (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
08:48 mlitn@deploy2003: Started scap sync-world: Backport for Squashed diff to master (1.48.0-wmf.3)
08:44 chlod@deploy2003: Finished scap sync-world: Backport for tcywiki: revert 10 anniversary logo (T434176), bjnwiktionary: update logos (T444085) (duration: 15m 43s)
08:43 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool esams [reason: network upgrade, T440349]
08:43 ayounsi@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool esams [reason: network upgrade, T440349]
08:42 ayounsi@cumin1004: END (FAIL) - Cookbook sre.dns.admin (exit_code=99) DNS admin: depool esams [reason: network upgrade, T440349]
08:42 ayounsi@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool esams [reason: network upgrade, T440349]
08:41 jgiannelos@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
08:41 jgiannelos@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
08:36 chlod@deploy2003: chlod, anzx: Continuing with deployment
08:35 chlod@deploy2003: chlod, anzx: Backport for tcywiki: revert 10 anniversary logo (T434176), bjnwiktionary: update logos (T444085) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
08:28 chlod@deploy2003: Started scap sync-world: Backport for tcywiki: revert 10 anniversary logo (T434176), bjnwiktionary: update logos (T444085)
08:15 jdrewniak@deploy2003: Finished scap sync-world: Backport for Bumping portals to master (T128546) (duration: 18m 26s)
08:04 jdrewniak@deploy2003: jdrewniak: Continuing with deployment
08:03 jdrewniak@deploy2003: jdrewniak: Backport for Bumping portals to master (T128546) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
07:56 jdrewniak@deploy2003: Started scap sync-world: Backport for Bumping portals to master (T128546)
05:02 mwpresync@deploy2003: Pruned MediaWiki: 1.47.0-wmf.24 (duration: 02m 26s)
04:42 mwpresync@deploy2003: Finished scap sync-world: testwikis to 1.48.0-wmf.4 refs T442222 (duration: 39m 18s)
04:03 mwpresync@deploy2003: Started scap sync-world: testwikis to 1.48.0-wmf.4 refs T442222
02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s)
02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
01:06 zabe: run populateLuAttachmentMethod.php # T442322
00:54 zabe@deploy2003: Finished scap sync-world: Backport for Move reads/writes of wbc_entity_usage to x1 for enwiki (T439753) (duration: 11m 47s)
00:47 zabe@deploy2003: zabe: Continuing with deployment
00:47 zabe@deploy2003: zabe: Backport for Move reads/writes of wbc_entity_usage to x1 for enwiki (T439753) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
00:42 zabe@deploy2003: Started scap sync-world: Backport for Move reads/writes of wbc_entity_usage to x1 for enwiki (T439753)
00:25 zabe@deploy2003: Finished scap sync-world: Backport for Disable automatic user page watching on loginwiki (T442779) (duration: 18m 24s)
00:16 zabe@deploy2003: zabe: Continuing with deployment
00:15 zabe@deploy2003: zabe: Backport for Disable automatic user page watching on loginwiki (T442779) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
00:07 zabe@deploy2003: Started scap sync-world: Backport for Disable automatic user page watching on loginwiki (T442779)
00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync

2026-11-09

23:11 maryum: Deployed security fix for T443539, T440392, T435882, T442600
21:23 akhatun@deploy2003: helmfile [codfw] DONE helmfile.d/services/lac-lambdas: apply
21:22 akhatun@deploy2003: helmfile [codfw] START helmfile.d/services/lac-lambdas: apply
21:19 akhatun@deploy2003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply
21:19 akhatun@deploy2003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply
21:18 egardner@deploy2003: Finished scap sync-world: Backport for Enable the ShareCard feature on all Wikipedias (Minerva only) (T439162) (duration: 12m 48s)
21:17 akhatun@deploy2003: helmfile [staging] DONE helmfile.d/services/lac-lambdas: apply
21:16 akhatun@deploy2003: helmfile [staging] START helmfile.d/services/lac-lambdas: apply
21:12 egardner@deploy2003: egardner: Continuing with deployment
21:11 egardner@deploy2003: egardner: Backport for Enable the ShareCard feature on all Wikipedias (Minerva only) (T439162) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
21:05 egardner@deploy2003: Started scap sync-world: Backport for Enable the ShareCard feature on all Wikipedias (Minerva only) (T439162)
20:19 gmodena@deploy2003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
20:19 gmodena@deploy2003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
19:44 gmodena@deploy2003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
19:44 gmodena@deploy2003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
19:01 btullis@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
19:00 btullis@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
18:06 cgoubert@deploy2003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
18:06 cgoubert@deploy2003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
18:06 cgoubert@deploy2003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
18:06 cgoubert@deploy2003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
17:54 cgoubert@deploy2003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
17:54 cgoubert@deploy2003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
17:49 cgoubert@deploy2003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
17:48 cgoubert@deploy2003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
17:45 trueg@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
17:40 ebernhardson: T444015 start cirrussearch full-cluster reindex in codfw and cloudelastic
17:36 trueg@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
17:35 trueg@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
17:28 ebernhardson: T444015 start full-cluster reindex in eqiad
17:25 trueg@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
17:23 trueg@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
17:08 trueg@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
16:45 klausman@deploy2003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
16:43 klausman@deploy2003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
16:15 jelto@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/etherpad: apply
16:10 klausman@deploy2003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
16:08 klausman@deploy2003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
16:07 otto@deploy2003: Finished scap sync-world: Backport for [Beta EventBus] Add revision_tags_change and log_tags_change dev0 streams, enable in beta (T434143) (duration: 11m 59s)
16:06 trueg@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
16:05 jelto@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/etherpad: apply
16:03 trueg@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
16:02 klausman@deploy2003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
16:01 klausman@deploy2003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
16:00 otto@deploy2003: tchin, otto: Continuing with deployment
15:59 otto@deploy2003: tchin, otto: Backport for [Beta EventBus] Add revision_tags_change and log_tags_change dev0 streams, enable in beta (T434143) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
15:55 otto@deploy2003: Started scap sync-world: Backport for [Beta EventBus] Add revision_tags_change and log_tags_change dev0 streams, enable in beta (T434143)
15:48 btullis@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-dumps: apply
15:48 btullis@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-dumps: apply
15:35 ladsgroup@deploy2003: Finished scap sync-world: Backport for Enable MPEG-DASH everywhere (T373546) (duration: 08m 48s)
15:33 btullis@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
15:32 btullis@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
15:29 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
15:29 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 267201
15:28 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 267201
15:28 dbrant@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
15:28 ladsgroup@deploy2003: ladsgroup: Backport for Enable MPEG-DASH everywhere (T373546) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
15:28 dbrant@deploy2003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
15:28 dbrant@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
15:27 dbrant@deploy2003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
15:27 dbrant@deploy2003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
15:27 klausman@deploy2003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
15:27 dbrant@deploy2003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
15:26 ladsgroup@deploy2003: Started scap sync-world: Backport for Enable MPEG-DASH everywhere (T373546)
15:23 otto@deploy2003: Finished scap sync-world: Backport for Prefetch edit_suggestions_counts with EventBus LinkedArtifacts (T443442) (duration: 08m 40s)
15:22 klausman@deploy2003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
15:17 otto@deploy2003: otto: Continuing with deployment
15:16 otto@deploy2003: otto: Backport for Prefetch edit_suggestions_counts with EventBus LinkedArtifacts (T443442) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
15:14 otto@deploy2003: Started scap sync-world: Backport for Prefetch edit_suggestions_counts with EventBus LinkedArtifacts (T443442)
15:11 otto@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
15:10 otto@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
15:10 otto@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
15:08 otto@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
15:07 otto@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
15:07 ottomata: deploying changeprop-jobqueue to pick up new queue for T443442
15:07 otto@deploy2003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
14:59 krinkle@deploy2003: Finished scap sync-world: Backport for Remove unused "TitleBlacklist-cache" channel (T85428 T441284) (duration: 11m 20s)
14:53 krinkle@deploy2003: krinkle: Continuing with deployment
14:52 krinkle@deploy2003: krinkle: Backport for Remove unused "TitleBlacklist-cache" channel (T85428 T441284) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
14:48 krinkle@deploy2003: Started scap sync-world: Backport for Remove unused "TitleBlacklist-cache" channel (T85428 T441284)
14:47 krinkle@deploy2003: sync-world aborted: Backport for RawAction: Enable full CDN caching for canonical CSS/JSON/JS URLs (T279120) (1.48.0-wmf.3) (duration: 00m 41s)
14:46 krinkle@deploy2003: Started scap sync-world: Backport for RawAction: Enable full CDN caching for canonical CSS/JSON/JS URLs (T279120) (1.48.0-wmf.3)
14:44 krinkle@deploy2003: Finished scap sync-world: Backport for RawAction: Enable full CDN caching for canonical CSS/JSON/JS URLs (T279120) (1.48.0-wmf.3), Always return bundled from getLexers, remove "fetch_lexers" shellbox call (T292663) (1.48.0-wmf.3) (duration: 13m 51s)
14:39 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1049.eqiad.wmnet
14:39 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1049.eqiad.wmnet
14:38 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker2012.codfw.wmnet
14:38 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker2012.codfw.wmnet
14:38 krinkle@deploy2003: krinkle: Continuing with deployment
14:34 krinkle@deploy2003: krinkle: Backport for RawAction: Enable full CDN caching for canonical CSS/JSON/JS URLs (T279120) (1.48.0-wmf.3), Always return bundled from getLexers, remove "fetch_lexers" shellbox call (T292663) (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
14:30 krinkle@deploy2003: Started scap sync-world: Backport for RawAction: Enable full CDN caching for canonical CSS/JSON/JS URLs (T279120) (1.48.0-wmf.3), Always return bundled from getLexers, remove "fetch_lexers" shellbox call (T292663) (1.48.0-wmf.3)
14:22 jdrewniak@deploy2003: Finished scap sync-world: Backport for Enable ReadingLists custom lists on all Wikipedias. (T441372) (duration: 18m 18s)
14:16 jdrewniak@deploy2003: jdrewniak: Continuing with deployment
14:11 jdrewniak@deploy2003: jdrewniak: Backport for Enable ReadingLists custom lists on all Wikipedias. (T441372) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
14:08 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2012.codfw.wmnet
14:07 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1049.eqiad.wmnet
14:04 jdrewniak@deploy2003: Started scap sync-world: Backport for Enable ReadingLists custom lists on all Wikipedias. (T441372)
14:01 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1049.eqiad.wmnet
14:01 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2012.codfw.wmnet
12:19 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1049.eqiad.wmnet with OS bookworm
12:17 Emperor: VACUUM 27G container database on thanos-be2008 T443888
12:11 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2012.codfw.wmnet with OS bookworm
12:08 Emperor: stop swift on thanos-be2006 to VACUUM 27G container DB T443888
12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
11:59 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1049.eqiad.wmnet with reason: host reimage
11:55 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1049.eqiad.wmnet with reason: host reimage
11:54 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2012.codfw.wmnet with reason: host reimage
11:52 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1049.eqiad.wmnet (remove config only)
11:52 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1049.eqiad.wmnet (remove config only)
11:48 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2012.codfw.wmnet with reason: host reimage
11:46 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker2012.codfw.wmnet (remove config only)
11:45 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker2012.codfw.wmnet (remove config only)
11:34 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1049.eqiad.wmnet (push config only)
11:34 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1049.eqiad.wmnet (push config only)
11:34 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1049.eqiad.wmnet with OS bookworm
11:33 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['dse-k8s-worker1049.eqiad.wmnet']
11:31 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker2012.codfw.wmnet (push config only)
11:31 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker2012.codfw.wmnet (push config only)
11:30 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2012.codfw.wmnet with OS bookworm
11:27 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from dse-k8s-wdqs-test2001 to dse-k8s-worker2012
11:26 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['dse-k8s-worker1049.eqiad.wmnet']
11:26 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2012
11:25 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2012
11:25 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2012 on all recursors
11:25 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2012 on all recursors
11:25 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
11:25 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming dse-k8s-wdqs-test2001 to dse-k8s-worker2012 - btullis@cumin1004"
11:25 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from dse-k8s-wdqs-test1001 to dse-k8s-worker1049
11:25 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming dse-k8s-wdqs-test2001 to dse-k8s-worker2012 - btullis@cumin1004"
11:24 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1049
11:24 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1049
11:24 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1049 on all recursors
11:24 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1049 on all recursors
11:24 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
11:24 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming dse-k8s-wdqs-test1001 to dse-k8s-worker1049 - btullis@cumin1004"
11:22 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming dse-k8s-wdqs-test1001 to dse-k8s-worker1049 - btullis@cumin1004"
11:21 btullis@cumin1004: START - Cookbook sre.dns.netbox
11:21 btullis@cumin1004: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
11:16 btullis@cumin1004: START - Cookbook sre.dns.netbox
11:10 btullis@cumin1004: START - Cookbook sre.dns.netbox
11:10 btullis@cumin1004: START - Cookbook sre.hosts.rename from dse-k8s-wdqs-test2001 to dse-k8s-worker2012
11:10 btullis@cumin1004: START - Cookbook sre.hosts.rename from dse-k8s-wdqs-test1001 to dse-k8s-worker1049
10:59 urbanecm@deploy2003: Finished scap sync-world: Backport for GrowthExperiments: Remove settings that nothing reads (T443648) (duration: 12m 54s)
10:58 btullis@cumin1004: conftool action : set/pooled=inactive; selector: name=dse-k8s-wdqs-test2001.codfw.wmnet
10:58 btullis@cumin1004: conftool action : set/pooled=inactive; selector: name=dse-k8s-wdqs-test1001.eqiad.wmnet
10:53 urbanecm@deploy2003: urbanecm: Continuing with deployment
10:53 urbanecm@deploy2003: urbanecm: Backport for GrowthExperiments: Remove settings that nothing reads (T443648) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
10:46 urbanecm@deploy2003: Started scap sync-world: Backport for GrowthExperiments: Remove settings that nothing reads (T443648)
10:42 Tchanders: Updated WikimediaAntiAbuse AbuseReview private code
10:41 tchanders@deploy2003: Finished scap sync-world: Backport for Take Special:AbuseReview vandalism queue out of alpha on frwiki (T443815) (duration: 15m 29s)
10:33 tchanders@deploy2003: tchanders: Continuing with deployment
10:32 tchanders@deploy2003: tchanders: Backport for Take Special:AbuseReview vandalism queue out of alpha on frwiki (T443815) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
10:25 tchanders@deploy2003: Started scap sync-world: Backport for Take Special:AbuseReview vandalism queue out of alpha on frwiki (T443815)
10:01 jgiannelos@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
10:00 jgiannelos@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
09:21 jgiannelos@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
09:21 jgiannelos@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
09:20 jgiannelos@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
09:19 jgiannelos@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
09:10 jgiannelos@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
09:10 jgiannelos@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
08:41 mlitn@deploy2003: Started scap sync-world: Backport for Enable en on bdwikimedia (T444031)
08:39 mlitn@deploy2003: Finished scap sync-world: Backport for Squashed diff to master (1.48.0-wmf.3) (duration: 36m 37s)
08:24 mlitn@deploy2003: mlitn: Continuing with deployment
08:23 mlitn@deploy2003: mlitn: Backport for Squashed diff to master (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
08:02 mlitn@deploy2003: Started scap sync-world: Backport for Squashed diff to master (1.48.0-wmf.3)
02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 08m 01s)
02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync

2026-11-08

12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 31s)
02:01 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync

2026-11-07

12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 45s)
02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
01:19 jasmine@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/sophroid: apply
01:18 jasmine@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/sophroid: apply
00:58 rzl@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/sophroid: apply
00:58 rzl@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/sophroid: apply
00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync

2026-11-06

18:30 aokoth@deploy2003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
18:29 aokoth@deploy2003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
18:29 aokoth@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
18:28 aokoth@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
17:10 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{kubestagemaster2005.codfw.wmnet} and (A:wikikube-staging-master-codfw or A:wikikube-staging-worker-codfw)
17:10 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2005.codfw.wmnet
17:10 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2005.codfw.wmnet
17:05 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2005.codfw.wmnet
17:05 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2005.codfw.wmnet
17:05 jayme@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{kubestagemaster2005.codfw.wmnet} and (A:wikikube-staging-master-codfw or A:wikikube-staging-worker-codfw)
16:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-kubelets-inplace (exit_code=0) Upgrade kubelet of staging-codfw to 1.34.13
16:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2001.codfw.wmnet
16:43 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2001.codfw.wmnet
16:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet
16:38 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet
16:37 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-kubelets-inplace Upgrade kubelet of staging-codfw to 1.34.13
16:32 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-kubelets-inplace (exit_code=0) Upgrade kubelet of staging-codfw to 1.34.13
16:32 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2005.codfw.wmnet
16:31 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2005.codfw.wmnet
16:31 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2005.codfw.wmnet
16:31 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2005.codfw.wmnet
16:31 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2004.codfw.wmnet
16:31 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2004.codfw.wmnet
16:30 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2004.codfw.wmnet
16:30 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2004.codfw.wmnet
16:30 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2003.codfw.wmnet
16:30 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2003.codfw.wmnet
16:29 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2003.codfw.wmnet
16:29 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2003.codfw.wmnet
16:29 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-kubelets-inplace Upgrade kubelet of staging-codfw to 1.34.13
16:18 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=0) Upgrade apiservers of staging-codfw to 1.34.13
16:18 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2004-2005].codfw.wmnet
16:18 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2004-2005].codfw.wmnet
16:09 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
16:09 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
16:08 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.13
16:08 jayme@cumin1004: END (ERROR) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=97) Upgrade apiservers of staging-codfw to 1.34.13
16:07 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
16:07 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
16:07 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.13
16:07 jayme@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=99) Upgrade apiservers of staging-codfw to 1.34.13
16:07 jayme@cumin1004: END (ERROR) - Cookbook sre.k8s.pool-depool-node (exit_code=97) depool for host kubestagemaster[2004-2005].codfw.wmnet
16:03 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
16:03 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.13
15:49 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet
15:41 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2013.codfw.wmnet
15:41 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker2013.codfw.wmnet
15:41 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1048.eqiad.wmnet
15:41 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1046.eqiad.wmnet
15:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1048.eqiad.wmnet
15:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1046.eqiad.wmnet
15:40 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1047.eqiad.wmnet
15:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1047.eqiad.wmnet
15:38 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet
15:31 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1048.eqiad.wmnet
15:28 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet
15:21 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1047.eqiad.wmnet
15:11 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
15:11 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
15:09 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet
15:01 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1046.eqiad.wmnet
15:00 jgiannelos@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
15:00 jgiannelos@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
14:30 jgiannelos@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
14:30 jgiannelos@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
14:07 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet with OS bookworm
13:48 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2013.codfw.wmnet with reason: host reimage
13:45 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet with OS bookworm
13:44 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2013.codfw.wmnet with reason: host reimage
13:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet with OS bookworm
13:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet (remove config only)
13:42 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker2013.codfw.wmnet (remove config only)
13:37 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet with OS bookworm
13:25 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1047.eqiad.wmnet with reason: host reimage
13:25 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet (push config only)
13:25 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker2013.codfw.wmnet (push config only)
13:25 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2013.codfw.wmnet with OS bookworm
13:22 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1048.eqiad.wmnet with reason: host reimage
13:18 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1046.eqiad.wmnet with reason: host reimage
13:18 moritzm: installing linux 6.12.115 on Trixie hosts
13:15 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1048.eqiad.wmnet with reason: host reimage
13:15 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1047.eqiad.wmnet with reason: host reimage
13:14 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1046.eqiad.wmnet with reason: host reimage
13:13 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet (remove config only)
13:13 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1048.eqiad.wmnet (remove config only)
13:12 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet (remove config only)
13:12 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1047.eqiad.wmnet (remove config only)
13:12 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet (remove config only)
13:12 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1046.eqiad.wmnet (remove config only)
13:08 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
13:07 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
13:03 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dse-k8s-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
13:02 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
13:02 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
13:00 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
13:00 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
12:55 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet (push config only)
12:55 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1048.eqiad.wmnet (push config only)
12:55 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet (push config only)
12:55 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1047.eqiad.wmnet (push config only)
12:54 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1048.eqiad.wmnet with OS bookworm
12:54 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1047.eqiad.wmnet with OS bookworm
12:54 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet (push config only)
12:54 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1046.eqiad.wmnet (push config only)
12:54 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1046.eqiad.wmnet with OS bookworm
12:54 btullis@cumin1004: START - Cookbook sre.hosts.provision for host dse-k8s-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
12:47 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs1031 to dse-k8s-worker1047
12:46 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1047
12:46 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs2025 to dse-k8s-worker2013
12:45 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2013
12:44 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2013
12:44 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2013 on all recursors
12:44 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2013 on all recursors
12:44 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
12:42 btullis@cumin1004: START - Cookbook sre.dns.netbox
12:41 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1047
12:41 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1047 on all recursors
12:41 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1047 on all recursors
12:41 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
12:40 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs1032 to dse-k8s-worker1048
12:40 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1048
12:39 btullis@cumin1004: START - Cookbook sre.dns.netbox
12:38 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1048
12:38 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1048 on all recursors
12:38 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1048 on all recursors
12:38 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
12:38 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1032 to dse-k8s-worker1048 - btullis@cumin1004"
12:38 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1032 to dse-k8s-worker1048 - btullis@cumin1004"
12:34 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs1030 to dse-k8s-worker1046
12:34 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1046
12:33 btullis@cumin1004: START - Cookbook sre.dns.netbox
12:33 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1046
12:33 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1046 on all recursors
12:33 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1046 on all recursors
12:33 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
12:33 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1030 to dse-k8s-worker1046 - btullis@cumin1004"
12:33 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1030 to dse-k8s-worker1046 - btullis@cumin1004"
12:31 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs2025 to dse-k8s-worker2013
12:30 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs1032 to dse-k8s-worker1048
12:29 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs1031 to dse-k8s-worker1047
12:29 btullis@cumin1004: START - Cookbook sre.dns.netbox
12:29 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
12:29 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs1030 to dse-k8s-worker1046
12:12 moritzm: imported openjdk-21 21.0.13+9-1~deb13u2~wmf12u1 to component/jdk21 for bookworm-wikimedia
12:03 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
12:03 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
12:02 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
11:58 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs1032.eqiad.wmnet
11:58 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1032.eqiad.wmnet
11:52 cgoubert@deploy2003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'.
11:52 cgoubert@deploy2003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'.
11:51 cgoubert@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'.
11:49 cgoubert@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'.
11:48 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs1032.eqiad.wmnet
11:47 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs1032.eqiad.wmnet
11:43 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs2025.codfw.wmnet
11:43 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs2025.codfw.wmnet
11:42 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs1031.eqiad.wmnet
11:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1031.eqiad.wmnet
11:39 fabfur: installing haproxykafka 0.4.0 on cp7001 for testing (T443632) (T443640)
11:36 a-pizzata@deploy2003: Finished deploy [analytics/refinery@f4aa212]: Regular analytics weekly train [analytics/refinery@f4aa2127] (duration: 04m 50s)
11:32 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs2025.codfw.wmnet
11:32 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs1031.eqiad.wmnet
11:31 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs2025.codfw.wmnet
11:31 a-pizzata@deploy2003: Started deploy [analytics/refinery@f4aa212]: Regular analytics weekly train [analytics/refinery@f4aa2127]
11:30 a-pizzata@deploy2003: Finished deploy [analytics/refinery@f4aa212] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@f4aa2127] (duration: 00m 46s)
11:30 a-pizzata@deploy2003: Started deploy [analytics/refinery@f4aa212] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@f4aa2127]
11:19 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs1031.eqiad.wmnet
11:18 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs1030.eqiad.wmnet
11:18 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1030.eqiad.wmnet
11:09 btullis@cumin1004: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts wdqs2025.codfw.wmnet
11:09 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs2025.codfw.wmnet
11:08 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs1030.eqiad.wmnet
11:00 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs1030.eqiad.wmnet
10:49 jiji@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) depool kartotherian in eqiad: maintenance
10:44 jiji@cumin1004: START - Cookbook sre.discovery.service-route depool kartotherian in eqiad: maintenance
10:42 effie: depool kartotherian from eqiad (ie maps) - T443438
10:34 jelto@deploy2003: Finished scap deploy-service miscweb-aux: Deploy to fix T443357 (duration: 03m 39s)
10:33 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-codfw Deploying miscweb
10:31 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-eqiad Deploying miscweb
10:31 jelto@deploy2003: Started scap deploy-service miscweb-aux: Deploy to fix T443357
10:29 marostegui: Limit icinga db user connections to 10 in ms* and pc* T442995
10:16 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
10:16 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
10:16 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
10:11 fabfur@cumin1004: conftool action : set/pooled=yes; selector: dc=magru,cluster=dnsbox
10:08 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
09:04 mvernon@cumin1004: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe
08:57 jelto@deploy2003: Finished scap deploy-service miscweb-aux: Deploy to fix T443826 (duration: 04m 11s)
08:56 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-codfw Deploying miscweb
08:53 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-eqiad Deploying miscweb
08:53 jelto@deploy2003: Started scap deploy-service miscweb-aux: Deploy to fix T443826
08:52 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe
08:48 mvernon@cumin1004: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe
08:47 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe
06:52 XioNoX: push pfw policies - T443852
00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync

2026-11-05

23:30 mutante: codesearch2001 initial repo pull from github. codesearch1001 temp puppet disabled for careful rollout.
23:18 maryum: Deployed security fix for T443424, T443467, T443036
23:10 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2198.codfw.wmnet with reason: Maintenance
23:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195 (T441633)', diff saved to https://phabricator.wikimedia.org/P97417 and previous config saved to /var/cache/conftool/dbconfig/20261105-231002-ladsgroup.json
23:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P97416 and previous config saved to /var/cache/conftool/dbconfig/20261105-230000-ladsgroup.json
22:51 sbassett: Deployed security fixes for T439878, T440323, T440657, T440656, T440655
22:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P97415 and previous config saved to /var/cache/conftool/dbconfig/20261105-224958-ladsgroup.json
22:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195 (T441633)', diff saved to https://phabricator.wikimedia.org/P97414 and previous config saved to /var/cache/conftool/dbconfig/20261105-223956-ladsgroup.json
22:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2195 (T441633)', diff saved to https://phabricator.wikimedia.org/P97413 and previous config saved to /var/cache/conftool/dbconfig/20261105-223248-ladsgroup.json
22:32 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2195.codfw.wmnet with reason: Maintenance
22:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181 (T441633)', diff saved to https://phabricator.wikimedia.org/P97412 and previous config saved to /var/cache/conftool/dbconfig/20261105-223225-ladsgroup.json
22:31 egardner@deploy2003: Finished scap sync-world: Backport for ShareCard: Log client errors with the topic "error.sharecard" (T439163) (1.48.0-wmf.3), ShareCard: Show animated GIFs as a still image (T442968) (1.48.0-wmf.3) (duration: 11m 46s)
22:24 egardner@deploy2003: egardner: Continuing with deployment
22:23 egardner@deploy2003: egardner: Backport for ShareCard: Log client errors with the topic "error.sharecard" (T439163) (1.48.0-wmf.3), ShareCard: Show animated GIFs as a still image (T442968) (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
22:22 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
22:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P97411 and previous config saved to /var/cache/conftool/dbconfig/20261105-222223-ladsgroup.json
22:22 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
22:19 egardner@deploy2003: Started scap sync-world: Backport for ShareCard: Log client errors with the topic "error.sharecard" (T439163) (1.48.0-wmf.3), ShareCard: Show animated GIFs as a still image (T442968) (1.48.0-wmf.3)
22:15 egardner@deploy2003: Finished scap sync-world: Backport for Add Creative Commons license icons for the share card (T439162) (duration: 14m 19s)
22:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P97410 and previous config saved to /var/cache/conftool/dbconfig/20261105-221221-ladsgroup.json
22:08 egardner@deploy2003: egardner: Continuing with deployment
22:07 egardner@deploy2003: egardner: Backport for Add Creative Commons license icons for the share card (T439162) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
22:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181 (T441633)', diff saved to https://phabricator.wikimedia.org/P97409 and previous config saved to /var/cache/conftool/dbconfig/20261105-220220-ladsgroup.json
22:01 egardner@deploy2003: Started scap sync-world: Backport for Add Creative Commons license icons for the share card (T439162)
21:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2181 (T441633)', diff saved to https://phabricator.wikimedia.org/P97408 and previous config saved to /var/cache/conftool/dbconfig/20261105-215453-ladsgroup.json
21:54 mutante: ci / contint1003/contint2003 - if "sudo docker info -f 'Template:.Driver'" says "overlay2" it's the classic way. after restart when it says "overlayfs2" that will be the NEW way! (T440668)
21:54 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2181.codfw.wmnet with reason: Maintenance
21:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167 (T441633)', diff saved to https://phabricator.wikimedia.org/P97407 and previous config saved to /var/cache/conftool/dbconfig/20261105-215432-ladsgroup.json
21:51 mutante: ci / contint1003/contint2003 - docker will be reconfigured to use containerd-snapshotter rather than dockers legacy store, overlay2 (a thing docker has been moving towards for a while). images pulled under layer2 become invisible on next docker restart. releng will handle that and prune the old images. new images will have to be pulled down. needed for ST6.1.4 (T440668)
21:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P97406 and previous config saved to /var/cache/conftool/dbconfig/20261105-214430-ladsgroup.json
21:34 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
21:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P97405 and previous config saved to /var/cache/conftool/dbconfig/20261105-213428-ladsgroup.json
21:33 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
21:30 topranks: re-enable BGP for pybal on lvs1019 to fail back from lvs1020 after vlan add T440859
21:28 kemayo@deploy2003: mwscript-k8s job started: namespaceDupes zh_min_nanwiki --fix # T430731
21:26 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs1019.eqiad.wmnet
21:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167 (T441633)', diff saved to https://phabricator.wikimedia.org/P97403 and previous config saved to /var/cache/conftool/dbconfig/20261105-212426-ladsgroup.json
21:24 kemayo@deploy2003: Finished scap sync-world: Backport for VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki (T442767), Add new namespace for zh_min_nanwiki (T430731) (duration: 16m 22s)
21:19 cmooney@cumin1004: START - Cookbook sre.hosts.reboot-single for host lvs1019.eqiad.wmnet
21:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2167 (T441633)', diff saved to https://phabricator.wikimedia.org/P97402 and previous config saved to /var/cache/conftool/dbconfig/20261105-211708-ladsgroup.json
21:17 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2167.codfw.wmnet with reason: Maintenance
21:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166 (T441633)', diff saved to https://phabricator.wikimedia.org/P97401 and previous config saved to /var/cache/conftool/dbconfig/20261105-211647-ladsgroup.json
21:15 kemayo@deploy2003: pppery, kemayo: Continuing with deployment
21:14 kemayo@deploy2003: pppery, kemayo: Backport for VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki (T442767), Add new namespace for zh_min_nanwiki (T430731) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
21:08 kemayo@deploy2003: Started scap sync-world: Backport for VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki (T442767), Add new namespace for zh_min_nanwiki (T430731)
21:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P97400 and previous config saved to /var/cache/conftool/dbconfig/20261105-210645-ladsgroup.json
20:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P97399 and previous config saved to /var/cache/conftool/dbconfig/20261105-205643-ladsgroup.json
20:56 James_F: mwscript sql.php --wiki=wikifunctionswiki --cluster extension1 extensions/WikiLambda/sql/mysql/table-abstractwiki_integrations.sql # T443295
20:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166 (T441633)', diff saved to https://phabricator.wikimedia.org/P97398 and previous config saved to /var/cache/conftool/dbconfig/20261105-204641-ladsgroup.json
20:44 topranks: disable pybal service on lvs1019 to failover to lvs1020 ahead of adding new vlan ints to the former T440859
20:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2166 (T441633)', diff saved to https://phabricator.wikimedia.org/P97397 and previous config saved to /var/cache/conftool/dbconfig/20261105-203938-ladsgroup.json
20:39 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2166.codfw.wmnet with reason: Maintenance
20:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165 (T441633)', diff saved to https://phabricator.wikimedia.org/P97396 and previous config saved to /var/cache/conftool/dbconfig/20261105-203917-ladsgroup.json
20:39 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs1019.eqiad.wmnet with reason: failover lvs1019 to lvs1020 to add new vlan sub-ints
20:29 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P97395 and previous config saved to /var/cache/conftool/dbconfig/20261105-202915-ladsgroup.json
20:19 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P97394 and previous config saved to /var/cache/conftool/dbconfig/20261105-201913-ladsgroup.json
20:09 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165 (T441633)', diff saved to https://phabricator.wikimedia.org/P97393 and previous config saved to /var/cache/conftool/dbconfig/20261105-200911-ladsgroup.json
20:06 bking@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
20:05 bking@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
20:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2165 (T441633)', diff saved to https://phabricator.wikimedia.org/P97392 and previous config saved to /var/cache/conftool/dbconfig/20261105-200211-ladsgroup.json
20:02 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2165.codfw.wmnet with reason: Maintenance
20:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164 (T441633)', diff saved to https://phabricator.wikimedia.org/P97391 and previous config saved to /var/cache/conftool/dbconfig/20261105-200150-ladsgroup.json
19:56 bking@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
19:55 bking@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
19:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P97390 and previous config saved to /var/cache/conftool/dbconfig/20261105-195148-ladsgroup.json
19:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P97389 and previous config saved to /var/cache/conftool/dbconfig/20261105-194146-ladsgroup.json
19:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164 (T441633)', diff saved to https://phabricator.wikimedia.org/P97388 and previous config saved to /var/cache/conftool/dbconfig/20261105-193144-ladsgroup.json
19:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2164 (T441633)', diff saved to https://phabricator.wikimedia.org/P97387 and previous config saved to /var/cache/conftool/dbconfig/20261105-192426-ladsgroup.json
19:24 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2164.codfw.wmnet with reason: Maintenance
19:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163 (T441633)', diff saved to https://phabricator.wikimedia.org/P97386 and previous config saved to /var/cache/conftool/dbconfig/20261105-192416-ladsgroup.json
19:14 bd808@deploy2003: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply
19:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P97385 and previous config saved to /var/cache/conftool/dbconfig/20261105-191414-ladsgroup.json
19:13 bd808@deploy2003: helmfile [eqiad] START helmfile.d/services/toolhub: apply
19:12 bd808@deploy2003: helmfile [codfw] DONE helmfile.d/services/toolhub: apply
19:12 bd808@deploy2003: helmfile [codfw] START helmfile.d/services/toolhub: apply
19:11 bd808@deploy2003: helmfile [staging] DONE helmfile.d/services/toolhub: apply
19:10 bd808@deploy2003: helmfile [staging] START helmfile.d/services/toolhub: apply
19:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P97384 and previous config saved to /var/cache/conftool/dbconfig/20261105-190412-ladsgroup.json
18:57 dancy@deploy2003: Finished scap sync-world: Testing T411063 (duration: 44m 09s)
18:54 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-b3-eqiad
18:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163 (T441633)', diff saved to https://phabricator.wikimedia.org/P97383 and previous config saved to /var/cache/conftool/dbconfig/20261105-185411-ladsgroup.json
18:54 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-b3-eqiad
18:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2163 (T441633)', diff saved to https://phabricator.wikimedia.org/P97382 and previous config saved to /var/cache/conftool/dbconfig/20261105-184550-ladsgroup.json
18:45 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2163.codfw.wmnet with reason: Maintenance
18:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154 (T441633)', diff saved to https://phabricator.wikimedia.org/P97381 and previous config saved to /var/cache/conftool/dbconfig/20261105-184529-ladsgroup.json
18:42 dancy@deploy2003: dancy: Continuing with deployment
18:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P97380 and previous config saved to /var/cache/conftool/dbconfig/20261105-183527-ladsgroup.json
18:35 dancy@deploy2003: dancy: Testing T411063 synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
18:32 bd808@deploy2003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
18:32 bd808@deploy2003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
18:32 bd808@deploy2003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
18:31 bd808@deploy2003: helmfile [codfw] START helmfile.d/services/developer-portal: apply
18:31 bd808@deploy2003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
18:31 bd808@deploy2003: helmfile [staging] START helmfile.d/services/developer-portal: apply
18:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P97379 and previous config saved to /var/cache/conftool/dbconfig/20261105-182526-ladsgroup.json
18:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154 (T441633)', diff saved to https://phabricator.wikimedia.org/P97378 and previous config saved to /var/cache/conftool/dbconfig/20261105-181524-ladsgroup.json
18:13 dancy@deploy2003: Started scap sync-world: Testing T411063
18:08 ladsgroup@deploy2003: Finished scap sync-world: Push small changes for error pages (duration: 05m 08s)
18:08 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2154 (T441633)', diff saved to https://phabricator.wikimedia.org/P97377 and previous config saved to /var/cache/conftool/dbconfig/20261105-180812-ladsgroup.json
18:08 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2154.codfw.wmnet with reason: Maintenance
18:06 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
18:04 ladsgroup@deploy2003: ladsgroup: Push small changes for error pages synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
18:03 ladsgroup@deploy2003: Started scap sync-world: Push small changes for error pages
17:59 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on lsw1-b3-eqiad IPv6,lsw1-b3-eqiad.mgmt with reason: need to test things on switch
17:53 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
17:52 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
17:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
17:51 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
17:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
17:45 jayme@cumin2003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts parsoidtest1001.eqiad.wmnet
17:45 jayme@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
17:45 jayme@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: parsoidtest1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin2003"
17:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
17:45 jayme@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: parsoidtest1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin2003"
17:43 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
17:43 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'.
17:40 hnowlan@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
17:40 jayme@cumin2003: START - Cookbook sre.dns.netbox
17:40 hnowlan@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
17:39 topranks: reboot lsw1-b3-eqiad to downgrade to SR-Linux 26.3.3 T443687
17:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
17:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
17:34 jayme@cumin2003: START - Cookbook sre.hosts.decommission for hosts parsoidtest1001.eqiad.wmnet
17:33 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on ssw1-a[1,8]-eqiad with reason: need to test things on switch
17:32 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on lsw1-b3-eqiad with reason: need to test things on switch
17:31 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on sretest1006.eqiad.wmnet with reason: need to test things on switch
17:04 fabfur: installing haproxykafka 0.4.0 on cp7002 for testing (T443632) (T443640)
16:48 sukhe@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for role: aux_k8s::worker@eqiad
16:47 sukhe@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for role: aux_k8s::worker@eqiad
16:44 kamila@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
16:43 kamila@deploy2003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
16:42 kamila@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
16:41 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
16:40 moritzm: installing krb5 security updates
16:40 kamila@deploy2003: Finished scap sync-world: switch mw-parsoid to php 8.5 - T439955 (duration: 03m 03s)
16:37 kamila@deploy2003: Started scap sync-world: switch mw-parsoid to php 8.5 - T439955
16:35 kamila@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
16:35 moritzm: installing krb5 security updates
16:34 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
15:28 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool magru [reason: done testing DNS admin cookbook changes, no task ID specified]
15:28 sukhe@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool magru [reason: done testing DNS admin cookbook changes, no task ID specified]
15:27 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool magru [reason: testing DNS admin cookbook changes, no task ID specified]
15:27 sukhe@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool magru [reason: testing DNS admin cookbook changes, no task ID specified]
15:14 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P{lvs2013.codfw.wmnet} and A:lvs
15:13 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P{lvs2013.codfw.wmnet} and A:lvs
15:13 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P{lvs2014.codfw.wmnet} and A:lvs
15:12 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P{lvs2014.codfw.wmnet} and A:lvs
15:10 stran@deploy2003: Finished scap sync-world: Backport for Round 2 on disabling createpage for anonymous users on English Wikibooks (T443449), Enable Special:SuggestedInvestigations for nlwiki (T443646) (duration: 12m 28s)
15:08 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
15:04 stran@deploy2003: stran, codenamenoreste: Continuing with deployment
15:02 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
15:02 stran@deploy2003: stran, codenamenoreste: Backport for Round 2 on disabling createpage for anonymous users on English Wikibooks (T443449), Enable Special:SuggestedInvestigations for nlwiki (T443646) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
15:02 jhancock@cumin2003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
15:02 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
14:58 stran@deploy2003: Started scap sync-world: Backport for Round 2 on disabling createpage for anonymous users on English Wikibooks (T443449), Enable Special:SuggestedInvestigations for nlwiki (T443646)
14:56 akhatun@deploy2003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply
14:56 akhatun@deploy2003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply
14:48 dbrant@deploy2003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
14:48 dbrant@deploy2003: helmfile [codfw] START helmfile.d/services/mobileapps: apply
14:47 dbrant@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
14:47 mlitn@deploy2003: Finished scap sync-world: Backport for Squashed diff to master (1.48.0-wmf.3) (duration: 12m 38s)
14:47 dbrant@deploy2003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
14:46 akhatun@deploy2003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply
14:46 akhatun@deploy2003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply
14:44 dbrant@deploy2003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
14:44 dbrant@deploy2003: helmfile [staging] START helmfile.d/services/mobileapps: apply
14:41 mlitn@deploy2003: mlitn: Continuing with deployment
14:38 mlitn@deploy2003: mlitn: Backport for Squashed diff to master (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
14:34 mlitn@deploy2003: Started scap sync-world: Backport for Squashed diff to master (1.48.0-wmf.3)
14:32 Tran: Deployed patch for T443598
14:17 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
14:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286 (T441633)', diff saved to https://phabricator.wikimedia.org/P97374 and previous config saved to /var/cache/conftool/dbconfig/20261105-141655-ladsgroup.json
14:15 stran@deploy2003: Finished scap sync-world: Backport for Enable SuggestedInvestigations for nlwiki (hidden), enwikivoyage (T443646 T443147) (duration: 11m 34s)
14:11 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1006.eqiad.wmnet with OS trixie
14:09 stran@deploy2003: stran: Continuing with deployment
14:08 stran@deploy2003: stran: Backport for Enable SuggestedInvestigations for nlwiki (hidden), enwikivoyage (T443646 T443147) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
14:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286', diff saved to https://phabricator.wikimedia.org/P97373 and previous config saved to /var/cache/conftool/dbconfig/20261105-140653-ladsgroup.json
14:03 stran@deploy2003: Started scap sync-world: Backport for Enable SuggestedInvestigations for nlwiki (hidden), enwikivoyage (T443646 T443147)
13:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286', diff saved to https://phabricator.wikimedia.org/P97372 and previous config saved to /var/cache/conftool/dbconfig/20261105-135651-ladsgroup.json
13:53 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
13:48 cmooney@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
13:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286 (T441633)', diff saved to https://phabricator.wikimedia.org/P97371 and previous config saved to /var/cache/conftool/dbconfig/20261105-134649-ladsgroup.json
13:45 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
13:45 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1286 (T441633)', diff saved to https://phabricator.wikimedia.org/P97370 and previous config saved to /var/cache/conftool/dbconfig/20261105-134145-ladsgroup.json
13:41 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1286.eqiad.wmnet with reason: Maintenance
13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281 (T441633)', diff saved to https://phabricator.wikimedia.org/P97369 and previous config saved to /var/cache/conftool/dbconfig/20261105-134125-ladsgroup.json
13:35 eevans@deploy2003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply
13:35 eevans@deploy2003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply
13:35 eevans@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply
13:35 eevans@deploy2003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply
13:35 eevans@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
13:34 eevans@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
13:33 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
13:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281', diff saved to https://phabricator.wikimedia.org/P97368 and previous config saved to /var/cache/conftool/dbconfig/20261105-133123-ladsgroup.json
13:26 cmooney@cumin1004: START - Cookbook sre.hosts.provision for host sretest1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
13:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281', diff saved to https://phabricator.wikimedia.org/P97367 and previous config saved to /var/cache/conftool/dbconfig/20261105-132122-ladsgroup.json
13:20 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) sretest1006.eqiad.wmnet on all recursors
13:19 cmooney@cumin1004: START - Cookbook sre.dns.wipe-cache sretest1006.eqiad.wmnet on all recursors
13:19 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
13:19 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
13:19 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
13:19 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004"
13:19 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004"
13:18 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
13:14 cmooney@cumin1004: START - Cookbook sre.dns.netbox
13:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281 (T441633)', diff saved to https://phabricator.wikimedia.org/P97366 and previous config saved to /var/cache/conftool/dbconfig/20261105-131120-ladsgroup.json
13:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1281 (T441633)', diff saved to https://phabricator.wikimedia.org/P97365 and previous config saved to /var/cache/conftool/dbconfig/20261105-130607-ladsgroup.json
13:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 7 hosts with reason: Maintenance
13:05 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1281.eqiad.wmnet with reason: Maintenance
13:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226 (T441633)', diff saved to https://phabricator.wikimedia.org/P97364 and previous config saved to /var/cache/conftool/dbconfig/20261105-130524-ladsgroup.json
12:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P97363 and previous config saved to /var/cache/conftool/dbconfig/20261105-125522-ladsgroup.json
12:52 kamila@deploy2003: Finished scap sync-world: Backport for Enroll 5% of client sessions in PHP 8.5 (T441689) (duration: 10m 50s)
12:46 kamila@deploy2003: kamila: Continuing with deployment
12:46 kamila@deploy2003: kamila: Backport for Enroll 5% of client sessions in PHP 8.5 (T441689) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
12:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P97362 and previous config saved to /var/cache/conftool/dbconfig/20261105-124521-ladsgroup.json
12:41 kamila@deploy2003: Started scap sync-world: Backport for Enroll 5% of client sessions in PHP 8.5 (T441689)
12:37 moritzm: installing alsa-lib security updates
12:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226 (T441633)', diff saved to https://phabricator.wikimedia.org/P97361 and previous config saved to /var/cache/conftool/dbconfig/20261105-123519-ladsgroup.json
12:31 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1006.eqiad.wmnet with OS trixie
12:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1226 (T441633)', diff saved to https://phabricator.wikimedia.org/P97360 and previous config saved to /var/cache/conftool/dbconfig/20261105-122614-ladsgroup.json
12:26 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1226.eqiad.wmnet with reason: Maintenance
12:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214 (T441633)', diff saved to https://phabricator.wikimedia.org/P97359 and previous config saved to /var/cache/conftool/dbconfig/20261105-122554-ladsgroup.json
12:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P97358 and previous config saved to /var/cache/conftool/dbconfig/20261105-121552-ladsgroup.json
12:13 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
12:11 moritzm: installing alsa-lib security updates
12:09 cmooney@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
12:07 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
12:07 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
12:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P97357 and previous config saved to /var/cache/conftool/dbconfig/20261105-120551-ladsgroup.json
12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
11:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214 (T441633)', diff saved to https://phabricator.wikimedia.org/P97356 and previous config saved to /var/cache/conftool/dbconfig/20261105-115549-ladsgroup.json
11:52 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
11:52 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
11:51 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
11:49 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) sretest1006.eqiad.wmnet on all recursors
11:49 cmooney@cumin1004: START - Cookbook sre.dns.wipe-cache sretest1006.eqiad.wmnet on all recursors
11:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1214 (T441633)', diff saved to https://phabricator.wikimedia.org/P97355 and previous config saved to /var/cache/conftool/dbconfig/20261105-114845-ladsgroup.json
11:48 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1214.eqiad.wmnet with reason: Maintenance
11:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209 (T441633)', diff saved to https://phabricator.wikimedia.org/P97354 and previous config saved to /var/cache/conftool/dbconfig/20261105-114825-ladsgroup.json
11:47 cmooney@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest1006.eqiad.wmnet with OS trixie
11:47 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
11:47 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
11:47 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004"
11:47 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
11:47 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004"
11:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P97353 and previous config saved to /var/cache/conftool/dbconfig/20261105-113823-ladsgroup.json
11:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P97352 and previous config saved to /var/cache/conftool/dbconfig/20261105-112821-ladsgroup.json
11:27 btullis@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-test-eqiad
11:22 btullis@cumin1004: END (ERROR) - Cookbook sre.hadoop.reboot-workers (exit_code=97) for Hadoop analytics cluster
11:20 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1262: Maintenance
11:18 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209 (T441633)', diff saved to https://phabricator.wikimedia.org/P97350 and previous config saved to /var/cache/conftool/dbconfig/20261105-111819-ladsgroup.json
11:17 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
11:16 cmooney@cumin1004: START - Cookbook sre.dns.netbox
11:16 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
11:15 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
11:14 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
11:13 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
11:12 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
11:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1209 (T441633)', diff saved to https://phabricator.wikimedia.org/P97349 and previous config saved to /var/cache/conftool/dbconfig/20261105-111113-ladsgroup.json
11:11 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1209.eqiad.wmnet with reason: Maintenance
11:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192 (T441633)', diff saved to https://phabricator.wikimedia.org/P97348 and previous config saved to /var/cache/conftool/dbconfig/20261105-111053-ladsgroup.json
11:09 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2250.codfw.wmnet with OS trixie
11:05 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
11:04 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
11:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-master-eqiad
11:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl1002.eqiad.wmnet
11:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl1002.eqiad.wmnet
11:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P97346 and previous config saved to /var/cache/conftool/dbconfig/20261105-110051-ladsgroup.json
10:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl1002.eqiad.wmnet
10:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl1002.eqiad.wmnet
10:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl1001.eqiad.wmnet
10:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl1001.eqiad.wmnet
10:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl1001.eqiad.wmnet
10:51 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl1001.eqiad.wmnet
10:51 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-master-eqiad
10:51 btullis@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-test-eqiad
10:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P97344 and previous config saved to /var/cache/conftool/dbconfig/20261105-105049-ladsgroup.json
10:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-master-codfw
10:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl2002.codfw.wmnet
10:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl2002.codfw.wmnet
10:43 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl2002.codfw.wmnet
10:43 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl2002.codfw.wmnet
10:43 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl2001.codfw.wmnet
10:43 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl2001.codfw.wmnet
10:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192 (T441633)', diff saved to https://phabricator.wikimedia.org/P97342 and previous config saved to /var/cache/conftool/dbconfig/20261105-104047-ladsgroup.json
10:38 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl2001.codfw.wmnet
10:38 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl2001.codfw.wmnet
10:38 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2250.codfw.wmnet with reason: host reimage
10:38 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-master-codfw
10:35 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1262: Maintenance
10:34 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on db2250.codfw.wmnet with reason: host reimage
10:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1192 (T441633)', diff saved to https://phabricator.wikimedia.org/P97340 and previous config saved to /var/cache/conftool/dbconfig/20261105-103234-ladsgroup.json
10:32 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1192.eqiad.wmnet with reason: Maintenance
10:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167 (T441633)', diff saved to https://phabricator.wikimedia.org/P97339 and previous config saved to /var/cache/conftool/dbconfig/20261105-103215-ladsgroup.json
10:32 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host db2250.codfw.wmnet (remove config only)
10:32 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host db2250.codfw.wmnet (remove config only)
10:32 moritzm: installing libgit2 security updates
10:27 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
10:27 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
10:26 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
10:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P97338 and previous config saved to /var/cache/conftool/dbconfig/20261105-102213-ladsgroup.json
10:21 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host db2250.codfw.wmnet (push config only)
10:21 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host db2250.codfw.wmnet (push config only)
10:20 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host db2250.codfw.wmnet with OS trixie
10:19 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2250.codfw.wmnet with reason: Reimage
10:16 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on sretest1006.eqiad.wmnet with reason: test on other vlan
10:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P97337 and previous config saved to /var/cache/conftool/dbconfig/20261105-101211-ladsgroup.json
10:06 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica1006.wikimedia.org
10:04 cwilliams@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1262: Maintenance
10:04 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1262: Maintenance
10:02 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica1006.wikimedia.org
10:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167 (T441633)', diff saved to https://phabricator.wikimedia.org/P97336 and previous config saved to /var/cache/conftool/dbconfig/20261105-100209-ladsgroup.json
09:57 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica1005.wikimedia.org
09:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1167 (T441633)', diff saved to https://phabricator.wikimedia.org/P97335 and previous config saved to /var/cache/conftool/dbconfig/20261105-095350-ladsgroup.json
09:53 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1167.eqiad.wmnet with reason: Maintenance
09:53 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica1005.wikimedia.org
09:49 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica2008.wikimedia.org
09:45 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica2008.wikimedia.org
09:36 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user jynus
09:36 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user jynus
09:36 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user jmoore111
09:35 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user jmoore111
09:35 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user andreawest
09:35 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user andreawest
09:31 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user suecarmol
09:31 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user suecarmol
09:24 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user hmonroy
09:23 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user hmonroy
09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user harrayo-wmf
09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user harrayo-wmf
09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user atieno
09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user atieno
09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user ksiebert
09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user ksiebert
09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user dmaza
09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user dmaza
09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user hmonroy
09:16 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user hmonroy
09:16 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user kgraessle
09:16 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user kgraessle
09:12 jnuche@deploy2003: rebuilt and synchronized wikiversions files: group2 to 1.48.0-wmf.3 refs T442221
09:09 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user nettrom
09:08 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user nettrom
08:44 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
08:42 moritzm: restarting FPM on phab1005 to pick up Expat security updates
08:42 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
08:22 dcausse@deploy2003: Finished scap sync-world: Backport for Restore the semantic highlighting batch ext (1.48.0-wmf.3) (duration: 16m 04s)
08:14 dcausse@deploy2003: dcausse: Continuing with deployment
08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.decommission (exit_code=0)
08:11 marostegui@cumin1004: Removing clouddb1014 from zarcillo T441395
08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts clouddb1014.eqiad.wmnet
08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: clouddb1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004"
08:11 marostegui@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: clouddb1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004"
08:10 dcausse@deploy2003: dcausse: Backport for Restore the semantic highlighting batch ext (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
08:07 marostegui@cumin1004: START - Cookbook sre.dns.netbox
08:06 dcausse@deploy2003: Started scap sync-world: Backport for Restore the semantic highlighting batch ext (1.48.0-wmf.3)
08:02 marostegui@cumin1004: START - Cookbook sre.hosts.decommission for hosts clouddb1014.eqiad.wmnet
08:01 marostegui@cumin1004: START - Cookbook sre.mysql.decommission
07:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org
07:56 moritzm: installing expat security updates
07:53 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org
07:37 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast2003.wikimedia.org
07:30 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast2003.wikimedia.org
07:26 marostegui: Drop thread_reaction tables T442289
07:23 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2007.codfw.wmnet with OS trixie
07:07 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2007.codfw.wmnet with reason: host reimage
07:03 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2007.codfw.wmnet with reason: host reimage
07:01 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2007.codfw.wmnet (remove config only)
07:01 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2007.codfw.wmnet (remove config only)
06:49 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2007.codfw.wmnet (push config only)
06:49 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2007.codfw.wmnet (push config only)
06:48 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2007.codfw.wmnet with OS trixie
05:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-worker-eqiad
05:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1044.eqiad.wmnet
05:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1044.eqiad.wmnet
05:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1044.eqiad.wmnet
05:21 denisse@cumin1004: DONE (FAIL) - Cookbook sre.idm.logout (exit_code=99) Logging Cfoster out of all services on: 4 hosts
05:18 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1044.eqiad.wmnet
05:18 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1043.eqiad.wmnet
05:18 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1043.eqiad.wmnet
05:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1043.eqiad.wmnet
05:08 denisse@cumin1004: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Cfoster out of all services on: 4 hosts
04:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1043.eqiad.wmnet
04:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1042.eqiad.wmnet
04:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1042.eqiad.wmnet
04:33 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2020.codfw.wmnet, repooling both afterwards
04:33 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1042.eqiad.wmnet
04:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1042.eqiad.wmnet
04:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1041.eqiad.wmnet
04:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1041.eqiad.wmnet
04:27 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1026.eqiad.wmnet, repooling both afterwards
04:26 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer (T441682, copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2020.codfw.wmnet, repooling both afterwards
04:26 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2019.codfw.wmnet, repooling both afterwards
04:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1041.eqiad.wmnet
04:19 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer (T441682, copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1026.eqiad.wmnet, repooling both afterwards
04:19 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1025.eqiad.wmnet, repooling both afterwards
04:18 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer (T441682, copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2019.codfw.wmnet, repooling both afterwards
04:18 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2018.codfw.wmnet, repooling both afterwards
04:11 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer (T441682, copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1025.eqiad.wmnet, repooling both afterwards
04:11 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer (T441682, copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2018.codfw.wmnet, repooling both afterwards
03:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1041.eqiad.wmnet
03:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1040.eqiad.wmnet
03:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1040.eqiad.wmnet
03:46 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1040.eqiad.wmnet
03:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1040.eqiad.wmnet
03:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1039.eqiad.wmnet
03:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1039.eqiad.wmnet
03:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1039.eqiad.wmnet
03:10 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1039.eqiad.wmnet
03:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1028.eqiad.wmnet
03:10 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1028.eqiad.wmnet
03:02 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1028.eqiad.wmnet
02:35 dani@deploy2003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
02:35 dani@deploy2003: helmfile [codfw] START helmfile.d/services/miscweb: apply
02:35 dani@deploy2003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
02:35 dani@deploy2003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
02:34 dani@deploy2003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
02:34 dani@deploy2003: helmfile [staging] START helmfile.d/services/miscweb: apply
02:32 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1028.eqiad.wmnet
02:32 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1027.eqiad.wmnet
02:32 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1027.eqiad.wmnet
02:24 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1027.eqiad.wmnet
02:23 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1027.eqiad.wmnet
02:23 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1026.eqiad.wmnet
02:23 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1026.eqiad.wmnet
02:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228 (T441633)', diff saved to https://phabricator.wikimedia.org/P97334 and previous config saved to /var/cache/conftool/dbconfig/20261105-022159-ladsgroup.json
02:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P97333 and previous config saved to /var/cache/conftool/dbconfig/20261105-021157-ladsgroup.json
02:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1026.eqiad.wmnet
02:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1026.eqiad.wmnet
02:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1025.eqiad.wmnet
02:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1025.eqiad.wmnet
02:09 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 31s)
02:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P97332 and previous config saved to /var/cache/conftool/dbconfig/20261105-020155-ladsgroup.json
02:01 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
02:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1025.eqiad.wmnet
01:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228 (T441633)', diff saved to https://phabricator.wikimedia.org/P97331 and previous config saved to /var/cache/conftool/dbconfig/20261105-015153-ladsgroup.json
01:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1025.eqiad.wmnet
01:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1024.eqiad.wmnet
01:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1024.eqiad.wmnet
01:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1024.eqiad.wmnet
00:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1024.eqiad.wmnet
00:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1023.eqiad.wmnet
00:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1023.eqiad.wmnet
00:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2228 (T441633)', diff saved to https://phabricator.wikimedia.org/P97330 and previous config saved to /var/cache/conftool/dbconfig/20261105-005129-ladsgroup.json
00:51 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2228.codfw.wmnet with reason: Maintenance
00:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222 (T441633)', diff saved to https://phabricator.wikimedia.org/P97329 and previous config saved to /var/cache/conftool/dbconfig/20261105-005107-ladsgroup.json
00:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1023.eqiad.wmnet
00:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P97328 and previous config saved to /var/cache/conftool/dbconfig/20261105-004105-ladsgroup.json
00:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P97327 and previous config saved to /var/cache/conftool/dbconfig/20261105-003103-ladsgroup.json
00:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222 (T441633)', diff saved to https://phabricator.wikimedia.org/P97326 and previous config saved to /var/cache/conftool/dbconfig/20261105-002101-ladsgroup.json
00:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2222 (T441633)', diff saved to https://phabricator.wikimedia.org/P97325 and previous config saved to /var/cache/conftool/dbconfig/20261105-001358-ladsgroup.json
00:13 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2222.codfw.wmnet with reason: Maintenance
00:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221 (T441633)', diff saved to https://phabricator.wikimedia.org/P97324 and previous config saved to /var/cache/conftool/dbconfig/20261105-001336-ladsgroup.json
00:11 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1023.eqiad.wmnet
00:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1022.eqiad.wmnet
00:11 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1022.eqiad.wmnet
00:05 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1022.eqiad.wmnet
00:04 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1022.eqiad.wmnet
00:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1021.eqiad.wmnet
00:04 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1021.eqiad.wmnet
00:03 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P97323 and previous config saved to /var/cache/conftool/dbconfig/20261105-000334-ladsgroup.json
00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync

2026-11-04

23:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1021.eqiad.wmnet
23:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P97322 and previous config saved to /var/cache/conftool/dbconfig/20261104-235332-ladsgroup.json
23:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221 (T441633)', diff saved to https://phabricator.wikimedia.org/P97321 and previous config saved to /var/cache/conftool/dbconfig/20261104-234330-ladsgroup.json
23:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2221 (T441633)', diff saved to https://phabricator.wikimedia.org/P97320 and previous config saved to /var/cache/conftool/dbconfig/20261104-233620-ladsgroup.json
23:36 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2221.codfw.wmnet with reason: Maintenance
23:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220 (T441633)', diff saved to https://phabricator.wikimedia.org/P97319 and previous config saved to /var/cache/conftool/dbconfig/20261104-233559-ladsgroup.json
23:27 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1021.eqiad.wmnet
23:27 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1020.eqiad.wmnet
23:27 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1020.eqiad.wmnet
23:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P97318 and previous config saved to /var/cache/conftool/dbconfig/20261104-232557-ladsgroup.json
23:24 jdlrobson@deploy2003: Finished scap sync-world: Backport for UnsaveDialog: Don't render in place in bottom sheet mode (T443481) (1.48.0-wmf.3) (duration: 16m 34s)
23:16 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
23:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P97317 and previous config saved to /var/cache/conftool/dbconfig/20261104-231555-ladsgroup.json
23:15 jdlrobson@deploy2003: jdlrobson: Backport for UnsaveDialog: Don't render in place in bottom sheet mode (T443481) (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
23:08 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1020.eqiad.wmnet
23:07 jdlrobson@deploy2003: Started scap sync-world: Backport for UnsaveDialog: Don't render in place in bottom sheet mode (T443481) (1.48.0-wmf.3)
23:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1020.eqiad.wmnet
23:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1019.eqiad.wmnet
23:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1019.eqiad.wmnet
23:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220 (T441633)', diff saved to https://phabricator.wikimedia.org/P97316 and previous config saved to /var/cache/conftool/dbconfig/20261104-230554-ladsgroup.json
22:58 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2220 (T441633)', diff saved to https://phabricator.wikimedia.org/P97315 and previous config saved to /var/cache/conftool/dbconfig/20261104-225843-ladsgroup.json
22:58 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2220.codfw.wmnet with reason: Maintenance
22:58 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218 (T441633)', diff saved to https://phabricator.wikimedia.org/P97314 and previous config saved to /var/cache/conftool/dbconfig/20261104-225821-ladsgroup.json
22:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218', diff saved to https://phabricator.wikimedia.org/P97312 and previous config saved to /var/cache/conftool/dbconfig/20261104-224819-ladsgroup.json
22:43 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1019.eqiad.wmnet
22:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1019.eqiad.wmnet
22:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1018.eqiad.wmnet
22:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1018.eqiad.wmnet
22:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218', diff saved to https://phabricator.wikimedia.org/P97311 and previous config saved to /var/cache/conftool/dbconfig/20261104-223817-ladsgroup.json
22:28 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1018.eqiad.wmnet
22:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218 (T441633)', diff saved to https://phabricator.wikimedia.org/P97310 and previous config saved to /var/cache/conftool/dbconfig/20261104-222815-ladsgroup.json
22:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1018.eqiad.wmnet
22:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1017.eqiad.wmnet
22:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1017.eqiad.wmnet
22:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2218 (T441633)', diff saved to https://phabricator.wikimedia.org/P97309 and previous config saved to /var/cache/conftool/dbconfig/20261104-222124-ladsgroup.json
22:21 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2218.codfw.wmnet with reason: Maintenance
22:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208 (T441633)', diff saved to https://phabricator.wikimedia.org/P97308 and previous config saved to /var/cache/conftool/dbconfig/20261104-222102-ladsgroup.json
22:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1017.eqiad.wmnet
22:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1017.eqiad.wmnet
22:19 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1016.eqiad.wmnet
22:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1016.eqiad.wmnet
22:17 ejegg@deploy2003: Finished scap sync-world: Backport for Update CentralNotice for new mixin (1.48.0-wmf.1, 1.48.0-wmf.3) (duration: 41m 33s)
22:13 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1016.eqiad.wmnet
22:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1016.eqiad.wmnet
22:12 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1015.eqiad.wmnet
22:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1015.eqiad.wmnet
22:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P97307 and previous config saved to /var/cache/conftool/dbconfig/20261104-221101-ladsgroup.json
22:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1015.eqiad.wmnet
22:05 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1015.eqiad.wmnet
22:05 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1014.eqiad.wmnet
22:05 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1014.eqiad.wmnet
22:02 ejegg@deploy2003: ejegg: Continuing with deployment
22:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P97306 and previous config saved to /var/cache/conftool/dbconfig/20261104-220059-ladsgroup.json
22:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1014.eqiad.wmnet
21:57 ejegg@deploy2003: ejegg: Backport for Update CentralNotice for new mixin (1.48.0-wmf.1, 1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
21:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208 (T441633)', diff saved to https://phabricator.wikimedia.org/P97305 and previous config saved to /var/cache/conftool/dbconfig/20261104-215057-ladsgroup.json
21:48 tsev@deploy2003: mwscript-k8s job started: purgeList.php # T443456
21:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2208 (T441633)', diff saved to https://phabricator.wikimedia.org/P97304 and previous config saved to /var/cache/conftool/dbconfig/20261104-214349-ladsgroup.json
21:43 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2208.codfw.wmnet with reason: Maintenance
21:41 tsev@deploy2003: mwscript-k8s job started: purgeList.php # T443456
21:36 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2198.codfw.wmnet with reason: Maintenance
21:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2182 (T441633)', diff saved to https://phabricator.wikimedia.org/P97303 and previous config saved to /var/cache/conftool/dbconfig/20261104-213547-ladsgroup.json
21:35 ejegg@deploy2003: Started scap sync-world: Backport for Update CentralNotice for new mixin (1.48.0-wmf.1, 1.48.0-wmf.3)
21:30 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1014.eqiad.wmnet
21:29 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1013.eqiad.wmnet
21:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1013.eqiad.wmnet
21:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P97302 and previous config saved to /var/cache/conftool/dbconfig/20261104-212545-ladsgroup.json
21:22 tsev@deploy2003: Finished scap sync-world: Backport for Allow ReadingLists path in Apple App Site Association File (T443456) (duration: 11m 26s)
21:16 tsev@deploy2003: tsev: Continuing with deployment
21:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P97301 and previous config saved to /var/cache/conftool/dbconfig/20261104-211544-ladsgroup.json
21:13 tsev@deploy2003: tsev: Backport for Allow ReadingLists path in Apple App Site Association File (T443456) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
19:37 mutante: !log is not !logging
19:29 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1018.eqiad.wmnet, repooling both afterwards
19:29 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2014.codfw.wmnet, repooling both afterwards
19:28 ladsgroup@deploy2003: Finished scap sync-world: Backport for Redesign 404 pages (T129433) (duration: 09m 46s)
19:22 ladsgroup@deploy2003: ladsgroup, ladsgroup-claude: Continuing with deployment
19:22 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1018.eqiad.wmnet, repooling both afterwards
19:21 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T441682, seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2014.codfw.wmnet, repooling both afterwards
19:20 ladsgroup@deploy2003: ladsgroup, ladsgroup-claude: Backport for Redesign 404 pages (T129433) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
19:18 ladsgroup@deploy2003: Started scap sync-world: Backport for Redesign 404 pages (T129433)
19:16 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2013.codfw.wmnet, repooling both afterwards
19:16 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1017.eqiad.wmnet, repooling both afterwards
19:08 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T441682, seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2013.codfw.wmnet, repooling both afterwards
19:08 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1017.eqiad.wmnet, repooling both afterwards
19:08 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1016.eqiad.wmnet, repooling both afterwards
19:07 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2012.codfw.wmnet, repooling both afterwards
19:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1009.eqiad.wmnet
19:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1008.eqiad.wmnet
19:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1008.eqiad.wmnet
19:00 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1016.eqiad.wmnet, repooling both afterwards
18:59 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T441682, seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2012.codfw.wmnet, repooling both afterwards
18:59 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1008.eqiad.wmnet
18:53 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2011.codfw.wmnet, repooling both afterwards
18:53 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1014.eqiad.wmnet, repooling both afterwards
18:48 ladsgroup@deploy2003: Finished scap sync-world: Pushing 503 redesigns (T129433) (duration: 09m 40s)
18:46 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T441682, seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2011.codfw.wmnet, repooling both afterwards
18:45 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1014.eqiad.wmnet, repooling both afterwards
18:45 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2010.codfw.wmnet, repooling both afterwards
18:45 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1013.eqiad.wmnet, repooling both afterwards
18:42 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
18:40 ladsgroup@deploy2003: ladsgroup: Pushing 503 redesigns (T129433) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
18:39 ladsgroup@deploy2003: Started scap sync-world: Pushing 503 redesigns (T129433)
18:37 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T441682, seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2010.codfw.wmnet, repooling both afterwards
18:37 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1013.eqiad.wmnet, repooling both afterwards
18:35 ladsgroup@deploy2003: Stopping before sync operations
18:34 ladsgroup@deploy2003: Started scap sync-world: Pushing 503 redesigns (T129433)
18:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1008.eqiad.wmnet
18:29 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1007.eqiad.wmnet
18:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1007.eqiad.wmnet
18:27 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2007.codfw.wmnet, repooling both afterwards
18:26 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1012.eqiad.wmnet, repooling both afterwards
18:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1007.eqiad.wmnet
18:19 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T441682, seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2007.codfw.wmnet, repooling both afterwards
18:18 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1012.eqiad.wmnet, repooling both afterwards
18:12 krinkle@deploy2003: Finished scap sync-world: Backport for Remove Mathoid SVG option from wgMathValidModes (T271001) (duration: 13m 43s)
18:11 mutante: gerrit2003/gerrit2002 - systemctl restart gerrit to deploy gerrit:1308081 to cap index query results to 5000 (T431398)
18:06 krinkle@deploy2003: krinkle: Continuing with deployment
18:01 krinkle@deploy2003: krinkle: Backport for Remove Mathoid SVG option from wgMathValidModes (T271001) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
17:58 krinkle@deploy2003: Started scap sync-world: Backport for Remove Mathoid SVG option from wgMathValidModes (T271001)
17:54 ariel@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
17:53 ariel@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
17:53 ariel@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
17:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1007.eqiad.wmnet
17:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1006.eqiad.wmnet
17:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1006.eqiad.wmnet
17:52 ariel@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
17:52 ariel@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
17:51 ariel@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
17:51 ariel@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
17:51 ariel@deploy2003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
17:49 ariel@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
17:48 ariel@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
17:48 ariel@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
17:47 ariel@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
17:46 ariel@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
17:45 ariel@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
17:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1006.eqiad.wmnet
17:44 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1006.eqiad.wmnet
17:44 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1005.eqiad.wmnet
17:44 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1005.eqiad.wmnet
17:37 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1005.eqiad.wmnet
17:31 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
17:26 fabfur: apt: included haproxykafka 0.3.18 into trixie-wikimedia (T443632)
17:26 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
17:13 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lsw1-b3-eqiad,ssw1-a[1,8]-eqiad with reason: desperate troubleshooting measure
17:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1005.eqiad.wmnet
17:07 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1004.eqiad.wmnet
17:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1004.eqiad.wmnet
17:02 btullis@cumin1004: START - Cookbook sre.hadoop.reboot-workers for Hadoop analytics cluster
17:02 ladsgroup@deploy2003: Finished scap sync-world: Backport for Redesign 503 pages (T129433) (duration: 11m 19s)
16:59 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1004.eqiad.wmnet
16:58 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1004.eqiad.wmnet
16:58 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1003.eqiad.wmnet
16:58 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1003.eqiad.wmnet
16:56 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
16:53 ladsgroup@deploy2003: ladsgroup: Backport for Redesign 503 pages (T129433) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
16:51 ladsgroup@deploy2003: Started scap sync-world: Backport for Redesign 503 pages (T129433)
16:47 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1003.eqiad.wmnet
16:47 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1003.eqiad.wmnet
16:47 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1002.eqiad.wmnet
16:46 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1002.eqiad.wmnet
16:36 gmodena@deploy2003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
16:36 gmodena@deploy2003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
16:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1002.eqiad.wmnet
16:32 gmodena@deploy2003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
16:31 gmodena@deploy2003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
16:29 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs1011.eqiad.wmnet, repooling both afterwards
16:21 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs1011.eqiad.wmnet, repooling both afterwards
16:19 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238 (T441633)', diff saved to https://phabricator.wikimedia.org/P97287 and previous config saved to /var/cache/conftool/dbconfig/20261104-161948-ladsgroup.json
16:19 gmodena@deploy2003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
16:18 gmodena@deploy2003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
16:09 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P97286 and previous config saved to /var/cache/conftool/dbconfig/20261104-160946-ladsgroup.json
16:06 kamila@deploy2003: Finished scap sync-world: Backport for Enroll 1% of client sessions in PHP 8.5 (T441689) (duration: 10m 02s)
16:03 pt1979@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool eqsin [reason: no reason specified, T416563]
16:03 pt1979@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool eqsin [reason: no reason specified, T416563]
16:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1002.eqiad.wmnet
16:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1001.eqiad.wmnet
16:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1001.eqiad.wmnet
16:00 kamila@deploy2003: kamila: Continuing with deployment
15:59 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P97285 and previous config saved to /var/cache/conftool/dbconfig/20261104-155943-ladsgroup.json
15:59 pt1979@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cr2-eqsin,cr2-eqsin IPv6,cr2-eqsin.mgmt
15:59 pt1979@cumin1004: START - Cookbook sre.hosts.remove-downtime for cr2-eqsin,cr2-eqsin IPv6,cr2-eqsin.mgmt
15:58 kamila@deploy2003: kamila: Backport for Enroll 1% of client sessions in PHP 8.5 (T441689) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
15:56 kamila@deploy2003: Started scap sync-world: Backport for Enroll 1% of client sessions in PHP 8.5 (T441689)
15:51 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
15:51 ayounsi@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove old cr2-eqord lo IPs - ayounsi@cumin1004"
15:51 ayounsi@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove old cr2-eqord lo IPs - ayounsi@cumin1004"
15:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238 (T441633)', diff saved to https://phabricator.wikimedia.org/P97284 and previous config saved to /var/cache/conftool/dbconfig/20261104-154941-ladsgroup.json
15:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1001.eqiad.wmnet
15:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1001.eqiad.wmnet
15:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test1001.eqiad.wmnet
15:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test1001.eqiad.wmnet
15:46 ayounsi@cumin1004: START - Cookbook sre.dns.netbox
15:42 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2238 (T441633)', diff saved to https://phabricator.wikimedia.org/P97283 and previous config saved to /var/cache/conftool/dbconfig/20261104-154218-ladsgroup.json
15:42 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2238.codfw.wmnet with reason: Maintenance
15:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226 (T441633)', diff saved to https://phabricator.wikimedia.org/P97282 and previous config saved to /var/cache/conftool/dbconfig/20261104-154156-ladsgroup.json
15:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test1001.eqiad.wmnet
15:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test1001.eqiad.wmnet
15:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1004.eqiad.wmnet
15:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1004.eqiad.wmnet
15:40 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
15:39 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
15:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1004.eqiad.wmnet
15:33 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1004.eqiad.wmnet
15:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1003.eqiad.wmnet
15:33 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1003.eqiad.wmnet
15:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P97281 and previous config saved to /var/cache/conftool/dbconfig/20261104-153154-ladsgroup.json
15:31 kharlan@deploy2003: Finished scap sync-world: Backport for AbuseReview: Add undo and rollback links to all queues (T443591) (1.48.0-wmf.3) (duration: 09m 14s)
15:28 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
15:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1003.eqiad.wmnet
15:26 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
15:25 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1003.eqiad.wmnet
15:25 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1002.eqiad.wmnet
15:25 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1002.eqiad.wmnet
15:25 kharlan@deploy2003: kharlan: Continuing with deployment
15:24 kharlan@deploy2003: kharlan: Backport for AbuseReview: Add undo and rollback links to all queues (T443591) (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
15:22 kharlan@deploy2003: Started scap sync-world: Backport for AbuseReview: Add undo and rollback links to all queues (T443591) (1.48.0-wmf.3)
15:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P97280 and previous config saved to /var/cache/conftool/dbconfig/20261104-152152-ladsgroup.json
15:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1002.eqiad.wmnet
15:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1002.eqiad.wmnet
15:19 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1001.eqiad.wmnet
15:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1001.eqiad.wmnet
15:18 cwilliams@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts db1176.eqiad.wmnet
15:18 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host db1176.eqiad.wmnet
15:13 sbisson@deploy2003: Finished scap sync-world: Backport for ArticleGuidance: Enable the reference risk model in production (T442164), Load ArticleGuidance on all Wikipedias (T442035) (duration: 12m 29s)
15:12 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1001.eqiad.wmnet
15:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1001.eqiad.wmnet
15:11 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-worker-eqiad
15:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226 (T441633)', diff saved to and previous config saved to /var/cache/conftool/dbconfig/20261104-151151-ladsgroup.json
15:10 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host db1176.eqiad.wmnet
15:08 jforrester@deploy2003: Finished scap deploy-service wikifunctions: Weekly service deploy, function-orchestrator second (1363248) (duration: 01m 46s)
15:08 cwilliams@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1176.eqiad.wmnet
15:07 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions
15:07 sbisson@deploy2003: sbisson: Continuing with deployment
15:06 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions
15:06 jforrester@deploy2003: scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions
15:06 jforrester@deploy2003: Started scap deploy-service wikifunctions: Weekly service deploy, function-orchestrator second (1363248)
15:05 sbisson@deploy2003: sbisson: Backport for ArticleGuidance: Enable the reference risk model in production (T442164), Load ArticleGuidance on all Wikipedias (T442035) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
15:05 papaul: start maintenance on cr2-eqsin
15:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2226 (T441633)', diff saved to https://phabricator.wikimedia.org/P97279 and previous config saved to /var/cache/conftool/dbconfig/20261104-150502-ladsgroup.json
15:05 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2226.codfw.wmnet with reason: Maintenance
15:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225 (T441633)', diff saved to https://phabricator.wikimedia.org/P97278 and previous config saved to /var/cache/conftool/dbconfig/20261104-150440-ladsgroup.json
15:03 jforrester@deploy2003: Finished scap deploy-service wikifunctions: Weekly service deploy, function-evaluators first (1363247) (duration: 02m 28s)
15:03 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 8 hosts with reason: Testing
15:03 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions
15:02 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions
15:01 jforrester@deploy2003: scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions
15:01 jforrester@deploy2003: Started scap deploy-service wikifunctions: Weekly service deploy, function-evaluators first (1363247)
15:00 sbisson@deploy2003: Started scap sync-world: Backport for ArticleGuidance: Enable the reference risk model in production (T442164), Load ArticleGuidance on all Wikipedias (T442035)
15:00 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary # T440137
15:00 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary --add-prefix=BROKEN --fix # T440137
14:59 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary # T440137
14:59 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary --fix # T440137
14:59 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary # T440137
14:57 urbanecm@deploy2003: Finished scap sync-world: Backport for Rename project namespace on Madurese Wiktionary (T440137) (duration: 11m 14s)
14:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P97277 and previous config saved to /var/cache/conftool/dbconfig/20261104-145438-ladsgroup.json
14:49 urbanecm@deploy2003: urbanecm, amire80: Continuing with deployment
14:49 cwilliams@cumin1004: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts db1176.eqiad.wmnet
14:48 urbanecm@deploy2003: urbanecm, amire80: Backport for Rename project namespace on Madurese Wiktionary (T440137) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
14:46 urbanecm@deploy2003: Started scap sync-world: Backport for Rename project namespace on Madurese Wiktionary (T440137)
14:45 urbanecm@deploy2003: Finished scap sync-world: Backport for Re-enable NewUserMessage for viwikivoyage (T442485), Configure cookie-based enrollment in PHP 8.5 (T441689) (duration: 29m 32s)
14:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P97275 and previous config saved to /var/cache/conftool/dbconfig/20261104-144436-ladsgroup.json
14:43 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
14:42 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-eqsin,cr2-eqsin IPv6,cr2-eqsin.mgmt with reason: Junos upgrade
14:39 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
14:38 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
14:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225 (T441633)', diff saved to https://phabricator.wikimedia.org/P97274 and previous config saved to /var/cache/conftool/dbconfig/20261104-143434-ladsgroup.json
14:33 cwilliams@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1176.eqiad.wmnet
14:32 pt1979@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool eqsin [reason: router upgrade, T416563]
14:32 pt1979@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool eqsin [reason: router upgrade, T416563]
14:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2225 (T441633)', diff saved to https://phabricator.wikimedia.org/P97273 and previous config saved to /var/cache/conftool/dbconfig/20261104-142729-ladsgroup.json
14:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2225.codfw.wmnet with reason: Maintenance
14:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207 (T441633)', diff saved to https://phabricator.wikimedia.org/P97272 and previous config saved to /var/cache/conftool/dbconfig/20261104-142707-ladsgroup.json
14:19 urbanecm@deploy2003: kamila, nvdtn19, urbanecm: Continuing with deployment
14:18 urbanecm@deploy2003: kamila, nvdtn19, urbanecm: Backport for Re-enable NewUserMessage for viwikivoyage (T442485), Configure cookie-based enrollment in PHP 8.5 (T441689) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
14:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P97270 and previous config saved to /var/cache/conftool/dbconfig/20261104-141705-ladsgroup.json
14:15 urbanecm@deploy2003: Started scap sync-world: Backport for Re-enable NewUserMessage for viwikivoyage (T442485), Configure cookie-based enrollment in PHP 8.5 (T441689)
14:15 urbanecm@deploy2003: urbanecm, nvdtn19: Continuing with deployment
14:08 urbanecm@deploy2003: urbanecm, nvdtn19: Backport for Re-enable NewUserMessage for viwikivoyage (T442485) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
14:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P97269 and previous config saved to /var/cache/conftool/dbconfig/20261104-140703-ladsgroup.json
14:06 kamila@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
14:05 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
14:05 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
14:03 urbanecm@deploy2003: Started scap sync-world: Backport for Re-enable NewUserMessage for viwikivoyage (T442485)
14:01 jelto@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/etherpad: apply
14:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-worker-codfw
14:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2011.codfw.wmnet
14:00 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2011.codfw.wmnet
13:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207 (T441633)', diff saved to https://phabricator.wikimedia.org/P97268 and previous config saved to /var/cache/conftool/dbconfig/20261104-135701-ladsgroup.json
13:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2011.codfw.wmnet
13:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2011.codfw.wmnet
13:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2010.codfw.wmnet
13:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2010.codfw.wmnet
13:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2207 (T441633)', diff saved to https://phabricator.wikimedia.org/P97267 and previous config saved to /var/cache/conftool/dbconfig/20261104-134951-ladsgroup.json
13:49 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2207.codfw.wmnet with reason: Maintenance
13:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2010.codfw.wmnet
13:44 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2197.codfw.wmnet with reason: Maintenance
13:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189 (T441633)', diff saved to https://phabricator.wikimedia.org/P97266 and previous config saved to /var/cache/conftool/dbconfig/20261104-134347-ladsgroup.json
13:41 jelto@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/etherpad: apply
13:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2010.codfw.wmnet
13:39 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2009.codfw.wmnet
13:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2009.codfw.wmnet
13:33 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P97265 and previous config saved to /var/cache/conftool/dbconfig/20261104-133345-ladsgroup.json
13:32 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2009.codfw.wmnet
13:32 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2009.codfw.wmnet
13:32 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2008.codfw.wmnet
13:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2008.codfw.wmnet
13:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2008.codfw.wmnet
13:23 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P97264 and previous config saved to /var/cache/conftool/dbconfig/20261104-132343-ladsgroup.json
13:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2008.codfw.wmnet
13:21 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2007.codfw.wmnet
13:21 mszwarc@deploy2003: Finished scap sync-world: Backport for SECURITY: AbuseLogLookup: Don't count logs the user can't see (T439885) (1.48.0-wmf.1, 1.48.0-wmf.3) (duration: 11m 06s)
13:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2007.codfw.wmnet
13:17 moritzm: installing expat security updates
13:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2007.codfw.wmnet
13:15 mszwarc@deploy2003: mszwarc: Continuing with deployment
13:15 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2007.codfw.wmnet
13:15 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2006.codfw.wmnet
13:15 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2006.codfw.wmnet
13:14 mszwarc@deploy2003: mszwarc: Backport for SECURITY: AbuseLogLookup: Don't count logs the user can't see (T439885) (1.48.0-wmf.1, 1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
13:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189 (T441633)', diff saved to https://phabricator.wikimedia.org/P97263 and previous config saved to /var/cache/conftool/dbconfig/20261104-131341-ladsgroup.json
13:10 mszwarc@deploy2003: Started scap sync-world: Backport for SECURITY: AbuseLogLookup: Don't count logs the user can't see (T439885) (1.48.0-wmf.1, 1.48.0-wmf.3)
13:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2006.codfw.wmnet
13:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2006.codfw.wmnet
13:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2005.codfw.wmnet
13:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2005.codfw.wmnet
13:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2005.codfw.wmnet
13:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2005.codfw.wmnet
13:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2004.codfw.wmnet
13:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2004.codfw.wmnet
13:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
13:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1006.eqiad.wmnet
13:01 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1006.eqiad.wmnet
13:01 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
13:01 ayounsi@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cr3-esams mgmt - ayounsi@cumin1004"
13:01 ayounsi@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cr3-esams mgmt - ayounsi@cumin1004"
12:58 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
12:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2004.codfw.wmnet
12:57 ayounsi@cumin1004: START - Cookbook sre.dns.netbox
12:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2004.codfw.wmnet
12:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2003.codfw.wmnet
12:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2003.codfw.wmnet
12:56 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
12:54 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1006.eqiad.wmnet
12:54 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1006.eqiad.wmnet
12:54 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1005.eqiad.wmnet
12:54 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1005.eqiad.wmnet
12:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2003.codfw.wmnet
12:51 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2003.codfw.wmnet
12:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2002.codfw.wmnet
12:50 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2002.codfw.wmnet
12:50 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
12:47 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
12:47 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1005.eqiad.wmnet
12:46 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
12:46 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
12:46 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1005.eqiad.wmnet
12:46 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1004.eqiad.wmnet
12:46 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1004.eqiad.wmnet
12:45 claime: donne with poolcounter reboots
12:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2002.codfw.wmnet
12:44 cgoubert@deploy2003: Finished scap sync-world: Backport for ProductionServices: Repool poolcounter2006 after reboot (#5/5) (T441653) (duration: 10m 58s)
12:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2002.codfw.wmnet
12:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2001.codfw.wmnet
12:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2001.codfw.wmnet
12:39 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1004.eqiad.wmnet
12:38 cgoubert@deploy2003: cgoubert: Continuing with deployment
12:37 cgoubert@deploy2003: cgoubert: Backport for ProductionServices: Repool poolcounter2006 after reboot (#5/5) (T441653) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
12:34 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1004.eqiad.wmnet
12:34 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1003.eqiad.wmnet
12:34 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1003.eqiad.wmnet
12:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2001.codfw.wmnet
12:33 cgoubert@deploy2003: Started scap sync-world: Backport for ProductionServices: Repool poolcounter2006 after reboot (#5/5) (T441653)
12:31 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2006.codfw.wmnet
12:28 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2001.codfw.wmnet
12:28 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1003.eqiad.wmnet
12:28 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test2001.codfw.wmnet
12:28 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test2001.codfw.wmnet
12:27 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1003.eqiad.wmnet
12:27 blake@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
12:27 blake@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
12:27 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2004.codfw.wmnet
12:27 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2004.codfw.wmnet
12:27 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter2006.codfw.wmnet
12:26 cgoubert@deploy2003: Finished scap sync-world: Backport for ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) (T441653) (duration: 10m 32s)
12:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test2001.codfw.wmnet
12:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test2001.codfw.wmnet
12:21 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2005.codfw.wmnet
12:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2005.codfw.wmnet
12:21 cgoubert@deploy2003: cgoubert: Continuing with deployment
12:20 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2004.codfw.wmnet
12:20 cgoubert@deploy2003: cgoubert: Backport for ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) (T441653) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
12:18 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
12:17 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
12:16 cgoubert@deploy2003: Started scap sync-world: Backport for ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) (T441653)
12:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2005.codfw.wmnet
12:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2005.codfw.wmnet
12:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2004.codfw.wmnet
12:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2004.codfw.wmnet
12:15 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2004.codfw.wmnet
12:15 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2003.codfw.wmnet
12:15 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2003.codfw.wmnet
12:14 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2005.codfw.wmnet
12:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2189 (T441633)', diff saved to https://phabricator.wikimedia.org/P97262 and previous config saved to /var/cache/conftool/dbconfig/20261104-121318-ladsgroup.json
12:13 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2189.codfw.wmnet with reason: Maintenance
12:13 moritzm: installing apt security updates
12:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T441633)', diff saved to https://phabricator.wikimedia.org/P97261 and previous config saved to /var/cache/conftool/dbconfig/20261104-121256-ladsgroup.json
12:12 mvolz@deploy2003: Finished scap deploy-service citoid: Update user-agent (duration: 05m 26s)
12:10 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter2005.codfw.wmnet
12:10 mvolz@deploy2003: scap deploy-service citoid: [PRODUCTION/DEFAULT] codfw Deploying citoid
12:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2004.codfw.wmnet
12:10 cgoubert@deploy2003: Finished scap sync-world: Backport for ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) (T441653) (duration: 15m 19s)
12:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2004.codfw.wmnet
12:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2003.codfw.wmnet
12:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2003.codfw.wmnet
12:09 mvolz@deploy2003: scap deploy-service citoid: [PRODUCTION/DEFAULT] eqiad Deploying citoid
12:08 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2003.codfw.wmnet
12:08 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2003.codfw.wmnet
12:08 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2002.codfw.wmnet
12:07 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2002.codfw.wmnet
12:07 mvolz@deploy2003: scap deploy-service citoid: [STAGING/DEFAULT] staging-eqiad Deploying citoid
12:07 mvolz@deploy2003: Started scap deploy-service citoid: Update user-agent
12:04 cgoubert@deploy2003: cgoubert: Continuing with deployment
12:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2003.codfw.wmnet
12:03 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
12:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2003.codfw.wmnet
12:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2002.codfw.wmnet
12:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2002.codfw.wmnet
12:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P97260 and previous config saved to /var/cache/conftool/dbconfig/20261104-120254-ladsgroup.json
12:02 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
12:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2002.codfw.wmnet
12:01 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2002.codfw.wmnet
12:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2001.codfw.wmnet
12:01 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2001.codfw.wmnet
12:01 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
11:59 cgoubert@deploy2003: cgoubert: Backport for ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) (T441653) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
11:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2002.codfw.wmnet
11:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2002.codfw.wmnet
11:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2001.codfw.wmnet
11:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2001.codfw.wmnet
11:55 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet
11:55 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet
11:55 blake@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
11:54 cgoubert@deploy2003: Started scap sync-world: Backport for ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) (T441653)
11:53 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1007.eqiad.wmnet
11:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2001.codfw.wmnet
11:52 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P97258 and previous config saved to /var/cache/conftool/dbconfig/20261104-115252-ladsgroup.json
11:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2001.codfw.wmnet
11:52 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-worker-codfw
11:49 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter1007.eqiad.wmnet
11:48 cgoubert@deploy2003: Finished scap sync-world: Backport for ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653), ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) (T441653) (duration: 10m 24s)
11:44 tappof: putting metamonitoring checks into "maintenance mode without sending a notification" to update the metamonitoring script T443406
11:42 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T441633)', diff saved to https://phabricator.wikimedia.org/P97257 and previous config saved to /var/cache/conftool/dbconfig/20261104-114251-ladsgroup.json
11:42 cgoubert@deploy2003: cgoubert: Continuing with deployment
11:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-mariadb1002.eqiad.wmnet
11:41 cgoubert@deploy2003: cgoubert: Backport for ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653), ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) (T441653) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
11:37 cgoubert@deploy2003: Started scap sync-world: Backport for ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653), ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) (T441653)
11:35 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-mariadb1002.eqiad.wmnet
11:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2175 (T441633)', diff saved to https://phabricator.wikimedia.org/P97256 and previous config saved to /var/cache/conftool/dbconfig/20261104-113517-ladsgroup.json
11:35 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2175.codfw.wmnet with reason: Maintenance
11:28 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1262.eqiad.wmnet with reason: Reboot
11:28 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1006.eqiad.wmnet
11:25 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter1006.eqiad.wmnet
11:24 cgoubert@deploy2003: Finished scap sync-world: Backport for ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653) (duration: 08m 39s)
11:21 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-mariadb1001.eqiad.wmnet
11:20 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1262.eqiad.wmnet with reason: Reboot
11:19 cgoubert@deploy2003: cgoubert: Continuing with deployment
11:18 cgoubert@deploy2003: cgoubert: Backport for ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
11:17 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
11:16 cgoubert@deploy2003: Started scap sync-world: Backport for ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)
11:15 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
11:15 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-mariadb1001.eqiad.wmnet
11:14 claime: Starting poolcounter reboot sequence
11:12 ladsgroup@deploy2003: Finished scap sync-world: Backport for Better browser support for DASH (T442988) (1.48.0-wmf.3) (duration: 12m 51s)
11:06 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
11:04 ladsgroup@deploy2003: ladsgroup: Backport for Better browser support for DASH (T442988) (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
11:02 fabfur: enabling puppet again on A:cp to apply fix https://gerrit.wikimedia.org/r/c/operations/puppet/+/1363067
10:59 ladsgroup@deploy2003: Started scap sync-world: Backport for Better browser support for DASH (T442988) (1.48.0-wmf.3)
10:52 fabfur: temporary disabling puppet on A:cp to fix https://gerrit.wikimedia.org/r/c/operations/puppet/+/1328543
10:49 moritzm: installing vim security updates
10:34 cmooney@cumin1004: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host lvs1020.eqiad.wmnet
10:32 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2004.codfw.wmnet with OS trixie
10:28 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
10:28 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
10:28 klausman@cumin1004: END (ERROR) - Cookbook sre.hadoop.check-user-leftovers (exit_code=97) Checking for leftover data for user akosiaris
10:28 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
10:27 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
10:27 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
10:19 gkyziridis@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
10:19 gkyziridis@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
10:18 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
10:17 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
10:14 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2004.codfw.wmnet with reason: host reimage
10:12 cmooney@cumin1004: START - Cookbook sre.hosts.reboot-single for host lvs1020.eqiad.wmnet
10:11 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
10:10 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
10:10 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
10:09 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
10:06 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2004.codfw.wmnet with reason: host reimage
10:03 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2004.codfw.wmnet (remove config only)
10:03 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2004.codfw.wmnet (remove config only)
09:59 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
09:59 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
09:58 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
09:58 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
09:54 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook: apply
09:54 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook: apply
09:53 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2036: Maintenance
09:50 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply
09:49 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2004.codfw.wmnet (push config only)
09:49 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2004.codfw.wmnet (push config only)
09:49 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply
09:48 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2004.codfw.wmnet with OS trixie
09:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/ferretdb-growthbook: apply
09:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/ferretdb-growthbook: apply
09:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply
09:44 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply
09:31 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2006.codfw.wmnet with OS trixie
09:21 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook: apply
09:21 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook: apply
09:14 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook: apply
09:14 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook: apply
09:13 jnuche@deploy2003: rebuilt and synchronized wikiversions files: group1 to 1.48.0-wmf.3 refs T442221
09:12 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2006.codfw.wmnet with reason: host reimage
09:08 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool es2036: Maintenance
09:07 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2006.codfw.wmnet with reason: host reimage
09:07 arnaudb@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on gerrit2002.wikimedia.org with reason: T440177 offline reindex test
09:05 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2006.codfw.wmnet (remove config only)
09:05 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2006.codfw.wmnet (remove config only)
08:51 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2006.codfw.wmnet (push config only)
08:51 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2006.codfw.wmnet (push config only)
08:50 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2006.codfw.wmnet with OS trixie
08:15 Tran: Deployed patch for T441775
08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2005.codfw.wmnet with OS trixie
08:10 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
08:06 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
08:00 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1274: after reboot
07:53 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2005.codfw.wmnet with reason: host reimage
07:47 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2005.codfw.wmnet with reason: host reimage
07:45 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2005.codfw.wmnet (remove config only)
07:45 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2005.codfw.wmnet (remove config only)
07:31 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2005.codfw.wmnet (push config only)
07:31 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2005.codfw.wmnet (push config only)
07:30 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2005.codfw.wmnet with OS trixie
07:15 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db1274: after reboot
07:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
07:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288 (T441633)', diff saved to https://phabricator.wikimedia.org/P97242 and previous config saved to /var/cache/conftool/dbconfig/20261104-070600-ladsgroup.json
06:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288', diff saved to https://phabricator.wikimedia.org/P97241 and previous config saved to /var/cache/conftool/dbconfig/20261104-065558-ladsgroup.json
06:55 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1274.eqiad.wmnet with reason: Server didn't come back
06:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288', diff saved to https://phabricator.wikimedia.org/P97240 and previous config saved to /var/cache/conftool/dbconfig/20261104-064556-ladsgroup.json
06:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288 (T441633)', diff saved to https://phabricator.wikimedia.org/P97239 and previous config saved to /var/cache/conftool/dbconfig/20261104-063554-ladsgroup.json
06:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1288 (T441633)', diff saved to https://phabricator.wikimedia.org/P97238 and previous config saved to /var/cache/conftool/dbconfig/20261104-063055-ladsgroup.json
06:30 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1288.eqiad.wmnet with reason: Maintenance
06:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1285.eqiad.wmnet with reason: Maintenance
06:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284 (T441633)', diff saved to https://phabricator.wikimedia.org/P97237 and previous config saved to /var/cache/conftool/dbconfig/20261104-062712-ladsgroup.json
06:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284', diff saved to https://phabricator.wikimedia.org/P97236 and previous config saved to /var/cache/conftool/dbconfig/20261104-061710-ladsgroup.json
06:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284', diff saved to https://phabricator.wikimedia.org/P97235 and previous config saved to /var/cache/conftool/dbconfig/20261104-060708-ladsgroup.json
05:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284 (T441633)', diff saved to https://phabricator.wikimedia.org/P97234 and previous config saved to /var/cache/conftool/dbconfig/20261104-055706-ladsgroup.json
05:52 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1284 (T441633)', diff saved to https://phabricator.wikimedia.org/P97233 and previous config saved to /var/cache/conftool/dbconfig/20261104-055201-ladsgroup.json
05:52 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1284.eqiad.wmnet with reason: Maintenance
05:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273 (T441633)', diff saved to https://phabricator.wikimedia.org/P97232 and previous config saved to /var/cache/conftool/dbconfig/20261104-055141-ladsgroup.json
05:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273', diff saved to https://phabricator.wikimedia.org/P97231 and previous config saved to /var/cache/conftool/dbconfig/20261104-054139-ladsgroup.json
05:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273', diff saved to https://phabricator.wikimedia.org/P97230 and previous config saved to /var/cache/conftool/dbconfig/20261104-053137-ladsgroup.json
05:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273 (T441633)', diff saved to https://phabricator.wikimedia.org/P97229 and previous config saved to /var/cache/conftool/dbconfig/20261104-052135-ladsgroup.json
05:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1273 (T441633)', diff saved to https://phabricator.wikimedia.org/P97228 and previous config saved to /var/cache/conftool/dbconfig/20261104-051423-ladsgroup.json
05:14 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 7 hosts with reason: Maintenance
05:14 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1273.eqiad.wmnet with reason: Maintenance
05:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253 (T441633)', diff saved to https://phabricator.wikimedia.org/P97227 and previous config saved to /var/cache/conftool/dbconfig/20261104-051351-ladsgroup.json
05:03 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P97226 and previous config saved to /var/cache/conftool/dbconfig/20261104-050349-ladsgroup.json
04:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P97225 and previous config saved to /var/cache/conftool/dbconfig/20261104-045347-ladsgroup.json
04:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253 (T441633)', diff saved to https://phabricator.wikimedia.org/P97224 and previous config saved to /var/cache/conftool/dbconfig/20261104-044345-ladsgroup.json
04:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1253 (T441633)', diff saved to https://phabricator.wikimedia.org/P97223 and previous config saved to /var/cache/conftool/dbconfig/20261104-043639-ladsgroup.json
04:36 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1253.eqiad.wmnet with reason: Maintenance
04:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231 (T441633)', diff saved to https://phabricator.wikimedia.org/P97222 and previous config saved to /var/cache/conftool/dbconfig/20261104-043619-ladsgroup.json
04:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P97221 and previous config saved to /var/cache/conftool/dbconfig/20261104-042617-ladsgroup.json
04:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P97220 and previous config saved to /var/cache/conftool/dbconfig/20261104-041615-ladsgroup.json
04:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231 (T441633)', diff saved to https://phabricator.wikimedia.org/P97219 and previous config saved to /var/cache/conftool/dbconfig/20261104-040613-ladsgroup.json
03:59 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1231 (T441633)', diff saved to https://phabricator.wikimedia.org/P97218 and previous config saved to /var/cache/conftool/dbconfig/20261104-035922-ladsgroup.json
03:59 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1231.eqiad.wmnet with reason: Maintenance
03:59 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227 (T441633)', diff saved to https://phabricator.wikimedia.org/P97217 and previous config saved to /var/cache/conftool/dbconfig/20261104-035901-ladsgroup.json
03:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P97216 and previous config saved to /var/cache/conftool/dbconfig/20261104-034859-ladsgroup.json
03:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P97215 and previous config saved to /var/cache/conftool/dbconfig/20261104-033857-ladsgroup.json
03:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227 (T441633)', diff saved to https://phabricator.wikimedia.org/P97214 and previous config saved to /var/cache/conftool/dbconfig/20261104-032855-ladsgroup.json
03:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1227 (T441633)', diff saved to https://phabricator.wikimedia.org/P97213 and previous config saved to /var/cache/conftool/dbconfig/20261104-032149-ladsgroup.json
03:21 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1227.eqiad.wmnet with reason: Maintenance
03:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202 (T441633)', diff saved to https://phabricator.wikimedia.org/P97212 and previous config saved to /var/cache/conftool/dbconfig/20261104-032129-ladsgroup.json
03:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P97211 and previous config saved to /var/cache/conftool/dbconfig/20261104-031127-ladsgroup.json
03:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P97210 and previous config saved to /var/cache/conftool/dbconfig/20261104-030125-ladsgroup.json
02:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202 (T441633)', diff saved to https://phabricator.wikimedia.org/P97209 and previous config saved to /var/cache/conftool/dbconfig/20261104-025123-ladsgroup.json
02:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1202 (T441633)', diff saved to https://phabricator.wikimedia.org/P97208 and previous config saved to /var/cache/conftool/dbconfig/20261104-024423-ladsgroup.json
02:44 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1202.eqiad.wmnet with reason: Maintenance
02:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194 (T441633)', diff saved to https://phabricator.wikimedia.org/P97207 and previous config saved to /var/cache/conftool/dbconfig/20261104-024403-ladsgroup.json
02:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P97206 and previous config saved to /var/cache/conftool/dbconfig/20261104-023401-ladsgroup.json
02:23 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P97205 and previous config saved to /var/cache/conftool/dbconfig/20261104-022358-ladsgroup.json
02:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194 (T441633)', diff saved to https://phabricator.wikimedia.org/P97204 and previous config saved to /var/cache/conftool/dbconfig/20261104-021356-ladsgroup.json
02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 33s)
02:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1194 (T441633)', diff saved to https://phabricator.wikimedia.org/P97203 and previous config saved to /var/cache/conftool/dbconfig/20261104-020545-ladsgroup.json
02:05 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1194.eqiad.wmnet with reason: Maintenance
02:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191 (T441633)', diff saved to https://phabricator.wikimedia.org/P97202 and previous config saved to /var/cache/conftool/dbconfig/20261104-020525-ladsgroup.json
02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
01:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P97201 and previous config saved to /var/cache/conftool/dbconfig/20261104-015523-ladsgroup.json
01:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P97200 and previous config saved to /var/cache/conftool/dbconfig/20261104-014521-ladsgroup.json
01:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191 (T441633)', diff saved to https://phabricator.wikimedia.org/P97199 and previous config saved to /var/cache/conftool/dbconfig/20261104-013519-ladsgroup.json
01:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1191 (T441633)', diff saved to https://phabricator.wikimedia.org/P97198 and previous config saved to /var/cache/conftool/dbconfig/20261104-012829-ladsgroup.json
01:28 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1191.eqiad.wmnet with reason: Maintenance
01:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190 (T441633)', diff saved to https://phabricator.wikimedia.org/P97197 and previous config saved to /var/cache/conftool/dbconfig/20261104-012809-ladsgroup.json
01:18 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P97196 and previous config saved to /var/cache/conftool/dbconfig/20261104-011807-ladsgroup.json
01:08 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P97195 and previous config saved to /var/cache/conftool/dbconfig/20261104-010805-ladsgroup.json
00:58 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190 (T441633)', diff saved to https://phabricator.wikimedia.org/P97194 and previous config saved to /var/cache/conftool/dbconfig/20261104-005803-ladsgroup.json
00:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1190 (T441633)', diff saved to https://phabricator.wikimedia.org/P97193 and previous config saved to /var/cache/conftool/dbconfig/20261104-004935-ladsgroup.json
00:49 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1190.eqiad.wmnet with reason: Maintenance
00:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181 (T441633)', diff saved to https://phabricator.wikimedia.org/P97192 and previous config saved to /var/cache/conftool/dbconfig/20261104-004926-ladsgroup.json
00:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P97191 and previous config saved to /var/cache/conftool/dbconfig/20261104-003924-ladsgroup.json
00:32 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/kartotherian: apply
00:30 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/kartotherian: apply
00:29 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P97190 and previous config saved to /var/cache/conftool/dbconfig/20261104-002922-ladsgroup.json
00:28 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/zotero: apply
00:28 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/zotero: apply
00:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
00:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
00:26 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
00:19 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181 (T441633)', diff saved to https://phabricator.wikimedia.org/P97189 and previous config saved to /var/cache/conftool/dbconfig/20261104-001920-ladsgroup.json
00:15 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
00:15 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
00:15 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
00:15 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply
00:14 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply
00:14 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/toolhub: apply
00:14 ladsgroup@deploy2003: Finished scap sync-world: Backport for Avoid unnecessary/aborted requests in DASH mode (T442988) (1.48.0-wmf.3), Enable chunked videos on testwiki (T373546) (duration: 13m 02s)
00:13 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/toolhub: apply
00:13 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/termbox: apply
00:13 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/termbox: apply
00:12 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply
00:12 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply
00:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1181 (T441633)', diff saved to https://phabricator.wikimedia.org/P97188 and previous config saved to /var/cache/conftool/dbconfig/20261104-001120-ladsgroup.json
00:11 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/sessionstore: apply
00:11 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1181.eqiad.wmnet with reason: Maintenance
00:10 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/sessionstore: apply
00:10 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/recommendation-api: apply
00:09 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/recommendation-api: apply
00:09 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/push-notifications: apply
00:08 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/push-notifications: apply
00:07 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
00:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1174.eqiad.wmnet with reason: Maintenance
00:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170 (T441633)', diff saved to https://phabricator.wikimedia.org/P97187 and previous config saved to /var/cache/conftool/dbconfig/20261104-000542-ladsgroup.json
00:05 ladsgroup@deploy2003: ladsgroup: Backport for Avoid unnecessary/aborted requests in DASH mode (T442988) (1.48.0-wmf.3), Enable chunked videos on testwiki (T373546) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
00:01 ladsgroup@deploy2003: Started scap sync-world: Backport for Avoid unnecessary/aborted requests in DASH mode (T442988) (1.48.0-wmf.3), Enable chunked videos on testwiki (T373546)
00:00 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/proton: apply
00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync

2026-11-03

23:59 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/proton: apply
23:58 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/page-analytics: apply
23:58 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/page-analytics: apply
23:57 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
23:56 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/mobileapps: apply
23:56 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
23:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P97186 and previous config saved to /var/cache/conftool/dbconfig/20261103-235540-ladsgroup.json
23:54 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/miscweb: apply
23:54 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/media-analytics: apply
23:53 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/media-analytics: apply
23:52 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/machinetranslation: apply
23:46 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/machinetranslation: apply
23:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P97185 and previous config saved to /var/cache/conftool/dbconfig/20261103-234538-ladsgroup.json
23:45 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/lac-lambdas: apply
23:44 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/lac-lambdas: apply
23:44 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/kartotherian: apply
23:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170 (T441633)', diff saved to https://phabricator.wikimedia.org/P97184 and previous config saved to /var/cache/conftool/dbconfig/20261103-233536-ladsgroup.json
23:34 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/kartotherian: apply
23:34 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/geo-analytics: apply
23:34 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/geo-analytics: apply
23:33 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply
23:33 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/eventstreams: apply
23:33 musikanimal@deploy2003: Finished scap sync-world: Backport for CommonSettings: remove redundant CodeMirror config settings (T440077), metawiki: Disable wish submissions on 2026-11-09 (17:00 UTC) (T442015) (duration: 14m 56s)
23:32 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/editor-analytics: apply
23:32 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/editor-analytics: apply
23:31 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/editcheck-headless: apply
23:31 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/editcheck-headless: apply
23:30 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/edit-analytics: apply
23:30 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/edit-analytics: apply
23:30 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/echostore: apply
23:30 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/echostore: apply
23:29 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/device-analytics: apply
23:29 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/device-analytics: apply
23:29 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
23:29 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/developer-portal: apply
23:28 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/data-gateway: apply
23:28 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/data-gateway: apply
23:28 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/cxserver: apply
23:28 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/cxserver: apply
23:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/commons-impact-analytics: apply
23:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/commons-impact-analytics: apply
23:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/citoid: apply
23:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/citoid: apply
23:27 musikanimal@deploy2003: musikanimal, amdrel: Continuing with deployment
23:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply
23:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply
23:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1170 (T441633)', diff saved to https://phabricator.wikimedia.org/P97183 and previous config saved to /var/cache/conftool/dbconfig/20261103-232657-ladsgroup.json
23:26 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1170.eqiad.wmnet with reason: Maintenance
23:26 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/chart-renderer: apply
23:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T441633)', diff saved to https://phabricator.wikimedia.org/P97182 and previous config saved to /var/cache/conftool/dbconfig/20261103-232637-ladsgroup.json
23:26 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/chart-renderer: apply
23:24 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/apertium: apply
23:24 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/apertium: apply
23:22 musikanimal@deploy2003: musikanimal, amdrel: Backport for CommonSettings: remove redundant CodeMirror config settings (T440077), metawiki: Disable wish submissions on 2026-11-09 (17:00 UTC) (T442015) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
23:18 musikanimal@deploy2003: Started scap sync-world: Backport for CommonSettings: remove redundant CodeMirror config settings (T440077), metawiki: Disable wish submissions on 2026-11-09 (17:00 UTC) (T442015)
23:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P97181 and previous config saved to /var/cache/conftool/dbconfig/20261103-231635-ladsgroup.json
23:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P97180 and previous config saved to /var/cache/conftool/dbconfig/20261103-230633-ladsgroup.json
22:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T441633)', diff saved to https://phabricator.wikimedia.org/P97179 and previous config saved to /var/cache/conftool/dbconfig/20261103-225631-ladsgroup.json
22:53 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.categories-reload (exit_code=0) reloading categories to wdqs2008.codfw.wmnet
22:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1158 (T441633)', diff saved to https://phabricator.wikimedia.org/P97178 and previous config saved to /var/cache/conftool/dbconfig/20261103-224816-ladsgroup.json
22:48 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1158.eqiad.wmnet with reason: Maintenance
22:44 kemayo@deploy2003: Finished scap sync-world: Backport for VisualEditor: Precompute edit suggestion counts on 5% of jawiki (T442767) (duration: 10m 49s)
22:38 kemayo@deploy2003: kemayo: Continuing with deployment
22:38 kemayo@deploy2003: kemayo: Backport for VisualEditor: Precompute edit suggestion counts on 5% of jawiki (T442767) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
22:33 kemayo@deploy2003: Started scap sync-world: Backport for VisualEditor: Precompute edit suggestion counts on 5% of jawiki (T442767)
22:27 jdlrobson@deploy2003: Finished scap sync-world: Backport for styles: Restore style to Special:MobileOptions (T363032) (1.48.0-wmf.3), Revert "footer: Hide separator before a hidden last footer place" (T387797) (1.48.0-wmf.3), downloadPageAction: Add the download link before the bookmark (T443387) (1.48.0-wmf.3) (duration: 13m 25s)
22:21 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
22:18 jdlrobson@deploy2003: jdlrobson: Backport for styles: Restore style to Special:MobileOptions (T363032) (1.48.0-wmf.3), Revert "footer: Hide separator before a hidden last footer place" (T387797) (1.48.0-wmf.3), downloadPageAction: Add the download link before the bookmark (T443387) (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki
22:14 jdlrobson@deploy2003: Started scap sync-world: Backport for styles: Restore style to Special:MobileOptions (T363032) (1.48.0-wmf.3), Revert "footer: Hide separator before a hidden last footer place" (T387797) (1.48.0-wmf.3), downloadPageAction: Add the download link before the bookmark (T443387) (1.48.0-wmf.3)
{{safesubst:SAL entry|1=21:42 ebernhardson@deploy2003: Finished scap sync-world: Backport for eswiki: add editprotected to botadmin (T443272), liwiktionary: Create Appendix namespace (T443269), Stop exempting item/property creation from hCaptcha (T440832), Remove WikibaseTmpMobileEditingUIBetaFeature config (T437054), [[gerrit:1362185|Revert "Reapply "cirrus: Send more_lik}}
21:42 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1274 (T441633)', diff saved to https://phabricator.wikimedia.org/P97176 and previous config saved to /var/cache/conftool/dbconfig/20261103-214206-ladsgroup.json
21:42 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1274.eqiad.wmnet with reason: Maintenance
21:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230 (T441633)', diff saved to https://phabricator.wikimedia.org/P97175 and previous config saved to /var/cache/conftool/dbconfig/20261103-214146-ladsgroup.json
21:36 ebernhardson@deploy2003: ebernhardson, ameisenigel, thiemowmde, mahmoud-abdelsattar: Continuing with deployment
21:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230', diff saved to https://phabricator.wikimedia.org/P97174 and previous config saved to /var/cache/conftool/dbconfig/20261103-213145-ladsgroup.json
{{safesubst:SAL entry|1=21:28 ebernhardson@deploy2003: ebernhardson, ameisenigel, thiemowmde, mahmoud-abdelsattar: Backport for eswiki: add editprotected to botadmin (T443272), liwiktionary: Create Appendix namespace (T443269), Stop exempting item/property creation from hCaptcha (T440832), Remove WikibaseTmpMobileEditingUIBetaFeature config (T437054), [[gerrit:1362185|Reve}}
{{safesubst:SAL entry|1=21:24 ebernhardson@deploy2003: Started scap sync-world: Backport for eswiki: add editprotected to botadmin (T443272), liwiktionary: Create Appendix namespace (T443269), Stop exempting item/property creation from hCaptcha (T440832), Remove WikibaseTmpMobileEditingUIBetaFeature config (T437054), [[gerrit:1362185|Revert "Reapply "cirrus: Send more_like}}
21:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230', diff saved to https://phabricator.wikimedia.org/P97173 and previous config saved to /var/cache/conftool/dbconfig/20261103-212143-ladsgroup.json
21:16 sbassett@deploy2003: Finished scap sync-world: Backport for Enable DBSC report-only mode on group0 and test2wiki (T393212) (duration: 11m 41s)
21:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230 (T441633)', diff saved to https://phabricator.wikimedia.org/P97172 and previous config saved to /var/cache/conftool/dbconfig/20261103-211141-ladsgroup.json
21:09 sbassett@deploy2003: sbassett: Continuing with deployment
21:09 sbassett@deploy2003: sbassett: Backport for Enable DBSC report-only mode on group0 and test2wiki (T393212) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
21:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1230 (T441633)', diff saved to https://phabricator.wikimedia.org/P97171 and previous config saved to /var/cache/conftool/dbconfig/20261103-210458-ladsgroup.json
21:05 sbassett@deploy2003: Started scap sync-world: Backport for Enable DBSC report-only mode on group0 and test2wiki (T393212)
21:04 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1230.eqiad.wmnet with reason: Maintenance
21:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207 (T441633)', diff saved to https://phabricator.wikimedia.org/P97170 and previous config saved to /var/cache/conftool/dbconfig/20261103-210435-ladsgroup.json
20:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P97169 and previous config saved to /var/cache/conftool/dbconfig/20261103-205434-ladsgroup.json
20:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P97168 and previous config saved to /var/cache/conftool/dbconfig/20261103-204432-ladsgroup.json
20:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207 (T441633)', diff saved to https://phabricator.wikimedia.org/P97167 and previous config saved to /var/cache/conftool/dbconfig/20261103-203430-ladsgroup.json
20:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1207 (T441633)', diff saved to https://phabricator.wikimedia.org/P97166 and previous config saved to /var/cache/conftool/dbconfig/20261103-202749-ladsgroup.json
20:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1207.eqiad.wmnet with reason: Maintenance
20:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200 (T441633)', diff saved to https://phabricator.wikimedia.org/P97165 and previous config saved to /var/cache/conftool/dbconfig/20261103-202739-ladsgroup.json
20:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P97164 and previous config saved to /var/cache/conftool/dbconfig/20261103-201737-ladsgroup.json
20:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P97163 and previous config saved to /var/cache/conftool/dbconfig/20261103-200735-ladsgroup.json
20:00 eevans@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
19:59 eevans@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
19:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200 (T441633)', diff saved to https://phabricator.wikimedia.org/P97162 and previous config saved to /var/cache/conftool/dbconfig/20261103-195734-ladsgroup.json
19:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1200 (T441633)', diff saved to https://phabricator.wikimedia.org/P97161 and previous config saved to /var/cache/conftool/dbconfig/20261103-195043-ladsgroup.json
19:50 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1200.eqiad.wmnet with reason: Maintenance
19:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185 (T441633)', diff saved to https://phabricator.wikimedia.org/P97160 and previous config saved to /var/cache/conftool/dbconfig/20261103-195023-ladsgroup.json
19:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P97159 and previous config saved to /var/cache/conftool/dbconfig/20261103-194021-ladsgroup.json
19:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P97158 and previous config saved to /var/cache/conftool/dbconfig/20261103-193019-ladsgroup.json
19:29 dancy@deploy2003: Installation of scap version "4.313.0" completed for 2 hosts
19:27 dancy@deploy2003: Installing scap version "4.313.0" for 2 host(s)
19:25 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
19:25 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
19:20 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185 (T441633)', diff saved to https://phabricator.wikimedia.org/P97157 and previous config saved to /var/cache/conftool/dbconfig/20261103-192017-ladsgroup.json
19:19 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
19:18 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
19:17 dancy@deploy2003: Installing scap version "4.313.0" for 1 host(s)
19:16 dancy@deploy2003: Installing scap version "4.313.0" for 1 host(s)
19:14 dancy@deploy2003: Installing scap version "4.313.0" for 2 host(s)
19:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1185 (T441633)', diff saved to https://phabricator.wikimedia.org/P97156 and previous config saved to /var/cache/conftool/dbconfig/20261103-191327-ladsgroup.json
19:13 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1185.eqiad.wmnet with reason: Maintenance
19:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T441633)', diff saved to https://phabricator.wikimedia.org/P97155 and previous config saved to /var/cache/conftool/dbconfig/20261103-191307-ladsgroup.json
19:11 dancy@deploy2003: Installing scap version "4.313.0" for 2 host(s)
19:09 ryankemper@cumin2003: START - Cookbook sre.wdqs.categories-reload reloading categories to wdqs2008.codfw.wmnet
19:03 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P97154 and previous config saved to /var/cache/conftool/dbconfig/20261103-190305-ladsgroup.json
18:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P97153 and previous config saved to /var/cache/conftool/dbconfig/20261103-185303-ladsgroup.json
18:51 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns7002.* [reason: [end] testing pdns-rec 5.4.9]
18:50 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
18:49 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
18:49 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=dns7002.* [reason: testing pdns-rec 5.4.9]
18:46 kamila@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy1003.eqiad.wmnet
18:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T441633)', diff saved to https://phabricator.wikimedia.org/P97152 and previous config saved to /var/cache/conftool/dbconfig/20261103-184301-ladsgroup.json
18:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1161 (T441633)', diff saved to https://phabricator.wikimedia.org/P97151 and previous config saved to /var/cache/conftool/dbconfig/20261103-183459-ladsgroup.json
18:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1161.eqiad.wmnet with reason: Maintenance
18:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159 (T441633)', diff saved to https://phabricator.wikimedia.org/P97150 and previous config saved to /var/cache/conftool/dbconfig/20261103-183439-ladsgroup.json
18:34 kamila@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy2003.codfw.wmnet
18:33 kamila@cumin1004: START - Cookbook sre.hosts.reboot-single for host deploy1003.eqiad.wmnet
18:31 cjming@deploy2003: Finished scap deploy-service test-kitchen: deploy v2.3.0 to TK production (duration: 02m 12s)
18:29 cjming@deploy2003: scap deploy-service test-kitchen: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying test-kitchen
18:29 cjming@deploy2003: Started scap deploy-service test-kitchen: deploy v2.3.0 to TK production
18:28 eevans@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply
18:28 eevans@deploy2003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply
18:28 eevans@deploy2003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply
18:28 eevans@deploy2003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply
18:27 eevans@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
18:27 eevans@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
18:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P97149 and previous config saved to /var/cache/conftool/dbconfig/20261103-182437-ladsgroup.json
18:23 kamila@cumin1004: START - Cookbook sre.hosts.reboot-single for host deploy2003.codfw.wmnet
18:22 akhatun@deploy2003: Finished scap sync-world: Backport for revision feature counts: increase timeout 20s to 30s (T443482) (duration: 10m 40s)
18:18 cjming@deploy2003: Finished scap deploy-service test-kitchen-next: deploy v2.3.0 to TK staging (duration: 01m 03s)
18:17 cjming@deploy2003: scap deploy-service test-kitchen-next: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying test-kitchen-next
18:17 cjming@deploy2003: Started scap deploy-service test-kitchen-next: deploy v2.3.0 to TK staging
18:16 akhatun@deploy2003: akhatun: Continuing with deployment
18:16 akhatun@deploy2003: akhatun: Backport for revision feature counts: increase timeout 20s to 30s (T443482) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
18:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P97148 and previous config saved to /var/cache/conftool/dbconfig/20261103-181435-ladsgroup.json
18:11 akhatun@deploy2003: Started scap sync-world: Backport for revision feature counts: increase timeout 20s to 30s (T443482)
18:05 tchanders@deploy2003: Finished scap sync-world: Backport for Enable link spam queue on Special:AbuseReview for alpha testing (T443432) (duration: 19m 00s)
18:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159 (T441633)', diff saved to https://phabricator.wikimedia.org/P97147 and previous config saved to /var/cache/conftool/dbconfig/20261103-180432-ladsgroup.json
17:59 tchanders@deploy2003: tchanders: Continuing with deployment
17:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1159 (T441633)', diff saved to https://phabricator.wikimedia.org/P97146 and previous config saved to /var/cache/conftool/dbconfig/20261103-175629-ladsgroup.json
17:56 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1159.eqiad.wmnet with reason: Maintenance
17:50 tchanders@deploy2003: tchanders: Backport for Enable link spam queue on Special:AbuseReview for alpha testing (T443432) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
17:46 tchanders@deploy2003: Started scap sync-world: Backport for Enable link spam queue on Special:AbuseReview for alpha testing (T443432)
17:03 kamila@deploy2003: Finished scap deploy-service shellbox: bump also shellbox score to php 8.5 (duration: 04m 51s)
17:00 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
16:59 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
16:58 kamila@deploy2003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
16:58 kamila@deploy2003: Started scap deploy-service shellbox: bump also shellbox score to php 8.5
16:40 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netbox-dev2003.codfw.wmnet
16:39 otto@deploy2003: Finished scap deploy-service eventgate-all: v1.38.0 for T440968 (duration: 14m 43s)
16:36 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netbox-dev2003.codfw.wmnet
16:36 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/DEFAULT] codfw Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
16:28 dancy@deploy2003: Installation of scap version "4.312.0" completed for 2 hosts
16:27 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/DEFAULT] eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
16:27 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/CANARY] codfw Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
16:26 dancy@deploy2003: Installing scap version "4.312.0" for 2 host(s)
16:26 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/CANARY] eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
16:24 otto@deploy2003: scap deploy-service eventgate-all: [STAGING/DEFAULT] staging-eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
16:24 otto@deploy2003: Started scap deploy-service eventgate-all: v1.38.0 for T440968
16:16 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{ml-serve101[2-5]*} and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad)
16:16 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1015.eqiad.wmnet
16:16 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1015.eqiad.wmnet
16:13 ozge@deploy2003: helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply
16:11 ozge@deploy2003: helmfile [codfw] START helmfile.d/services/linkrecommendation: apply
16:10 ozge@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply
16:10 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1015.eqiad.wmnet
16:08 ozge@deploy2003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply
16:06 ozge@deploy2003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply
16:05 ozge@deploy2003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply
16:05 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1015.eqiad.wmnet
16:05 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1014.eqiad.wmnet
16:05 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1014.eqiad.wmnet
15:53 otto@deploy2003: Finished scap deploy-service eventgate-analytics: v1.38.0 for T440968 (duration: 11m 24s)
15:53 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/DEFAULT] codfw Deploying eventgate-analytics
15:46 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2003-2005].codfw.wmnet
15:46 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2003-2005].codfw.wmnet
15:44 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/DEFAULT] eqiad Deploying eventgate-analytics
15:44 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/CANARY] codfw Deploying eventgate-analytics
15:44 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/CANARY] eqiad Deploying eventgate-analytics
15:42 otto@deploy2003: scap deploy-service eventgate-analytics: [STAGING/DEFAULT] staging-eqiad Deploying eventgate-analytics
15:42 otto@deploy2003: Started scap deploy-service eventgate-analytics: v1.38.0 for T440968
15:40 dancy@deploy2003: Installation of scap version "4.311.0" completed for 155 hosts
15:39 jayme@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=99) Upgrade apiservers of staging-codfw to 1.31.14
15:36 dancy@deploy2003: Installing scap version "4.311.0" for 155 host(s)
15:36 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
15:36 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
15:35 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.31.14
15:35 samtar@deploy2003: Finished scap sync-world: Backport for Squashed diff to master (1.48.0-wmf.1, 1.48.0-wmf.3), Revert "Enable NewUserMessage for viwikivoyage" (duration: 33m 48s)
15:32 jelto@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-kubelets-inplace (exit_code=1) Upgrade kubelet of staging-codfw to 1.34.12
15:23 samtar@deploy2003: samtar, mlitn: Continuing with deployment
15:23 samtar@deploy2003: samtar, mlitn: Backport for Squashed diff to master (1.48.0-wmf.1, 1.48.0-wmf.3), Revert "Enable NewUserMessage for viwikivoyage" synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
15:20 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
15:19 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
15:16 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1014.eqiad.wmnet
15:12 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for es2035.codfw.wmnet
15:12 cwilliams@cumin1004: START - Cookbook sre.hosts.remove-downtime for es2035.codfw.wmnet
15:10 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2036.codfw.wmnet with reason: Maintenance
15:09 slyngshede@cumin1004: END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) pool all active/active services in eqiad: Datacenter services switchover - T439676
15:07 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2035.codfw.wmnet with reason: Maintenance
15:06 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1014.eqiad.wmnet
15:06 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1013.eqiad.wmnet
15:06 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1013.eqiad.wmnet
15:02 samtar@deploy2003: Started scap sync-world: Backport for Squashed diff to master (1.48.0-wmf.1, 1.48.0-wmf.3), Revert "Enable NewUserMessage for viwikivoyage"
15:00 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1013.eqiad.wmnet
14:59 samtar@deploy2003: Finished scap sync-world: Backport for Add Draft/Draft_talk namespace aliases on wikis using ns 118 as draft namespace (T440179), Add missing groupOverrides for sysop and bot on ltwiki. (T440216) (duration: 21m 43s)
14:00 jelto@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet
14:00 jelto@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet
13:59 jelto@cumin1004: START - Cookbook sre.k8s.upgrade-kubelets-inplace Upgrade kubelet of staging-codfw to 1.34.12
13:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1271', diff saved to https://phabricator.wikimedia.org/P97137 and previous config saved to /var/cache/conftool/dbconfig/20261103-135521-ladsgroup.json
13:47 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
13:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1271 (T441633)', diff saved to https://phabricator.wikimedia.org/P97136 and previous config saved to /var/cache/conftool/dbconfig/20261103-134519-ladsgroup.json
13:42 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=0) Upgrade apiservers of staging-codfw to 1.34.12
13:42 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2004-2005].codfw.wmnet
13:42 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2004-2005].codfw.wmnet
13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1271 (T441633)', diff saved to https://phabricator.wikimedia.org/P97135 and previous config saved to /var/cache/conftool/dbconfig/20261103-134118-ladsgroup.json
13:41 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 7 hosts with reason: Maintenance
13:40 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1271.eqiad.wmnet with reason: Maintenance
13:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259 (T441633)', diff saved to https://phabricator.wikimedia.org/P97134 and previous config saved to /var/cache/conftool/dbconfig/20261103-134036-ladsgroup.json
13:34 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
13:34 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
13:34 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.12
13:34 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2004-2005].codfw.wmnet
13:34 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2004-2005].codfw.wmnet
13:33 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
13:33 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
13:31 jayme@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=99) Upgrade apiservers of staging-codfw to 1.34.12
13:31 jayme@cumin1004: END (ERROR) - Cookbook sre.k8s.pool-depool-node (exit_code=97) depool for host kubestagemaster[2004-2005].codfw.wmnet
13:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P97133 and previous config saved to /var/cache/conftool/dbconfig/20261103-133034-ladsgroup.json
13:29 jnuche@deploy2003: Started scap sync-world: testwikis to 1.48.0-wmf.3 refs T442221
13:28 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
13:27 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
13:27 kamila@deploy2003: Finished scap deploy-service shellbox: update to 8.5 + new image (duration: 05m 00s)
13:27 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
13:27 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
13:25 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
13:23 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
13:22 kamila@deploy2003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
13:22 kamila@deploy2003: Started scap deploy-service shellbox: update to 8.5 + new image
13:20 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
13:20 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.12
13:20 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P97132 and previous config saved to /var/cache/conftool/dbconfig/20261103-132031-ladsgroup.json
13:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259 (T441633)', diff saved to https://phabricator.wikimedia.org/P97131 and previous config saved to /var/cache/conftool/dbconfig/20261103-131029-ladsgroup.json
13:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1259 (T441633)', diff saved to https://phabricator.wikimedia.org/P97130 and previous config saved to /var/cache/conftool/dbconfig/20261103-130419-ladsgroup.json
13:04 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1259.eqiad.wmnet with reason: Maintenance
13:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254 (T441633)', diff saved to https://phabricator.wikimedia.org/P97129 and previous config saved to /var/cache/conftool/dbconfig/20261103-130409-ladsgroup.json
12:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P97128 and previous config saved to /var/cache/conftool/dbconfig/20261103-125407-ladsgroup.json
12:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P97127 and previous config saved to /var/cache/conftool/dbconfig/20261103-124405-ladsgroup.json
12:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254 (T441633)', diff saved to https://phabricator.wikimedia.org/P97126 and previous config saved to /var/cache/conftool/dbconfig/20261103-123403-ladsgroup.json
12:32 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc1003.wikimedia.org
12:31 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1006.eqiad.wmnet with OS trixie
12:28 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host irc1003.wikimedia.org
12:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1254 (T441633)', diff saved to https://phabricator.wikimedia.org/P97125 and previous config saved to /var/cache/conftool/dbconfig/20261103-122803-ladsgroup.json
12:28 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1254.eqiad.wmnet with reason: Maintenance
12:22 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1239.eqiad.wmnet with reason: Maintenance
12:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233 (T441633)', diff saved to https://phabricator.wikimedia.org/P97124 and previous config saved to /var/cache/conftool/dbconfig/20261103-122223-ladsgroup.json
12:12 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
12:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P97123 and previous config saved to /var/cache/conftool/dbconfig/20261103-121221-ladsgroup.json
12:07 cmooney@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
12:05 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
12:05 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
12:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P97122 and previous config saved to /var/cache/conftool/dbconfig/20261103-120219-ladsgroup.json
11:57 ladsgroup@deploy2003: Finished scap sync-world: Backport for Add --listfile to transcode maint scripts (T435744 T373546) (1.48.0-wmf.1) (duration: 10m 58s)
11:56 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) sretest1006.eqiad.wmnet on all recursors
11:56 cmooney@cumin1004: START - Cookbook sre.dns.wipe-cache sretest1006.eqiad.wmnet on all recursors
11:56 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
11:56 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004"
11:56 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004"
11:52 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233 (T441633)', diff saved to https://phabricator.wikimedia.org/P97121 and previous config saved to /var/cache/conftool/dbconfig/20261103-115217-ladsgroup.json
11:51 cmooney@cumin1004: START - Cookbook sre.dns.netbox
11:51 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
11:50 ladsgroup@deploy2003: ladsgroup: Backport for Add --listfile to transcode maint scripts (T435744 T373546) (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
11:50 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
11:50 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
11:49 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
11:46 ladsgroup@deploy2003: Started scap sync-world: Backport for Add --listfile to transcode maint scripts (T435744 T373546) (1.48.0-wmf.1)
11:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1233 (T441633)', diff saved to https://phabricator.wikimedia.org/P97120 and previous config saved to /var/cache/conftool/dbconfig/20261103-114609-ladsgroup.json
11:46 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1233.eqiad.wmnet with reason: Maintenance
11:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229 (T441633)', diff saved to https://phabricator.wikimedia.org/P97119 and previous config saved to /var/cache/conftool/dbconfig/20261103-114548-ladsgroup.json
11:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P97118 and previous config saved to /var/cache/conftool/dbconfig/20261103-113546-ladsgroup.json
11:30 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d6-eqiad
11:29 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-d6-eqiad
11:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P97117 and previous config saved to /var/cache/conftool/dbconfig/20261103-112544-ladsgroup.json
11:21 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1228: Repooling
11:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229 (T441633)', diff saved to https://phabricator.wikimedia.org/P97115 and previous config saved to /var/cache/conftool/dbconfig/20261103-111542-ladsgroup.json
11:12 moritzm: installing chromium security updates
11:09 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1229 (T441633)', diff saved to https://phabricator.wikimedia.org/P97114 and previous config saved to /var/cache/conftool/dbconfig/20261103-110944-ladsgroup.json
11:09 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1229.eqiad.wmnet with reason: Maintenance
11:04 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1225.eqiad.wmnet with reason: Maintenance
11:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222 (T441633)', diff saved to https://phabricator.wikimedia.org/P97112 and previous config saved to /var/cache/conftool/dbconfig/20261103-110410-ladsgroup.json
10:54 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
10:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P97111 and previous config saved to /var/cache/conftool/dbconfig/20261103-105408-ladsgroup.json
10:53 jmm@dns1004: END - running authdns-update
10:53 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
10:51 jmm@dns1004: START - running authdns-update
10:46 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
10:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P97109 and previous config saved to /var/cache/conftool/dbconfig/20261103-104406-ladsgroup.json
10:43 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
10:42 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
10:42 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
10:36 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1228: Repooling
10:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222 (T441633)', diff saved to https://phabricator.wikimedia.org/P97107 and previous config saved to /var/cache/conftool/dbconfig/20261103-103404-ladsgroup.json
10:33 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
10:33 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
10:31 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
10:31 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=eswiki --olderThan=1790175600 --verbose # T438366
10:31 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
10:30 urbanecm@deploy2003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist GrowthExperiments:revalidateLinkRecommendations.php --olderThan=1790175600 --verbose # T438366 (dblist: https://phabricator.wikimedia.org/P97106)
10:30 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
10:30 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
10:29 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
10:29 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
10:27 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
10:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1222 (T441633)', diff saved to https://phabricator.wikimedia.org/P97105 and previous config saved to /var/cache/conftool/dbconfig/20261103-102742-ladsgroup.json
10:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1222.eqiad.wmnet with reason: Maintenance
10:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197 (T441633)', diff saved to https://phabricator.wikimedia.org/P97104 and previous config saved to /var/cache/conftool/dbconfig/20261103-102721-ladsgroup.json
10:23 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
10:23 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=itwiki --olderThan=1790175600 --verbose # T438366
10:23 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
10:23 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=frwiki --olderThan=1790175600 --verbose # T438366
10:23 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
10:23 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=enwiki --olderThan=1790175600 --verbose # T438366
10:22 moritzm: installing git-lfs security updates
10:22 urbanecm@deploy2003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist GrowthExperiments:revalidateLinkRecommendations.php --olderThan=1790175600 --verbose # T438366 (dblist: https://phabricator.wikimedia.org/P97103)
10:22 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
10:22 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
10:22 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
10:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P97102 and previous config saved to /var/cache/conftool/dbconfig/20261103-101719-ladsgroup.json
10:15 moritzm: installing imagemagick security updates
10:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P97101 and previous config saved to /var/cache/conftool/dbconfig/20261103-100717-ladsgroup.json
10:04 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
10:00 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
09:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197 (T441633)', diff saved to https://phabricator.wikimedia.org/P97099 and previous config saved to /var/cache/conftool/dbconfig/20261103-095715-ladsgroup.json
09:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1197 (T441633)', diff saved to https://phabricator.wikimedia.org/P97098 and previous config saved to /var/cache/conftool/dbconfig/20261103-094843-ladsgroup.json
09:48 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1197.eqiad.wmnet with reason: Maintenance
09:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188 (T441633)', diff saved to https://phabricator.wikimedia.org/P97097 and previous config saved to /var/cache/conftool/dbconfig/20261103-094823-ladsgroup.json
09:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P97096 and previous config saved to /var/cache/conftool/dbconfig/20261103-093820-ladsgroup.json
09:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P97095 and previous config saved to /var/cache/conftool/dbconfig/20261103-092818-ladsgroup.json
09:21 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
09:18 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188 (T441633)', diff saved to https://phabricator.wikimedia.org/P97094 and previous config saved to /var/cache/conftool/dbconfig/20261103-091816-ladsgroup.json
09:12 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2003.wikimedia.org
09:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1188 (T441633)', diff saved to https://phabricator.wikimedia.org/P97093 and previous config saved to /var/cache/conftool/dbconfig/20261103-091203-ladsgroup.json
09:12 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1188.eqiad.wmnet with reason: Maintenance
09:08 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host urldownloader2003.wikimedia.org
09:03 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2004.wikimedia.org
09:02 Tran: Deployed patch for T441675
08:58 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host urldownloader2004.wikimedia.org
08:50 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
08:50 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
08:24 denisse: increasing disk size in arclamp1001
08:14 denisse: deleting stale .tmp logs from 2025 in arclamp2001
08:13 denisse: incresing disk in arclamp2001
05:02 mwpresync@deploy2003: Pruned MediaWiki: 1.47.0-wmf.23 (duration: 02m 26s)
04:40 krinkle@deploy2003: Finished scap sync-world: Backport for MathML+MathJax rollout to phase 6 (remaining large Wikipedias) (T271001) (duration: 24m 34s)
04:33 krinkle@deploy2003: krinkle: Continuing with deployment
04:24 krinkle@deploy2003: krinkle: Backport for MathML+MathJax rollout to phase 6 (remaining large Wikipedias) (T271001) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
04:16 krinkle@deploy2003: Started scap sync-world: Backport for MathML+MathJax rollout to phase 6 (remaining large Wikipedias) (T271001)
01:30 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply
01:30 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/zotero: apply
01:30 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
01:29 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
01:29 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
01:28 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
01:28 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply
01:28 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply
01:27 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply
01:27 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/toolhub: apply
01:27 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply
01:26 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/termbox: apply
01:26 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
01:25 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
01:25 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply
01:25 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply
01:25 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply
01:24 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply
01:24 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply
01:24 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply
01:23 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply
01:23 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply
01:23 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply
01:22 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox: apply
01:22 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/sessionstore: apply
01:22 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/sessionstore: apply
01:21 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/recommendation-api: apply
01:21 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/recommendation-api: apply
01:20 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/push-notifications: apply
01:20 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/push-notifications: apply
01:20 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/proton: apply
01:18 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/proton: apply
01:18 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/page-analytics: apply
01:18 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/page-analytics: apply
01:17 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
01:16 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
01:16 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
01:13 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
01:13 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/media-analytics: apply
01:13 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/media-analytics: apply
01:13 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/machinetranslation: apply
01:05 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/machinetranslation: apply
01:05 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply
01:04 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply
01:03 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply
01:03 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply
01:03 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply
01:03 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply
01:03 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/kartotherian: apply
01:01 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/kartotherian: apply
01:01 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/geo-analytics: apply
01:01 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/geo-analytics: apply
01:00 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply
01:00 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply
00:57 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: apply
00:56 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/eventgate-analytics: apply
00:55 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/editor-analytics: apply
00:55 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/editor-analytics: apply
00:55 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/editcheck-headless: apply
00:54 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/editcheck-headless: apply
00:54 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/edit-analytics: apply
00:53 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/edit-analytics: apply
00:53 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/echostore: apply
00:53 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/echostore: apply
00:51 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/device-analytics: apply
00:51 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/device-analytics: apply
00:50 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
00:50 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
00:49 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/data-gateway: apply
00:49 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/data-gateway: apply
00:49 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply
00:48 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/cxserver: apply
00:47 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/commons-impact-analytics: apply
00:47 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/commons-impact-analytics: apply
00:47 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/citoid: apply
00:46 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/citoid: apply
00:46 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
00:46 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
00:46 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/chart-renderer: apply
00:46 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/chart-renderer: apply
00:41 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/apertium: apply
00:41 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/apertium: apply
00:10 maryum: Deployed security fix for T440469, T441384, T440735, T440501, T439716

2026-11-02

23:24 maryum: Deployed security fix for T440502, T440330, T442830, T442861
22:35 cscott@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
22:35 cscott@deploy2003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
22:25 krinkle@deploy2003: Finished scap sync-world: Backport for Handle \mathcal in mmlFilter instead of the font (1.48.0-wmf.1), Keep \mathrm upright inside \mathbf (T442312) (1.48.0-wmf.1), Draw no outer line for a nested \hline (1.48.0-wmf.1) (duration: 13m 50s)
22:19 krinkle@deploy2003: krinkle: Continuing with deployment
22:15 krinkle@deploy2003: krinkle: Backport for Handle \mathcal in mmlFilter instead of the font (1.48.0-wmf.1), Keep \mathrm upright inside \mathbf (T442312) (1.48.0-wmf.1), Draw no outer line for a nested \hline (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
22:11 krinkle@deploy2003: Started scap sync-world: Backport for Handle \mathcal in mmlFilter instead of the font (1.48.0-wmf.1), Keep \mathrm upright inside \mathbf (T442312) (1.48.0-wmf.1), Draw no outer line for a nested \hline (1.48.0-wmf.1)
22:05 jdlrobson@deploy2003: Finished scap sync-world: Backport for Enable Vector CTA on all wikipedias (T441705) (duration: 13m 05s)
21:57 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
21:56 jdlrobson@deploy2003: jdlrobson: Backport for Enable Vector CTA on all wikipedias (T441705) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
21:52 jdlrobson@deploy2003: Started scap sync-world: Backport for Enable Vector CTA on all wikipedias (T441705)
21:50 jdlrobson@deploy2003: Finished scap sync-world: Backport for Skin: Update skin-responsive logic (T439273) (1.48.0-wmf.1), Skin: skin--responsive class no longer varies on user (T439273) (1.48.0-wmf.1), Fast forward wmf.1 branch to 0626e6 (T440916 T441259 T441260 T441477 T442906) (1.48.0-wmf.1) (duration: 35m 12s)
21:38 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
21:36 jdlrobson@deploy2003: jdlrobson: Backport for Skin: Update skin-responsive logic (T439273) (1.48.0-wmf.1), Skin: skin--responsive class no longer varies on user (T439273) (1.48.0-wmf.1), Fast forward wmf.1 branch to 0626e6 (T440916 T441259 T441260 T441477 T442906) (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes
21:15 jdlrobson@deploy2003: Started scap sync-world: Backport for Skin: Update skin-responsive logic (T439273) (1.48.0-wmf.1), Skin: skin--responsive class no longer varies on user (T439273) (1.48.0-wmf.1), Fast forward wmf.1 branch to 0626e6 (T440916 T441259 T441260 T441477 T442906) (1.48.0-wmf.1)
20:43 cscott@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
20:43 cscott@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
20:34 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
20:33 rzl@deploy2003: helmfile [staging] START helmfile.d/services/miscweb: apply
20:02 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/zotero: apply
20:01 rzl@deploy2003: helmfile [staging] START helmfile.d/services/zotero: apply
20:01 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
20:00 rzl@deploy2003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
20:00 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
20:00 rzl@deploy2003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
20:00 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply
19:59 rzl@deploy2003: helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply
19:59 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/toolhub: apply
19:58 rzl@deploy2003: helmfile [staging] START helmfile.d/services/toolhub: apply
19:58 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
19:58 rzl@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
19:58 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/termbox: apply
19:58 rzl@deploy2003: helmfile [staging] START helmfile.d/services/termbox: apply
19:57 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply
19:57 rzl@deploy2003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply
19:57 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply
19:57 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-video: apply
19:56 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply
19:56 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply
19:55 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply
19:54 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-media: apply
19:54 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply
19:54 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply
19:54 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: apply
19:54 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: apply
19:54 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/sessionstore: apply
19:53 rzl@deploy2003: helmfile [staging] START helmfile.d/services/sessionstore: apply
19:53 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
19:53 rzl@deploy2003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
19:53 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/recommendation-api: apply
19:53 rzl@deploy2003: helmfile [staging] START helmfile.d/services/recommendation-api: apply
19:52 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/push-notifications: apply
19:52 rzl@deploy2003: helmfile [staging] START helmfile.d/services/push-notifications: apply
19:52 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/proton: apply
19:52 rzl@deploy2003: helmfile [staging] START helmfile.d/services/proton: apply
19:52 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/page-analytics: apply
19:51 rzl@deploy2003: helmfile [staging] START helmfile.d/services/page-analytics: apply
19:51 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
19:50 rzl@deploy2003: helmfile [staging] START helmfile.d/services/mobileapps: apply
19:45 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/media-analytics: apply
19:45 rzl@deploy2003: helmfile [staging] START helmfile.d/services/media-analytics: apply
19:45 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/machinetranslation: apply
19:41 rzl@deploy2003: helmfile [staging] START helmfile.d/services/machinetranslation: apply
19:41 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply
19:41 rzl@deploy2003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply
19:41 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
19:40 rzl@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
19:40 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/lac-lambdas: apply
19:40 rzl@deploy2003: helmfile [staging] START helmfile.d/services/lac-lambdas: apply
19:40 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/kartotherian: apply
19:39 rzl@deploy2003: helmfile [staging] START helmfile.d/services/kartotherian: apply
19:39 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/geo-analytics: apply
19:39 rzl@deploy2003: helmfile [staging] START helmfile.d/services/geo-analytics: apply
19:39 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply
19:39 rzl@deploy2003: helmfile [staging] START helmfile.d/services/eventstreams: apply
19:38 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/eventgate-analytics: apply
19:37 rzl@deploy2003: helmfile [staging] START helmfile.d/services/eventgate-analytics: apply
19:37 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/editor-analytics: apply
19:37 rzl@deploy2003: helmfile [staging] START helmfile.d/services/editor-analytics: apply
19:37 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply
19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/editcheck-headless: apply
19:36 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/edit-analytics: apply
19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/edit-analytics: apply
19:36 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/echostore: apply
19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/echostore: apply
19:36 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/device-analytics: apply
19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/device-analytics: apply
19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
19:35 rzl@deploy2003: helmfile [staging] START helmfile.d/services/developer-portal: apply
19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/data-gateway: apply
19:35 rzl@deploy2003: helmfile [staging] START helmfile.d/services/data-gateway: apply
19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/cxserver: apply
19:35 rzl@deploy2003: helmfile [staging] START helmfile.d/services/cxserver: apply
19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/commons-impact-analytics: apply
19:34 rzl@deploy2003: helmfile [staging] START helmfile.d/services/commons-impact-analytics: apply
19:34 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/citoid: apply
19:34 rzl@deploy2003: helmfile [staging] START helmfile.d/services/citoid: apply
19:34 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply
19:34 rzl@deploy2003: helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply
19:33 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/chart-renderer: apply
19:33 rzl@deploy2003: helmfile [staging] START helmfile.d/services/chart-renderer: apply
19:30 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/apertium: apply
19:29 rzl@deploy2003: helmfile [staging] START helmfile.d/services/apertium: apply
18:57 dancy@deploy2003: Installation of scap version "4.310.0" completed for 2 hosts
18:55 dancy@deploy2003: Installing scap version "4.310.0" for 2 host(s)
18:54 akhatun@deploy2003: Finished scap sync-world: Backport for Prefetch the LAC revision_feature_counts artifact on all wikipedias (duration: 10m 43s)
18:48 akhatun@deploy2003: akhatun: Continuing with deployment
18:47 akhatun@deploy2003: akhatun: Backport for Prefetch the LAC revision_feature_counts artifact on all wikipedias synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
18:43 akhatun@deploy2003: Started scap sync-world: Backport for Prefetch the LAC revision_feature_counts artifact on all wikipedias
18:37 rsilvola@deploy2003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
18:36 rsilvola@deploy2003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
18:36 rsilvola@deploy2003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
18:36 rsilvola@deploy2003: helmfile [codfw] START helmfile.d/services/miscweb: apply
18:31 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-videoscaler: apply
18:31 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/mw-videoscaler: apply
18:30 btullis@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on an-presto1015.eqiad.wmnet with reason: Recreate data RAID without failed disk
18:29 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-videoscaler: apply
18:29 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-videoscaler: apply
18:25 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
18:24 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
18:24 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
18:23 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
18:18 rzl@deploy2003: Finished scap sync-world: https://gerrit.wikimedia.org/r/1360005 T436338 (duration: 07m 48s)
18:11 rzl@deploy2003: Started scap sync-world: https://gerrit.wikimedia.org/r/1360005 T436338
18:03 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply
18:03 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply
18:03 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
18:02 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
18:02 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply
18:02 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply
17:59 ladsgroup@deploy2003: Finished scap sync-world: Backport for Adjust circuit breaking thresholds (T443274), rdbms: Allow overriding of load monitor per lb cluster (T443274) (1.48.0-wmf.1) (duration: 11m 51s)
17:54 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply
17:53 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
17:53 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply
17:53 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
17:52 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
17:51 ladsgroup@deploy2003: ladsgroup: Backport for Adjust circuit breaking thresholds (T443274), rdbms: Allow overriding of load monitor per lb cluster (T443274) (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
17:50 otto@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventgate-analytics: apply
17:49 otto@deploy2003: helmfile [codfw] START helmfile.d/services/eventgate-analytics: apply
17:49 otto@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: apply
17:48 otto@deploy2003: helmfile [eqiad] START helmfile.d/services/eventgate-analytics: apply
17:47 ladsgroup@deploy2003: Started scap sync-world: Backport for Adjust circuit breaking thresholds (T443274), rdbms: Allow overriding of load monitor per lb cluster (T443274) (1.48.0-wmf.1)
17:46 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply
17:46 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply
17:45 dancy: Investigating spiderpig
17:41 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
17:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289 (T441633)', diff saved to https://phabricator.wikimedia.org/P97090 and previous config saved to /var/cache/conftool/dbconfig/20261102-174043-ladsgroup.json
17:32 otto@deploy2003: scap deploy-service eventgate-analytics: [STAGING/DEFAULT] staging-eqiad Deploying eventgate-analytics
17:32 otto@deploy2003: Started scap deploy-service eventgate-analytics: Time to persist metric - T440968
17:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289', diff saved to https://phabricator.wikimedia.org/P97089 and previous config saved to /var/cache/conftool/dbconfig/20261102-173041-ladsgroup.json
17:20 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289', diff saved to https://phabricator.wikimedia.org/P97088 and previous config saved to /var/cache/conftool/dbconfig/20261102-172039-ladsgroup.json
17:12 ladsgroup@deploy2003: Finished scap sync-world: Backport for Temporarily increase circuit breaking (T443274) (duration: 13m 39s)
17:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289 (T441633)', diff saved to https://phabricator.wikimedia.org/P97087 and previous config saved to /var/cache/conftool/dbconfig/20261102-171037-ladsgroup.json
17:07 cwilliams@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1228: Maintenance
17:07 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1228: Maintenance
17:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1289 (T441633)', diff saved to https://phabricator.wikimedia.org/P97086 and previous config saved to /var/cache/conftool/dbconfig/20261102-170629-ladsgroup.json
17:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1289.eqiad.wmnet with reason: Maintenance
17:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280 (T441633)', diff saved to https://phabricator.wikimedia.org/P97085 and previous config saved to /var/cache/conftool/dbconfig/20261102-170609-ladsgroup.json
17:06 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
17:03 ladsgroup@deploy2003: ladsgroup: Backport for Temporarily increase circuit breaking (T443274) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
16:59 ladsgroup@deploy2003: Started scap sync-world: Backport for Temporarily increase circuit breaking (T443274)
16:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280', diff saved to https://phabricator.wikimedia.org/P97084 and previous config saved to /var/cache/conftool/dbconfig/20261102-165607-ladsgroup.json
16:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280', diff saved to https://phabricator.wikimedia.org/P97083 and previous config saved to /var/cache/conftool/dbconfig/20261102-164605-ladsgroup.json
16:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280 (T441633)', diff saved to https://phabricator.wikimedia.org/P97082 and previous config saved to /var/cache/conftool/dbconfig/20261102-163602-ladsgroup.json
16:32 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1253.eqiad.wmnet
16:32 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1253.eqiad.wmnet
16:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1280 (T441633)', diff saved to https://phabricator.wikimedia.org/P97081 and previous config saved to /var/cache/conftool/dbconfig/20261102-163154-ladsgroup.json
16:31 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1280.eqiad.wmnet with reason: Maintenance
16:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272 (T441633)', diff saved to https://phabricator.wikimedia.org/P97080 and previous config saved to /var/cache/conftool/dbconfig/20261102-163134-ladsgroup.json
16:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272', diff saved to https://phabricator.wikimedia.org/P97079 and previous config saved to /var/cache/conftool/dbconfig/20261102-162132-ladsgroup.json
16:19 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1253.eqiad.wmnet
16:18 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1253.eqiad.wmnet
16:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272', diff saved to https://phabricator.wikimedia.org/P97078 and previous config saved to /var/cache/conftool/dbconfig/20261102-161129-ladsgroup.json
16:04 volans@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2036.codfw.wmnet with reason: host crashed
16:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272 (T441633)', diff saved to https://phabricator.wikimedia.org/P97077 and previous config saved to /var/cache/conftool/dbconfig/20261102-160127-ladsgroup.json
15:57 tchin@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply
15:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1272 (T441633)', diff saved to https://phabricator.wikimedia.org/P97076 and previous config saved to /var/cache/conftool/dbconfig/20261102-155719-ladsgroup.json
15:57 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1272.eqiad.wmnet with reason: Maintenance
15:57 tchin@deploy2003: helmfile [codfw] START helmfile.d/services/eventstreams: apply
15:56 tchin@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply
15:56 tchin@deploy2003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply
15:54 tchin@deploy2003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply
15:54 tchin@deploy2003: helmfile [staging] START helmfile.d/serv
[Page truncated to keep GeezerVison snappy.]