1-50 of 10000 results (18ms)
2026-03-30 §
08:34 <jmm@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host bast4006.wikimedia.org with OS trixie [production]
08:17 <javiermonton@deploy1003> Finished scap sync-world: Backport for [[gerrit:1261377|stream: mediawiki.page_html_content_change (T421341)]] (duration: 35m 10s) [production]
08:03 <javiermonton@deploy1003> javiermonton: Continuing with sync [production]
08:00 <javiermonton@deploy1003> javiermonton: Backport for [[gerrit:1261377|stream: mediawiki.page_html_content_change (T421341)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
07:54 <godog> deploy rabbitmq changes to allow cli communication - T420923 [production]
07:48 <jmm@cumin2002> START - Cookbook sre.hosts.reimage for host bast4006.wikimedia.org with OS trixie [production]
07:48 <jmm@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host bast4006.wikimedia.org with OS trixie [production]
07:48 <jmm@cumin2002> START - Cookbook sre.hosts.reimage for host bast4006.wikimedia.org with OS trixie [production]
07:45 <jmm@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host bast4006.wikimedia.org with OS trixie [production]
07:45 <jmm@cumin2002> START - Cookbook sre.hosts.reimage for host bast4006.wikimedia.org with OS trixie [production]
07:42 <javiermonton@deploy1003> Started scap sync-world: Backport for [[gerrit:1261377|stream: mediawiki.page_html_content_change (T421341)]] [production]
07:38 <jmm@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host bast4006.wikimedia.org with OS trixie [production]
07:24 <tappof> prometheus7002: switch to nftables and reboot (T419960) [production]
07:18 <tappof> prometheus6002: switch to nftables and reboot (T419960) [production]
07:11 <tappof> prometheus5002: switch to nftables and reboot (T419960) [production]
07:08 <tappof> prometheus4003: reboot (T419960) [production]
07:05 <jmm@cumin2002> START - Cookbook sre.hosts.reimage for host bast4006.wikimedia.org with OS trixie [production]
07:04 <tappof> prometheus3004: switch to nftables and reboot (T419960) [production]
05:16 <marostegui@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb1022.eqiad.wmnet with reason: Downgrade clouddb1022 to 10.11.13 [production]
05:14 <marostegui@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on clouddb1022.eqiad.wmnet with reason: Downgrade clouddb1022 to 10.11.13 [production]
02:07 <mwpresync@deploy1003> Finished scap build-images: Publishing wmf/next image (duration: 06m 50s) [production]
02:00 <mwpresync@deploy1003> Started scap build-images: Publishing wmf/next image [production]
2026-03-29 §
02:07 <mwpresync@deploy1003> Finished scap build-images: Publishing wmf/next image (duration: 06m 13s) [production]
02:00 <mwpresync@deploy1003> Started scap build-images: Publishing wmf/next image [production]
2026-03-28 §
14:48 <dzahn@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on zuul2002.codfw.wmnet with reason: T421398 [production]
14:48 <dzahn@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on zuul1002.eqiad.wmnet with reason: T421398 [production]
14:30 <wmftkbot> Test Kitchen edge-unique experiments (poll 38152) - adds: none; removes: share-highlight-baseline; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [analytics]
14:16 <mutante> releases1003 - re-enabled puppet which was disabled due to T418109 but should not have been disabled during switch of the deployment server; leading to T421532 [production]
02:57 <ejegg> payments-wiki upgraded from d79a98b5 to b239a6b7 [fundraising]
02:27 <eileen> civicrm upgraded from 7138d524 to 58426b1e [fundraising]
2026-03-27 §
22:58 <wmbot~jeanfred@tools-bastion-15> Deploy 2bad8b3 (Catch ModuleNotFoundError when unpickling stale cache entries) [tools.integraality]
22:37 <eileen> civicrm upgraded from c51e98cc to 7138d524 [fundraising]
21:00 <James_F> Docker: [quibble-bullseye] Drop Python 2 from images [releng]
18:11 <jhathaway@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-ctrl2004.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [production]
18:00 <jhathaway@cumin2002> START - Cookbook sre.hosts.provision for host wikikube-ctrl2004.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [production]
17:50 <dwisehaupt> Corrected value: on frdb1005 running the following in mysql to up the buffer pool to ~412G: set global innodb_buffer_pool_size = 431669379072; [fundraising]
17:50 <jhathaway@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-ctrl2004.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [production]
17:46 <wm-bot2> Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659656791 (https://github.com/cluebotng/component-configs/commits/f4a494492433360a06326a918985c51c6d0828d4) [tools.cluebotng-staging]
17:46 <wm-bot2> Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659741719 (https://github.com/cluebotng/component-configs/commits/4dfdf7978d2efacf8a432919fa719c707a917da8) [tools.cluebot-syncer]
17:45 <wm-bot2> Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23659656811 (https://github.com/cluebotng/component-configs/commits/f4a494492433360a06326a918985c51c6d0828d4) [tools.cluebotng-review]
17:44 <wm-bot2> Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23659602695 (https://github.com/cluebotng/component-configs/commits/6f1ef446ee988193ccd8d9c62eb4841267976159) [tools.cluebotng-trainer]
17:44 <wm-bot2> Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23659656823 (https://github.com/cluebotng/component-configs/commits/f4a494492433360a06326a918985c51c6d0828d4) [tools.cluebotng]
17:44 <wm-bot2> Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23659587521 (https://github.com/cluebotng/component-configs/commits/6d62fa6482ab7ce2ff8a99029633d738f068af76) [tools.cluebotng-monitoring]
17:43 <wm-bot2> Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659537327 (https://github.com/cluebotng/component-configs/commits/c3f980e28e95bd1081b2ed9c903d2ac4d51b2c3b) [tools.cluebotng-review]
17:42 <wm-bot2> Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23659656791 (https://github.com/cluebotng/component-configs/commits/f4a494492433360a06326a918985c51c6d0828d4) [tools.cluebotng-staging]
17:42 <wm-bot2> Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659535101 (https://github.com/cluebotng/component-configs/commits/b825610c7f23870a8561b00b5f8546b107643015) [tools.cluebotng]
17:42 <wm-bot2> Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659545398 (https://github.com/cluebotng/component-configs/commits/8bb9fa3878bfe2763bc9fc12643727602969ae50) [tools.cluebotng-monitoring]
17:41 <wm-bot2> Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659537335 (https://github.com/cluebotng/component-configs/commits/c3f980e28e95bd1081b2ed9c903d2ac4d51b2c3b) [tools.cluebotng-staging]
17:41 <wm-bot2> Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659537319 (https://github.com/cluebotng/component-configs/commits/c3f980e28e95bd1081b2ed9c903d2ac4d51b2c3b) [tools.cluebotng-trainer]
17:41 <wm-bot2> Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659544147 (https://github.com/cluebotng/component-configs/commits/5a24521cc1a0802f0441809142015e96acf003ad) [tools.cluebotng-editsets]