451-500 of 10000 results (92ms)
2025-10-23 §
02:05 <tstarling@deploy2002> Started scap sync-world: Backport for [[gerrit:1198178|recentchanges: QueryRateEstimator improvements (T403798)]], [[gerrit:1198185|recentchanges: Restore table qualifiers in change tag field expressions (T408040)]] [production]
01:35 <fceratto@cumin1003> END (FAIL) - Cookbook sre.mysql.clone_es (exit_code=99) of es2034.codfw.wmnet onto es2057.codfw.wmnet [production]
00:18 <cwhite> restart apache2 on gerrit1003 [production]
2025-10-22 §
22:49 <Reedy> T407057 - ran mwscript extensions/OATHAuth/maintenance/MoveRecoveryCodesFromTOTP.php --wiki=metawiki [production]
22:35 <jhathaway@cumin1003> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['sretest1003'] [production]
22:34 <jhathaway@cumin1003> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['sretest1003'] [production]
22:33 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ms-be2078'] [production]
22:32 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ms-be2078'] [production]
22:32 <Reedy> T407057 - ran foreachwikiindblist private.dblist extensions/OATHAuth/maintenance/MoveRecoveryCodesFromTOTP.php [production]
22:31 <Reedy> T407057 - ran foreachwikiindblist fishbowl.dblist extensions/OATHAuth/maintenance/MoveRecoveryCodesFromTOTP.php [production]
22:30 <jdlrobson@deploy2002> Finished scap sync-world: Backport for [[gerrit:1194309|Enable QuickSurveys on all wikis (T317841)]] (duration: 10m 12s) [production]
22:29 <jhathaway@cumin1003> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['sretest1003'] [production]
22:26 <jdlrobson@deploy2002> jdlrobson: Continuing with sync [production]
22:25 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ms-be2078'] [production]
22:25 <Reedy> T407057 - ran mwscript extensions/OATHAuth/maintenance/MoveRecoveryCodesFromTOTP.php --wiki=officewiki [production]
22:25 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ms-be2078'] [production]
22:24 <jdlrobson@deploy2002> jdlrobson: Backport for [[gerrit:1194309|Enable QuickSurveys on all wikis (T317841)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
22:24 <pt1979@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['ms-be2078'] [production]
22:23 <jhathaway@cumin1003> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['sretest1003'] [production]
22:20 <jdlrobson@deploy2002> Started scap sync-world: Backport for [[gerrit:1194309|Enable QuickSurveys on all wikis (T317841)]] [production]
22:17 <jhathaway@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cp2058'] [production]
22:17 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ms-be2078'] [production]
22:17 <jhathaway@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp2058'] [production]
22:15 <jhathaway@cumin1003> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['sretest1003'] [production]
22:15 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ms-be2078'] [production]
22:15 <jhathaway@cumin1003> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['sretest1003'] [production]
22:15 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ms-be2078'] [production]
22:14 <jhathaway@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cp2058'] [production]
22:13 <jhathaway@cumin1003> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['sretest1002'] [production]
22:12 <jhathaway@cumin1003> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['sretest1002'] [production]
22:06 <jhathaway@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp2058'] [production]
22:03 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ms-be2078'] [production]
22:03 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ms-be2078'] [production]
21:09 <andrew@cumin2002> START - Cookbook sre.hosts.reimage for host cloudcontrol2010-dev.codfw.wmnet with OS trixie [production]
21:09 <andrew@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcontrol2010-dev.codfw.wmnet with OS bookworm [production]
20:57 <kgraessle@deploy2002> Finished scap sync-world: Backport for [[gerrit:1198147|Fix InvalidArgumentException in Watchlist (T407996)]] (duration: 10m 49s) [production]
20:52 <kgraessle@deploy2002> kgraessle: Continuing with sync [production]
20:50 <kgraessle@deploy2002> kgraessle: Backport for [[gerrit:1198147|Fix InvalidArgumentException in Watchlist (T407996)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
20:46 <kgraessle@deploy2002> Started scap sync-world: Backport for [[gerrit:1198147|Fix InvalidArgumentException in Watchlist (T407996)]] [production]
20:31 <andrew@cumin2002> START - Cookbook sre.hosts.reimage for host cloudcontrol2010-dev.codfw.wmnet with OS bookworm [production]
20:30 <andrew@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcontrol2010-dev.codfw.wmnet with OS trixie [production]
20:26 <krinkle@deploy2002> Finished scap sync-world: Backport for [[gerrit:1197866|fix(MentorDashboard): fix caching for PHP 8.1 -> 8.3 migration (T407403)]] (duration: 12m 38s) [production]
20:23 <cmooney@cumin1003> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest1005.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
20:22 <krinkle@deploy2002> krinkle: Continuing with sync [production]
20:22 <cmooney@cumin1003> START - Cookbook sre.hosts.provision for host sretest1005.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
20:20 <cmooney@cumin1003> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest1005.eqiad.wmnet with OS trixie [production]
20:18 <krinkle@deploy2002> krinkle: Backport for [[gerrit:1197866|fix(MentorDashboard): fix caching for PHP 8.1 -> 8.3 migration (T407403)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
20:14 <krinkle@deploy2002> Started scap sync-world: Backport for [[gerrit:1197866|fix(MentorDashboard): fix caching for PHP 8.1 -> 8.3 migration (T407403)]] [production]
20:04 <ebernhardson@deploy2002> helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
20:04 <ebernhardson@deploy2002> helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply [production]