4301-4350 of 10000 results (126ms)
2024-10-28 §
08:42 <dcausse> T378227: deleting broken cirrus titlesugest index dewiki_titlesuggest_1729824440 [production]
08:38 <kostajh> UTC morning deploys done [production]
08:38 <kharlan@deploy2002> Finished scap sync-world: Backport for [[gerrit:1083591|ContributionsPager: Fix getTemplateParams() parameter (T378132)]], [[gerrit:1083592|Fix getTemplateParams() $classes parameter (T378132)]] (duration: 09m 38s) [production]
08:33 <kharlan@deploy2002> kharlan: Continuing with sync [production]
08:31 <kharlan@deploy2002> kharlan: Backport for [[gerrit:1083591|ContributionsPager: Fix getTemplateParams() parameter (T378132)]], [[gerrit:1083592|Fix getTemplateParams() $classes parameter (T378132)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
08:28 <kharlan@deploy2002> Started scap sync-world: Backport for [[gerrit:1083591|ContributionsPager: Fix getTemplateParams() parameter (T378132)]], [[gerrit:1083592|Fix getTemplateParams() $classes parameter (T378132)]] [production]
08:27 <hashar> Pushed https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CheckUser/+/1083592 and https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1083591 for wmf/1.43.0-wmf.28 / T378132 due to a dependency loop [production]
08:24 <hashar> Pushed https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CheckUser/+/1083592 and https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CheckUser/+/1083592 for wmf/1.43.0-wmf.28 / T378132 due to a dependency loop [production]
08:19 <hashar> Changed UTC morning backport window from 00:00 SF to 09:00 CET (aka 08:00 UTC) | UTC morning backport window [production]
08:07 <kartik@deploy2002> Finished scap sync-world: Backport for [[gerrit:1083292|Disable MT in Content Translation on Lithuanian Wikipedia (T364073)]] (duration: 22m 24s) [production]
08:01 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on db1234.eqiad.wmnet with reason: maintenance T378267 [production]
08:01 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on db1234.eqiad.wmnet with reason: maintenance T378267 [production]
08:01 <hashar> Restarted CI Jenkins to update the Collapsible Sections plugin | T378327 [production]
07:57 <kartik@deploy2002> kartik: Continuing with sync [production]
07:56 <kartik@deploy2002> kartik: Backport for [[gerrit:1083292|Disable MT in Content Translation on Lithuanian Wikipedia (T364073)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
07:45 <kartik@deploy2002> Started scap sync-world: Backport for [[gerrit:1083292|Disable MT in Content Translation on Lithuanian Wikipedia (T364073)]] [production]
07:14 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db[1169,1234].eqiad.wmnet with reason: maintenance [production]
07:14 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on db[1169,1234].eqiad.wmnet with reason: maintenance [production]
06:07 <taavi@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1169.eqiad.wmnet with reason: replication broken T378320 [production]
06:06 <taavi@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1169.eqiad.wmnet with reason: replication broken T378320 [production]
06:03 <taavi@cumin1002> dbctl commit (dc=all): 'depool db1169', diff saved to https://phabricator.wikimedia.org/P70590 and previous config saved to /var/cache/conftool/dbconfig/20241028-060327-taavi.json [production]
2024-10-27 §
13:41 <Dreamy_Jazz> Starting MediaModeration scanning on group1 wikis [production]
13:37 <Dreamy_Jazz> Starting MediaModeration scanning on group2 wikis [production]
2024-10-26 §
16:29 <mvernon@cumin1002> dbctl commit (dc=all): 'Depool db1234', diff saved to https://phabricator.wikimedia.org/P70589 and previous config saved to /var/cache/conftool/dbconfig/20241026-162946-mvernon.json [production]
16:29 <mvernon@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db1234.eqiad.wmnet with reason: spontaneous reboot, depooling 'til Monday [production]
16:28 <mvernon@cumin1002> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db1234.eqiad.wmnet with reason: spontaneous reboot, depooling 'til Monday [production]
02:03 <tzatziki> removing 9 files for legal compliance [production]
2024-10-25 §
18:29 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host backup2012.codfw.wmnet with OS bookworm [production]
18:28 <jhancock@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" [production]
18:18 <jhancock@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" [production]
17:54 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on backup2012.codfw.wmnet with reason: host reimage [production]
17:50 <jhancock@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on backup2012.codfw.wmnet with reason: host reimage [production]
16:42 <jhancock@cumin2002> START - Cookbook sre.hosts.reimage for host backup2012.codfw.wmnet with OS bookworm [production]
16:32 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host backup2012.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
16:28 <JustHannah> T378170 Ran mwscript-k8s extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=trwiki --logwiki=metawiki 'Peter.kerepesi' 'Peakbagger77' @ 11:57:19 UTC [production]
15:59 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host backup2012.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
15:57 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host backup2012.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
15:56 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host backup2012.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
15:53 <jhancock@cumin2002> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host backup2012 [production]
15:52 <jhancock@cumin2002> START - Cookbook sre.network.configure-switch-interfaces for host backup2012 [production]
15:52 <jhancock@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
15:52 <jhancock@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding backup2012 to codfw - jhancock@cumin2002" [production]
15:51 <jhancock@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding backup2012 to codfw - jhancock@cumin2002" [production]
15:47 <jhancock@cumin2002> START - Cookbook sre.dns.netbox [production]
15:07 <dancy@deploy2002> Installation of scap version "4.118.0" completed for 209 hosts [production]
15:03 <dancy@deploy2002> Installing scap version "4.118.0" for 209 hosts [production]
14:31 <herron> alert1002: manually killed stunnel4 process to clear puppet failure T375143 [production]
14:02 <sukhe> running authdns-update for CR 1082548 [production]
10:31 <arnaudb@cumin1002> dbctl commit (dc=all): 'maintenance', diff saved to https://phabricator.wikimedia.org/P70588 and previous config saved to /var/cache/conftool/dbconfig/20241025-103157-arnaudb.json [production]
10:21 <elukey@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [production]