5351-5400 of 10000 results (80ms)
2022-12-19 §
09:07 <ladsgroup@deploy1002> ladsgroup and ladsgroup: Backport for [[gerrit:868867|Emergency: discussiontoolspageinfo return empty response in non-talk ns (T325477)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet [production]
09:06 <elukey@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . [production]
09:05 <ladsgroup@deploy1002> Started scap: Backport for [[gerrit:868867|Emergency: discussiontoolspageinfo return empty response in non-talk ns (T325477)]] [production]
09:04 <dcausse> restarting blazegraph on wdqs1015 (BlazegraphFreeAllocatorsDecreasingRapidly) [production]
09:03 <elukey@deploy1002> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . [production]
09:02 <elukey@deploy1002> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . [production]
09:01 <elukey@deploy1002> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . [production]
08:59 <elukey@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . [production]
08:58 <elukey@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . [production]
08:56 <elukey@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . [production]
08:37 <elukey@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . [production]
08:34 <elukey@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . [production]
08:33 <ayounsi@cumin1001> END (FAIL) - Cookbook sre.network.peering (exit_code=99) with action 'email' for AS: 29535 [production]
08:32 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'email' for AS: 29535 [production]
08:02 <moritzm> installing openexr security updates [production]
07:22 <phedenskog@deploy1002> Finished deploy [performance/navtiming@5770d46]: (no justification provided) (duration: 00m 08s) [production]
07:22 <phedenskog@deploy1002> Started deploy [performance/navtiming@5770d46]: (no justification provided) [production]
07:05 <ayounsi@cumin1001> END (FAIL) - Cookbook sre.network.peering (exit_code=99) with action 'configure' for AS: 28398 [production]
07:05 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'configure' for AS: 28398 [production]
07:04 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 11686 [production]
06:48 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'configure' for AS: 11686 [production]
03:08 <tstarling@deploy1002> Synchronized wmf-config/InitialiseSettings.php: disable wgDiscussionToolsABTest T325477 T321961 (duration: 15m 23s) [production]
2022-12-18 §
19:40 <sukhe> ran sudo cumin -b 1 -s 30 'A:mw-api and A:eqiad' 'restart-php7.4-fpm' [at 18:55 UTC]: T325477 [production]
19:31 <sukhe> [FINISHED] running sudo cumin -b 1 -s 30 'A:mw-api and A:eqiad' 'restart-php7.4-fpm' [production]
19:31 <sukhe> [FINISHED] running sudo cumin -b 1 -s 30 'A:mw-api and A:eqiad' 'restart-php7.4-fpm' [production]
18:55 <sukhe> running sudo cumin -b 1 -s 30 'A:mw-api and A:eqiad' 'restart-php7.4-fpm' [production]
18:28 <cgoubert@cumin1001> END (FAIL) - Cookbook sre.mediawiki.restart-appservers (exit_code=99) [production]
18:28 <cgoubert@cumin1001> START - Cookbook sre.mediawiki.restart-appservers [production]
18:20 <cgoubert@cumin1001> END (FAIL) - Cookbook sre.mediawiki.restart-appservers (exit_code=99) [production]
18:20 <cgoubert@cumin1001> START - Cookbook sre.mediawiki.restart-appservers [production]
2022-12-17 §
14:36 <aikochou@deploy1002> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . [production]
14:30 <aikochou@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . [production]
2022-12-16 §
19:55 <robh@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti4007.ulsfo.wmnet with OS bullseye [production]
19:55 <robh@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - robh@cumin2002" [production]
19:00 <robh@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - robh@cumin2002" [production]
18:44 <robh@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti4007.ulsfo.wmnet with reason: host reimage [production]
18:41 <robh@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti4007.ulsfo.wmnet with reason: host reimage [production]
18:19 <robh@cumin2002> START - Cookbook sre.hosts.reimage for host ganeti4007.ulsfo.wmnet with OS bullseye [production]
18:08 <robh@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['ganeti4007'] [production]
17:56 <robh@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ganeti4007'] [production]
17:53 <robh@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['ganeti4007'] [production]
17:52 <robh@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ganeti4007'] [production]
17:28 <mutante> etherpad1003 - testing new: sudo systemctl start wmf_auto_restart_envoyproxy [production]
16:46 <robh@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti4007.mgmt.ulsfo.wmnet with reboot policy FORCED [production]
16:24 <robh@cumin2002> START - Cookbook sre.hosts.provision for host ganeti4007.mgmt.ulsfo.wmnet with reboot policy FORCED [production]
16:24 <robh@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
16:23 <robh@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: gaenti4007 - robh@cumin2002" [production]
16:22 <robh@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: gaenti4007 - robh@cumin2002" [production]
16:20 <robh@cumin2002> START - Cookbook sre.dns.netbox [production]
16:19 <robh@cumin2002> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti4007 [production]