3301-3350 of 10000 results (80ms)
2023-01-16 §
01:35 <Amir1> rolling restart of php-fpm across the fleet [production]
01:30 <thcipriani> 01:29:56 php-fpm-restart: 100% (in-flight: 0; ok: 184; fail: 112; left: 0) [production]
01:29 <thcipriani@deploy1002> Finished scap: Backport for [[gerrit:879798|LanguageDropdown: Check if the page is in talk namespaces instead (T316559 T326788)]] (duration: 24m 47s) [production]
01:15 <thcipriani@deploy1002> thcipriani and func: Backport for [[gerrit:879798|LanguageDropdown: Check if the page is in talk namespaces instead (T316559 T326788)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet [production]
01:05 <thcipriani@deploy1002> Started scap: Backport for [[gerrit:879798|LanguageDropdown: Check if the page is in talk namespaces instead (T316559 T326788)]] [production]
2023-01-14 §
09:46 <godog> issue 'request system reboot member 2' - T327001 [production]
09:20 <mvernon@cumin2002> conftool action : set/pooled=no; selector: name=thanos-fe2002.codfw.wmnet [production]
09:19 <Emperor> depool thanos-fe2002 T327001 [production]
09:19 <mvernon@cumin2002> conftool action : set/pooled=no; selector: name=ms-fe2010.codfw.wmnet [production]
09:19 <Emperor> depool ms-fe2010 T327001 [production]
2023-01-13 §
23:39 <mutante> people2002 - systemctl reset-failed after removing auto_restart_rsync timers [production]
22:26 <mutante> mirror1001 - systemctl start update-ubuntu-mirror (sometimes sync fails) [production]
20:58 <pt1979@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=False) upgrade firmware for hosts ['druid1011'] [production]
20:58 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['druid1011'] [production]
20:56 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=True) upgrade firmware for hosts ['druid1011'] [production]
20:49 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['druid1011'] [production]
20:47 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['druid1011'] [production]
20:37 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['druid1011'] [production]
20:37 <pt1979@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=False) upgrade firmware for hosts ['druid1010'] [production]
20:36 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['druid1010'] [production]
20:36 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=True) upgrade firmware for hosts ['druid1010'] [production]
20:35 <pt1979@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=False) upgrade firmware for hosts ['druid1009'] [production]
20:35 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['druid1009'] [production]
20:35 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=True) upgrade firmware for hosts ['druid1009'] [production]
20:16 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['druid1010'] [production]
20:16 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['druid1009'] [production]
20:04 <dzahn@cumin2002> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host aphlict2001.codfw.wmnet [production]
19:58 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-mariadb1002.eqiad.wmnet with OS bullseye [production]
19:58 <pt1979@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002" [production]
19:54 <dzahn@cumin2002> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) aphlict2001.codfw.wmnet on all recursors [production]
19:54 <dzahn@cumin2002> START - Cookbook sre.dns.wipe-cache aphlict2001.codfw.wmnet on all recursors [production]
19:54 <dzahn@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
19:54 <dzahn@cumin2002> END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM aphlict2001.codfw.wmnet - dzahn@cumin2002" [production]
19:52 <dzahn@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM aphlict2001.codfw.wmnet - dzahn@cumin2002" [production]
19:51 <pt1979@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002" [production]
19:49 <dzahn@cumin2002> START - Cookbook sre.dns.netbox [production]
19:49 <dzahn@cumin2002> START - Cookbook sre.ganeti.makevm for new host aphlict2001.codfw.wmnet [production]
19:40 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-mariadb1001.eqiad.wmnet with OS bullseye [production]
19:40 <pt1979@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002" [production]
19:38 <pt1979@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002" [production]
19:37 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-mariadb1002.eqiad.wmnet with reason: host reimage [production]
19:34 <pt1979@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on an-mariadb1002.eqiad.wmnet with reason: host reimage [production]
19:26 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-mariadb1001.eqiad.wmnet with reason: host reimage [production]
19:22 <pt1979@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on an-mariadb1001.eqiad.wmnet with reason: host reimage [production]
19:22 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host an-mariadb1002.eqiad.wmnet with OS bullseye [production]
18:25 <zabe> mwscript extensions/GlobalBlocking/maintenance/FixBlockerUsername.php --wiki metawiki "Green Giant" "Cromium" # T298707 [production]
17:34 <thcipriani@deploy1002> Finished scap: Backport for [[gerrit:879793|TranslationNotificationsSubmitJob: Ensure LanguageSet is in proper format (T63125)]] (duration: 13m 25s) [production]
17:22 <thcipriani@deploy1002> thcipriani and abi: Backport for [[gerrit:879793|TranslationNotificationsSubmitJob: Ensure LanguageSet is in proper format (T63125)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet [production]
17:20 <thcipriani@deploy1002> Started scap: Backport for [[gerrit:879793|TranslationNotificationsSubmitJob: Ensure LanguageSet is in proper format (T63125)]] [production]
15:34 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-coord1004.eqiad.wmnet with OS bullseye [production]