601-650 of 10000 results (9ms)
2023-01-16 §
01:35 <Amir1> rolling restart of php-fpm across the fleet [production]
01:30 <thcipriani> 01:29:56 php-fpm-restart: 100% (in-flight: 0; ok: 184; fail: 112; left: 0) [production]
01:29 <thcipriani@deploy1002> Finished scap: Backport for [[gerrit:879798|LanguageDropdown: Check if the page is in talk namespaces instead (T316559 T326788)]] (duration: 24m 47s) [production]
01:15 <thcipriani@deploy1002> thcipriani and func: Backport for [[gerrit:879798|LanguageDropdown: Check if the page is in talk namespaces instead (T316559 T326788)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet [production]
01:05 <thcipriani@deploy1002> Started scap: Backport for [[gerrit:879798|LanguageDropdown: Check if the page is in talk namespaces instead (T316559 T326788)]] [production]
00:19 <Krinkle> Reloading Zuul [releng]
00:10 <Krinkle> Move https://github.com/countervandalism/cvn-clerkbot to https://github.com/wikimedia/countervandalism-cvn-clerkbot (with HTTP and Git redirect preserved), and replace with Gerrit mirror [cvn]
2023-01-15 §
23:25 <wikibugs> Updated channels.yaml to: 3fc8f04d85b953d989704f66c6ff345b127cd614 Enable phab and gerrit notifs for #countervandalism [tools.wikibugs]
23:12 <Krinkle> Create 'labs-cvn' permission group in Gerrit with CVN staff members [cvn]
23:12 <Krinkle> Move https://github.com/countervandalism/cvn-api to https://github.com/wikimedia/countervandalism-cvn-api (with HTTP and Git redirect preserved), and replace with Gerrit mirror [cvn]
22:02 <Krinkle> Switch new cvn.wmcloud.org proxy from cvn-apache9 to cvn-apache10 (Leave main cvn.wmflabs.org as-is for now). [cvn]
2023-01-14 §
21:45 <AntiComposite> move cvn-clerkbot from cvn-app9 to cvn-app12 (deploy 4cee27a) [cvn]
21:22 <AntiComposite> move cvn-clerbot back to cvn-app9 (deploy 371ba2a) [cvn]
21:10 <AntiComposite> move cvn-clerkbot from cvn-app9 to cvn-app12 (deploy 3f3f40f) [cvn]
09:46 <godog> issue 'request system reboot member 2' - T327001 [production]
09:20 <mvernon@cumin2002> conftool action : set/pooled=no; selector: name=thanos-fe2002.codfw.wmnet [production]
09:19 <Emperor> depool thanos-fe2002 T327001 [production]
09:19 <mvernon@cumin2002> conftool action : set/pooled=no; selector: name=ms-fe2010.codfw.wmnet [production]
09:19 <Emperor> depool ms-fe2010 T327001 [production]
2023-01-13 §
23:39 <mutante> people2002 - systemctl reset-failed after removing auto_restart_rsync timers [production]
22:26 <mutante> mirror1001 - systemctl start update-ubuntu-mirror (sometimes sync fails) [production]
20:58 <pt1979@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=False) upgrade firmware for hosts ['druid1011'] [production]
20:58 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['druid1011'] [production]
20:56 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=True) upgrade firmware for hosts ['druid1011'] [production]
20:49 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['druid1011'] [production]
20:47 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['druid1011'] [production]
20:37 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['druid1011'] [production]
20:37 <pt1979@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=False) upgrade firmware for hosts ['druid1010'] [production]
20:36 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['druid1010'] [production]
20:36 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=True) upgrade firmware for hosts ['druid1010'] [production]
20:35 <pt1979@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=False) upgrade firmware for hosts ['druid1009'] [production]
20:35 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['druid1009'] [production]
20:35 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=True) upgrade firmware for hosts ['druid1009'] [production]
20:16 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['druid1010'] [production]
20:16 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['druid1009'] [production]
20:04 <dzahn@cumin2002> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host aphlict2001.codfw.wmnet [production]
19:58 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-mariadb1002.eqiad.wmnet with OS bullseye [production]
19:58 <pt1979@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002" [production]
19:54 <dzahn@cumin2002> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) aphlict2001.codfw.wmnet on all recursors [production]
19:54 <dzahn@cumin2002> START - Cookbook sre.dns.wipe-cache aphlict2001.codfw.wmnet on all recursors [production]
19:54 <dzahn@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
19:54 <dzahn@cumin2002> END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM aphlict2001.codfw.wmnet - dzahn@cumin2002" [production]
19:52 <dzahn@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM aphlict2001.codfw.wmnet - dzahn@cumin2002" [production]
19:51 <pt1979@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002" [production]
19:49 <dzahn@cumin2002> START - Cookbook sre.dns.netbox [production]
19:49 <dzahn@cumin2002> START - Cookbook sre.ganeti.makevm for new host aphlict2001.codfw.wmnet [production]
19:40 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-mariadb1001.eqiad.wmnet with OS bullseye [production]
19:40 <pt1979@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002" [production]
19:38 <pt1979@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002" [production]
19:37 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-mariadb1002.eqiad.wmnet with reason: host reimage [production]