101-150 of 10000 results (44ms)
2023-02-23 ยง
17:31 <cdanis@cumin1001> dbctl commit (dc=all): 'db1127 running very hot', diff saved to https://phabricator.wikimedia.org/P44752 and previous config saved to /var/cache/conftool/dbconfig/20230223-173127-cdanis.json [production]
17:31 <dcaro@cumin1001> START - Cookbook sre.dns.netbox [production]
17:27 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dns4003.wikimedia.org with reason: host reimage [production]
17:25 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on dns4003.wikimedia.org with reason: host reimage [production]
17:24 <bking@cumin1001> END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) [production]
17:23 <bking@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
17:19 <bking@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
17:18 <bking@cumin1001> END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) [production]
17:18 <bking@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
17:17 <hnowlan@deploy1002> helmfile [staging] DONE helmfile.d/services/thumbor: apply [production]
17:07 <hnowlan@deploy1002> helmfile [staging] START helmfile.d/services/thumbor: apply [production]
17:06 <sukhe@cumin2002> START - Cookbook sre.hosts.reimage for host dns4003.wikimedia.org with OS bullseye [production]
16:58 <papaul> raplacing fpc2 to fpc1 DAC cable complete [production]
16:52 <papaul> raplacing fpc2 to fpc1 DAC cable [production]
16:52 <hnowlan@cumin1001> END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-eqiad and (A:swift-fe or A:swift-fe-canary or A:swift-fe-codfw or A:swift-fe-eqiad) [production]
16:42 <hnowlan@cumin1001> START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-eqiad and (A:swift-fe or A:swift-fe-canary or A:swift-fe-codfw or A:swift-fe-eqiad) [production]
16:41 <hnowlan> eqiad: roll-restarting swift frontends and thumbor hosts for key rotation [production]
16:29 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['wdqs2021'] [production]
16:29 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['wdqs2021'] [production]
16:26 <jbond@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts sretest1002.eqiad.wmnet [production]
16:18 <hnowlan@cumin1001> END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-codfw and (A:swift-fe or A:swift-fe-canary or A:swift-fe-codfw or A:swift-fe-eqiad) [production]
16:15 <hnowlan@cumin1001> START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-codfw and (A:swift-fe or A:swift-fe-canary or A:swift-fe-codfw or A:swift-fe-eqiad) [production]
16:14 <hnowlan> codfw: roll-restarting swift frontends and thumbor hosts for key rotation [production]
16:13 <jbond@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts sretest1002.eqiad.wmnet [production]
16:13 <jbond@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts sretest1002.eqiad.wmnet [production]
16:07 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['wdqs2021'] [production]
16:07 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['wdqs2021'] [production]
16:07 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['wdqs2021'] [production]
16:07 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['wdqs2021'] [production]
16:04 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['wdqs2020'] [production]
16:04 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['wdqs2020'] [production]
16:04 <dcaro@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
16:04 <dcaro@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Moved cloudcephosd1003/1004 to new racks - dcaro@cumin1001" [production]
16:03 <moritzm> installing c-ares security updates [production]
16:03 <jclark@cumin1001> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1002 [production]
16:03 <jclark@cumin1001> START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1002 [production]
16:03 <jclark@cumin1001> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudcephosd1001 [production]
16:03 <jclark@cumin1001> START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd1001 [production]
16:03 <dcaro@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Moved cloudcephosd1003/1004 to new racks - dcaro@cumin1001" [production]
16:00 <bking@cumin1001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
16:00 <dcaro@cumin1001> START - Cookbook sre.dns.netbox [production]
15:57 <dcaro@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
15:57 <dcaro@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Moved cloudcephosd1003/1004 to new racks - dcaro@cumin1001" [production]
15:51 <dcaro@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Moved cloudcephosd1003/1004 to new racks - dcaro@cumin1001" [production]
15:45 <dcaro@cumin1001> START - Cookbook sre.dns.netbox [production]
15:42 <klausman@cumin1001> END (PASS) - Cookbook sre.ores.roll-restart-workers (exit_code=0) for ORES eqiad cluster: Roll restart of ORES's daemons. [production]
15:38 <dcaro@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
15:36 <dcaro@cumin1001> START - Cookbook sre.dns.netbox [production]
15:27 <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Move more ns-related config out of InitialiseSettings, part II (T308932) (duration: 06m 35s) [production]
15:22 <klausman@cumin1001> START - Cookbook sre.ores.roll-restart-workers for ORES eqiad cluster: Roll restart of ORES's daemons. [production]