1251-1300 of 10000 results (95ms)
2023-01-25 ยง
23:21 <zabe@deploy1002> Started scap: (no justification provided) [production]
23:20 <zabe@deploy1002> Backport cancelled. [production]
23:14 <brett@cumin1001> START - Cookbook sre.hosts.reimage for host cp6004.drmrs.wmnet with OS bullseye [production]
23:13 <brett@cumin1001> conftool action : set/pooled=yes; selector: name=cp6012.drmrs.wmnet [production]
23:07 <brett@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp6012.drmrs.wmnet with OS bullseye [production]
22:43 <brett@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp6012.drmrs.wmnet with reason: host reimage [production]
22:40 <brett@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cp6012.drmrs.wmnet with reason: host reimage [production]
22:21 <brett@cumin1001> START - Cookbook sre.hosts.reimage for host cp6012.drmrs.wmnet with OS bullseye [production]
22:14 <brett@cumin1001> conftool action : set/pooled=yes; selector: name=cp6003.drmrs.wmnet [production]
21:49 <bking@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/services/flink-app-example: apply [production]
21:49 <bking@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/services/flink-app-example: apply [production]
21:44 <bking@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/services/flink-app-example: apply [production]
21:44 <bking@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/services/flink-app-example: apply [production]
21:34 <samtar@deploy1002> Finished scap: Backport for [[gerrit:883617|Define grid template row for .mw-body grid container to ensure the grid cell containing the content will expand in height when needed (T327714)]], [[gerrit:883616|Define grid template row for .mw-body grid container to ensure the grid cell containing the content will expand in height when needed (T327714)]] (duration: 09m 27s) [production]
21:26 <samtar@deploy1002> jdrewniak and samtar: Backport for [[gerrit:883617|Define grid template row for .mw-body grid container to ensure the grid cell containing the content will expand in height when needed (T327714)]], [[gerrit:883616|Define grid template row for .mw-body grid container to ensure the grid cell containing the content will expand in height when needed (T327714)]] synced to the testservers: mwdebug2002.cod [production]
21:25 <bking@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
21:24 <bking@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
21:24 <samtar@deploy1002> Started scap: Backport for [[gerrit:883617|Define grid template row for .mw-body grid container to ensure the grid cell containing the content will expand in height when needed (T327714)]], [[gerrit:883616|Define grid template row for .mw-body grid container to ensure the grid cell containing the content will expand in height when needed (T327714)]] [production]
21:06 <sukhe@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts cp2028.codfw.wmnet [production]
20:59 <sukhe@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp2028.codfw.wmnet [production]
20:59 <brett@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp6003.drmrs.wmnet with OS bullseye [production]
20:59 <sukhe@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=True) upgrade firmware for hosts cp2028.codfw.wmnet [production]
20:58 <sukhe@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp2028.codfw.wmnet [production]
20:56 <sukhe@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts cp2028.codfw.wmnet [production]
20:49 <sukhe@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp2028.codfw.wmnet [production]
20:49 <sukhe@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts cp2028.codfw.wmnet [production]
20:49 <sukhe@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp2028.codfw.wmnet [production]
20:49 <sukhe@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts cp2028.codfw.wmnet [production]
20:49 <ejegg> updated employers.csv on paymentswiki [production]
20:49 <sukhe@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp2028.codfw.wmnet [production]
20:33 <brett@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp6003.drmrs.wmnet with reason: host reimage [production]
20:32 <btullis@cumin1001> END (PASS) - Cookbook sre.kafka.reboot-workers (exit_code=0) for Kafka jumbo-eqiad cluster: Reboot kafka nodes [production]
20:30 <brett@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cp6003.drmrs.wmnet with reason: host reimage [production]
20:10 <brett@cumin1001> START - Cookbook sre.hosts.reimage for host cp6003.drmrs.wmnet with OS bullseye [production]
20:00 <brett@cumin1001> conftool action : set/pooled=yes; selector: name=cp6011.drmrs.wmnet [production]
19:58 <brett@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp6011.drmrs.wmnet with OS bullseye [production]
19:52 <denisse@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host centrallog1002.eqiad.wmnet with OS bullseye [production]
19:38 <denisse@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on centrallog1002.eqiad.wmnet with reason: host reimage [production]
19:36 <brett@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp6011.drmrs.wmnet with reason: host reimage [production]
19:33 <denisse@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on centrallog1002.eqiad.wmnet with reason: host reimage [production]
19:33 <brett@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cp6011.drmrs.wmnet with reason: host reimage [production]
19:21 <denisse@cumin1001> START - Cookbook sre.hosts.reimage for host centrallog1002.eqiad.wmnet with OS bullseye [production]
19:17 <brennen@deploy1002> Synchronized php: group1 wikis to 1.40.0-wmf.20 refs T325583 (duration: 07m 04s) [production]
19:12 <brett@cumin1001> START - Cookbook sre.hosts.reimage for host cp6011.drmrs.wmnet with OS bullseye [production]
19:10 <brennen@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.40.0-wmf.20 refs T325583 [production]
19:06 <brett@cumin1001> conftool action : set/pooled=yes; selector: name=cp6002.drmrs.wmnet [production]
19:01 <brennen> 1.40.0-wmf.20 train (T325583): no blockers, rolling to group1. [production]
19:00 <denisse@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host centrallog1002.eqiad.wmnet with OS bullseye [production]
19:00 <denisse@cumin1001> START - Cookbook sre.hosts.reimage for host centrallog1002.eqiad.wmnet with OS bullseye [production]
18:59 <brett@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp6002.drmrs.wmnet with OS bullseye [production]