6201-6250 of 10000 results (71ms)
2022-06-08 ยง
10:43 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host webperf1004.eqiad.wmnet [production]
10:41 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance [production]
10:41 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance [production]
10:41 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 (T310011)', diff saved to https://phabricator.wikimedia.org/P29522 and previous config saved to /var/cache/conftool/dbconfig/20220608-104150-marostegui.json [production]
10:38 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM webperf1004.eqiad.wmnet [production]
10:32 <jmm@cumin2002> START - Cookbook sre.ganeti.reboot-vm for VM webperf1004.eqiad.wmnet [production]
10:26 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P29521 and previous config saved to /var/cache/conftool/dbconfig/20220608-102645-marostegui.json [production]
10:12 <btullis@cumin1001> END (FAIL) - Cookbook sre.cassandra.roll-restart (exit_code=99) for nodes matching aqs100*: T309526 - btullis@cumin1001 [production]
10:11 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P29520 and previous config saved to /var/cache/conftool/dbconfig/20220608-101140-marostegui.json [production]
10:02 <btullis@cumin1001> START - Cookbook sre.cassandra.roll-restart for nodes matching aqs100*: T309526 - btullis@cumin1001 [production]
09:56 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 (T310011)', diff saved to https://phabricator.wikimedia.org/P29519 and previous config saved to /var/cache/conftool/dbconfig/20220608-095635-marostegui.json [production]
09:55 <btullis@cumin1001> END (PASS) - Cookbook sre.kafka.roll-restart-mirror-maker (exit_code=0) restart MirrorMaker for Kafka A:kafka-mirror-maker-jumbo-eqiad cluster: Roll restart of jvm daemons. [production]
09:51 <btullis@cumin1001> END (FAIL) - Cookbook sre.cassandra.roll-restart (exit_code=99) for nodes matching A:aqs: T309526 - btullis@cumin1001 [production]
09:49 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1096:3315 (T310011)', diff saved to https://phabricator.wikimedia.org/P29518 and previous config saved to /var/cache/conftool/dbconfig/20220608-094952-marostegui.json [production]
09:49 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1096.eqiad.wmnet with reason: Maintenance [production]
09:49 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1096.eqiad.wmnet with reason: Maintenance [production]
09:47 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 8 hosts with reason: Maintenance [production]
09:47 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on 8 hosts with reason: Maintenance [production]
09:47 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2123.codfw.wmnet with reason: Maintenance [production]
09:47 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2123.codfw.wmnet with reason: Maintenance [production]
09:47 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1161 (T310011)', diff saved to https://phabricator.wikimedia.org/P29517 and previous config saved to /var/cache/conftool/dbconfig/20220608-094716-marostegui.json [production]
09:46 <btullis@cumin1001> START - Cookbook sre.cassandra.roll-restart for nodes matching A:aqs: T309526 - btullis@cumin1001 [production]
09:40 <kartik@deploy1002> helmfile [eqiad] DONE helmfile.d/services/cxserver: apply [production]
09:40 <kartik@deploy1002> helmfile [eqiad] START helmfile.d/services/cxserver: apply [production]
09:39 <kartik@deploy1002> helmfile [codfw] DONE helmfile.d/services/cxserver: apply [production]
09:38 <kartik@deploy1002> helmfile [codfw] START helmfile.d/services/cxserver: apply [production]
09:37 <btullis@cumin1001> START - Cookbook sre.kafka.roll-restart-mirror-maker restart MirrorMaker for Kafka A:kafka-mirror-maker-jumbo-eqiad cluster: Roll restart of jvm daemons. [production]
09:32 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P29516 and previous config saved to /var/cache/conftool/dbconfig/20220608-093211-marostegui.json [production]
09:19 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1056.eqiad.wmnet with OS bullseye [production]
09:18 <kartik@deploy1002> helmfile [eqiad] DONE helmfile.d/services/cxserver: apply [production]
09:17 <kartik@deploy1002> helmfile [eqiad] START helmfile.d/services/cxserver: apply [production]
09:17 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P29515 and previous config saved to /var/cache/conftool/dbconfig/20220608-091706-marostegui.json [production]
09:13 <marostegui@cumin1001> dbctl commit (dc=all): 'Pool db1143 on s4 with small weight after installing 10.6 T310114', diff saved to https://phabricator.wikimedia.org/P29514 and previous config saved to /var/cache/conftool/dbconfig/20220608-091331-root.json [production]
09:10 <kartik@deploy1002> helmfile [codfw] DONE helmfile.d/services/cxserver: apply [production]
09:09 <kartik@deploy1002> helmfile [codfw] START helmfile.d/services/cxserver: apply [production]
09:08 <kartik@deploy1002> helmfile [staging] DONE helmfile.d/services/cxserver: apply [production]
09:07 <kartik@deploy1002> helmfile [staging] START helmfile.d/services/cxserver: apply [production]
09:06 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1056.eqiad.wmnet with reason: host reimage [production]
09:03 <mvernon@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1056.eqiad.wmnet with reason: host reimage [production]
09:02 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1161 (T310011)', diff saved to https://phabricator.wikimedia.org/P29513 and previous config saved to /var/cache/conftool/dbconfig/20220608-090201-marostegui.json [production]
08:55 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1161 (T310011)', diff saved to https://phabricator.wikimedia.org/P29512 and previous config saved to /var/cache/conftool/dbconfig/20220608-085554-marostegui.json [production]
08:55 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
08:55 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
08:55 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1161.eqiad.wmnet with reason: Maintenance [production]
08:55 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1161.eqiad.wmnet with reason: Maintenance [production]
08:55 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T310011)', diff saved to https://phabricator.wikimedia.org/P29511 and previous config saved to /var/cache/conftool/dbconfig/20220608-085541-marostegui.json [production]
08:50 <mvernon@cumin2002> START - Cookbook sre.hosts.reimage for host ms-be1056.eqiad.wmnet with OS bullseye [production]
08:47 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1055.eqiad.wmnet with OS bullseye [production]
08:44 <jnuche@deploy1002> Installation of scap version "4.9.0" completed for 540 hosts [production]
08:43 <jnuche@deploy1002> Installing scap version "4.9.0" for 540 hosts [production]