5801-5850 of 10000 results (67ms)
2022-06-09 ยง
14:17 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on webperf2002.codfw.wmnet with reason: Migration to new Bullseye nodes [production]
14:17 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on webperf1002.eqiad.wmnet with reason: Migration to new Bullseye nodes [production]
14:17 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on webperf1002.eqiad.wmnet with reason: Migration to new Bullseye nodes [production]
14:09 <moritzm> masking Excimer/Arclamp services/timers on webperf1002/2002 T305460 [production]
14:07 <mvernon@cumin2002> START - Cookbook sre.hosts.reimage for host ms-be1063.eqiad.wmnet with OS bullseye [production]
13:54 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1062.eqiad.wmnet with OS bullseye [production]
13:47 <btullis@cumin1001> END (FAIL) - Cookbook sre.hadoop.roll-restart-masters (exit_code=99) restart masters for Hadoop analytics cluster: Restart of jvm daemons. [production]
13:45 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T310011)', diff saved to https://phabricator.wikimedia.org/P29603 and previous config saved to /var/cache/conftool/dbconfig/20220609-134558-marostegui.json [production]
13:37 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1062.eqiad.wmnet with reason: host reimage [production]
13:34 <mvernon@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1062.eqiad.wmnet with reason: host reimage [production]
13:30 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P29602 and previous config saved to /var/cache/conftool/dbconfig/20220609-133053-marostegui.json [production]
13:15 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P29601 and previous config saved to /var/cache/conftool/dbconfig/20220609-131548-marostegui.json [production]
13:15 <btullis@cumin1001> START - Cookbook sre.hadoop.roll-restart-masters restart masters for Hadoop analytics cluster: Restart of jvm daemons. [production]
13:15 <mvernon@cumin2002> START - Cookbook sre.hosts.reimage for host ms-be1062.eqiad.wmnet with OS bullseye [production]
13:00 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T310011)', diff saved to https://phabricator.wikimedia.org/P29600 and previous config saved to /var/cache/conftool/dbconfig/20220609-130042-marostegui.json [production]
13:00 <moritzm> installing libjpeg-turbo security updates [production]
12:57 <moritzm> installing xen security updates (client-side libs only) [production]
12:49 <btullis@cumin1001> END (PASS) - Cookbook sre.hadoop.roll-restart-masters (exit_code=0) restart masters for Hadoop test cluster: Restart of jvm daemons. [production]
12:45 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1098:3317 (T310011)', diff saved to https://phabricator.wikimedia.org/P29599 and previous config saved to /var/cache/conftool/dbconfig/20220609-124529-marostegui.json [production]
12:45 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1098.eqiad.wmnet with reason: Maintenance [production]
12:45 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1098.eqiad.wmnet with reason: Maintenance [production]
12:33 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 10 hosts with reason: Maintenance [production]
12:33 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on 10 hosts with reason: Maintenance [production]
12:33 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2121.codfw.wmnet with reason: Maintenance [production]
12:33 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2121.codfw.wmnet with reason: Maintenance [production]
12:32 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T310011)', diff saved to https://phabricator.wikimedia.org/P29598 and previous config saved to /var/cache/conftool/dbconfig/20220609-123256-marostegui.json [production]
12:17 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P29597 and previous config saved to /var/cache/conftool/dbconfig/20220609-121750-marostegui.json [production]
12:16 <btullis@cumin1001> START - Cookbook sre.hadoop.roll-restart-masters restart masters for Hadoop test cluster: Restart of jvm daemons. [production]
12:15 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on ganeti3002.esams.wmnet with reason: Remove from cluster for firmware update and eventual reimage [production]
12:15 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on ganeti3002.esams.wmnet with reason: Remove from cluster for firmware update and eventual reimage [production]
12:02 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P29596 and previous config saved to /var/cache/conftool/dbconfig/20220609-120245-marostegui.json [production]
11:52 <btullis@cumin1001> END (PASS) - Cookbook sre.kafka.roll-restart-mirror-maker (exit_code=0) restart MirrorMaker for Kafka A:kafka-mirror-maker-test-eqiad cluster: Roll restart of jvm daemons. [production]
11:47 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T310011)', diff saved to https://phabricator.wikimedia.org/P29595 and previous config saved to /var/cache/conftool/dbconfig/20220609-114740-marostegui.json [production]
11:42 <btullis@cumin1001> START - Cookbook sre.kafka.roll-restart-mirror-maker restart MirrorMaker for Kafka A:kafka-mirror-maker-test-eqiad cluster: Roll restart of jvm daemons. [production]
11:38 <btullis@cumin1001> END (PASS) - Cookbook sre.kafka.roll-restart-brokers (exit_code=0) for Kafka A:kafka-test-eqiad cluster: Roll restart of jvm daemons. [production]
11:29 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1101:3317 (T310011)', diff saved to https://phabricator.wikimedia.org/P29594 and previous config saved to /var/cache/conftool/dbconfig/20220609-112945-marostegui.json [production]
11:29 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1101.eqiad.wmnet with reason: Maintenance [production]
11:29 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1101.eqiad.wmnet with reason: Maintenance [production]
11:28 <mmandere@cumin1001> conftool action : set/pooled=yes; selector: name=cp5006.* [production]
11:26 <mmandere> pool cp5006 after restart [production]
11:17 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance [production]
11:17 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance [production]
11:17 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1158 (T310011)', diff saved to https://phabricator.wikimedia.org/P29593 and previous config saved to /var/cache/conftool/dbconfig/20220609-111719-marostegui.json [production]
11:02 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P29592 and previous config saved to /var/cache/conftool/dbconfig/20220609-110214-marostegui.json [production]
10:55 <mmandere> restart cp5006 [production]
10:47 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P29591 and previous config saved to /var/cache/conftool/dbconfig/20220609-104709-marostegui.json [production]
10:32 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1158 (T310011)', diff saved to https://phabricator.wikimedia.org/P29590 and previous config saved to /var/cache/conftool/dbconfig/20220609-103204-marostegui.json [production]
09:58 <btullis@cumin1001> START - Cookbook sre.kafka.roll-restart-brokers for Kafka A:kafka-test-eqiad cluster: Roll restart of jvm daemons. [production]
09:31 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1158 (T310011)', diff saved to https://phabricator.wikimedia.org/P29588 and previous config saved to /var/cache/conftool/dbconfig/20220609-093148-marostegui.json [production]
09:31 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance [production]