101-150 of 10000 results (32ms)
2022-03-30 ยง
18:03 <cmooney@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1069.eqiad.wmnet with reason: host reimage [production]
18:01 <razzi@cumin1001> START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet [production]
18:00 <razzi@cumin1001> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host zookeeper-test1002.eqiad.wmnet [production]
18:00 <razzi@cumin1001> START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet [production]
18:00 <cmooney@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1069.eqiad.wmnet with reason: host reimage [production]
17:59 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T298565)', diff saved to https://phabricator.wikimedia.org/P23872 and previous config saved to /var/cache/conftool/dbconfig/20220330-175930-ladsgroup.json [production]
17:58 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1170:3312 (T298565)', diff saved to https://phabricator.wikimedia.org/P23871 and previous config saved to /var/cache/conftool/dbconfig/20220330-175822-ladsgroup.json [production]
17:58 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance [production]
17:58 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance [production]
17:58 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1162 (T298565)', diff saved to https://phabricator.wikimedia.org/P23870 and previous config saved to /var/cache/conftool/dbconfig/20220330-175814-ladsgroup.json [production]
17:47 <cmooney@cumin1001> START - Cookbook sre.hosts.reimage for host ms-be1069.eqiad.wmnet with OS stretch [production]
17:46 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 10 hosts with reason: Maintenance [production]
17:46 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on 10 hosts with reason: Maintenance [production]
17:46 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2121.codfw.wmnet with reason: Maintenance [production]
17:46 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db2121.codfw.wmnet with reason: Maintenance [production]
17:44 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1142 (T298557)', diff saved to https://phabricator.wikimedia.org/P23869 and previous config saved to /var/cache/conftool/dbconfig/20220330-174426-marostegui.json [production]
17:44 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1142.eqiad.wmnet with reason: Maintenance [production]
17:44 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on db1142.eqiad.wmnet with reason: Maintenance [production]
17:44 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1141 (T298557)', diff saved to https://phabricator.wikimedia.org/P23868 and previous config saved to /var/cache/conftool/dbconfig/20220330-174418-marostegui.json [production]
17:43 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P23867 and previous config saved to /var/cache/conftool/dbconfig/20220330-174309-ladsgroup.json [production]
17:29 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P23866 and previous config saved to /var/cache/conftool/dbconfig/20220330-172913-marostegui.json [production]
17:28 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P23865 and previous config saved to /var/cache/conftool/dbconfig/20220330-172804-ladsgroup.json [production]
17:17 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1112 (T297189)', diff saved to https://phabricator.wikimedia.org/P23864 and previous config saved to /var/cache/conftool/dbconfig/20220330-171732-marostegui.json [production]
17:14 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P23862 and previous config saved to /var/cache/conftool/dbconfig/20220330-171408-marostegui.json [production]
17:13 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1162 (T298565)', diff saved to https://phabricator.wikimedia.org/P23861 and previous config saved to /var/cache/conftool/dbconfig/20220330-171259-ladsgroup.json [production]
17:07 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance [production]
17:07 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance [production]
17:02 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P23859 and previous config saved to /var/cache/conftool/dbconfig/20220330-170227-marostegui.json [production]
17:01 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1162 (T298565)', diff saved to https://phabricator.wikimedia.org/P23858 and previous config saved to /var/cache/conftool/dbconfig/20220330-170150-ladsgroup.json [production]
17:01 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1162.eqiad.wmnet with reason: Maintenance [production]
17:01 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1162.eqiad.wmnet with reason: Maintenance [production]
17:01 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1156 (T298565)', diff saved to https://phabricator.wikimedia.org/P23857 and previous config saved to /var/cache/conftool/dbconfig/20220330-170142-ladsgroup.json [production]
16:59 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1141 (T298557)', diff saved to https://phabricator.wikimedia.org/P23856 and previous config saved to /var/cache/conftool/dbconfig/20220330-165903-marostegui.json [production]
16:52 <topranks> "Manually decommissioning xe-0/0/1 on lsw1-e2-eqiad before reimage of ms-be1069 from scratch, attempt to replicate ARP error seen previously while running debug." [production]
16:52 <volans> sudo systemctl reload icinga.service on alert1001 [production]
16:47 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P23855 and previous config saved to /var/cache/conftool/dbconfig/20220330-164722-marostegui.json [production]
16:46 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P23854 and previous config saved to /var/cache/conftool/dbconfig/20220330-164637-ladsgroup.json [production]
16:32 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1112 (T297189)', diff saved to https://phabricator.wikimedia.org/P23853 and previous config saved to /var/cache/conftool/dbconfig/20220330-163217-marostegui.json [production]
16:31 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P23852 and previous config saved to /var/cache/conftool/dbconfig/20220330-163132-ladsgroup.json [production]
16:30 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance [production]
16:30 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance [production]
16:28 <btullis@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-test-presto1001.eqiad.wmnet [production]
16:24 <btullis@cumin1001> START - Cookbook sre.hosts.reboot-single for host an-test-presto1001.eqiad.wmnet [production]
16:21 <btullis@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-test-druid1001.eqiad.wmnet [production]
16:16 <btullis@cumin1001> START - Cookbook sre.hosts.reboot-single for host an-test-druid1001.eqiad.wmnet [production]
16:16 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1156 (T298565)', diff saved to https://phabricator.wikimedia.org/P23850 and previous config saved to /var/cache/conftool/dbconfig/20220330-161626-ladsgroup.json [production]
16:14 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1156 (T298565)', diff saved to https://phabricator.wikimedia.org/P23849 and previous config saved to /var/cache/conftool/dbconfig/20220330-161418-ladsgroup.json [production]
16:14 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance [production]
16:14 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance [production]
16:14 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1156.eqiad.wmnet with reason: Maintenance [production]