5751-5800 of 10000 results (62ms)
2022-05-01 §
20:44 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
20:44 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance [production]
20:44 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance [production]
13:51 <cwhite> reload nginx on conf1004/1005 to pick up cert changes [production]
06:37 <cwhite> restart etcdmirror-conftool.eqiad.wmnet on conf2005 [production]
05:44 <mutante> puppetmaster1001 - sudo puppet cert clean etcd.eqiad.wmnet (expired) [production]
02:32 <Krinkle> repool mw1340 [production]
2022-04-29 §
23:11 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1149 (T306560)', diff saved to https://phabricator.wikimedia.org/P27163 and previous config saved to /var/cache/conftool/dbconfig/20220429-231136-ladsgroup.json [production]
22:56 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P27162 and previous config saved to /var/cache/conftool/dbconfig/20220429-225631-ladsgroup.json [production]
22:49 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs2003.codfw.wmnet with OS bullseye [production]
22:48 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host aqs2004.codfw.wmnet with OS bullseye [production]
22:41 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P27161 and previous config saved to /var/cache/conftool/dbconfig/20220429-224125-ladsgroup.json [production]
22:37 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs2003.codfw.wmnet with reason: host reimage [production]
22:33 <pt1979@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on aqs2003.codfw.wmnet with reason: host reimage [production]
22:28 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host aqs2003.codfw.wmnet with OS bullseye [production]
22:26 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1149 (T306560)', diff saved to https://phabricator.wikimedia.org/P27160 and previous config saved to /var/cache/conftool/dbconfig/20220429-222620-ladsgroup.json [production]
22:25 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host aqs2003.codfw.wmnet with OS bullseye [production]
22:16 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host aqs2004.codfw.wmnet with OS bullseye [production]
22:15 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host aqs2004.codfw.wmnet with OS bullseye [production]
22:15 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host aqs2004.codfw.wmnet with OS bullseye [production]
22:13 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1149 (T306560)', diff saved to https://phabricator.wikimedia.org/P27159 and previous config saved to /var/cache/conftool/dbconfig/20220429-221331-ladsgroup.json [production]
22:13 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1149.eqiad.wmnet with reason: Maintenance [production]
22:13 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1149.eqiad.wmnet with reason: Maintenance [production]
22:13 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1148 (T306560)', diff saved to https://phabricator.wikimedia.org/P27158 and previous config saved to /var/cache/conftool/dbconfig/20220429-221323-ladsgroup.json [production]
21:58 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P27157 and previous config saved to /var/cache/conftool/dbconfig/20220429-215818-ladsgroup.json [production]
21:54 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host aqs2003.codfw.wmnet with OS bullseye [production]
21:43 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P27156 and previous config saved to /var/cache/conftool/dbconfig/20220429-214313-ladsgroup.json [production]
21:42 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs2002.codfw.wmnet with OS bullseye [production]
21:41 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance [production]
21:41 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance [production]
21:29 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs2002.codfw.wmnet with reason: host reimage [production]
21:28 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1148 (T306560)', diff saved to https://phabricator.wikimedia.org/P27155 and previous config saved to /var/cache/conftool/dbconfig/20220429-212808-ladsgroup.json [production]
21:27 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 6 hosts with reason: Maintenance [production]
21:27 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on 6 hosts with reason: Maintenance [production]
21:27 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2105.codfw.wmnet with reason: Maintenance [production]
21:26 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db2105.codfw.wmnet with reason: Maintenance [production]
21:26 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1175 (T306560)', diff saved to https://phabricator.wikimedia.org/P27154 and previous config saved to /var/cache/conftool/dbconfig/20220429-212652-ladsgroup.json [production]
21:26 <pt1979@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on aqs2002.codfw.wmnet with reason: host reimage [production]
21:21 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host aqs2002.codfw.wmnet with OS bullseye [production]
21:16 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1148 (T306560)', diff saved to https://phabricator.wikimedia.org/P27153 and previous config saved to /var/cache/conftool/dbconfig/20220429-211609-ladsgroup.json [production]
21:16 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1148.eqiad.wmnet with reason: Maintenance [production]
21:16 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1148.eqiad.wmnet with reason: Maintenance [production]
21:16 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1143 (T306560)', diff saved to https://phabricator.wikimedia.org/P27152 and previous config saved to /var/cache/conftool/dbconfig/20220429-211601-ladsgroup.json [production]
21:11 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P27151 and previous config saved to /var/cache/conftool/dbconfig/20220429-211146-ladsgroup.json [production]
21:00 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P27150 and previous config saved to /var/cache/conftool/dbconfig/20220429-210055-ladsgroup.json [production]
20:56 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P27149 and previous config saved to /var/cache/conftool/dbconfig/20220429-205641-ladsgroup.json [production]
20:46 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host aqs2002.codfw.wmnet with OS bullseye [production]
20:45 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P27148 and previous config saved to /var/cache/conftool/dbconfig/20220429-204550-ladsgroup.json [production]
20:41 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1175 (T306560)', diff saved to https://phabricator.wikimedia.org/P27147 and previous config saved to /var/cache/conftool/dbconfig/20220429-204136-ladsgroup.json [production]
20:31 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs2001.codfw.wmnet with OS bullseye [production]