551-600 of 10000 results (88ms)
2023-08-23 ยง
06:25 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2127', diff saved to https://phabricator.wikimedia.org/P51028 and previous config saved to /var/cache/conftool/dbconfig/20230823-062513-ladsgroup.json [production]
06:22 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P51027 and previous config saved to /var/cache/conftool/dbconfig/20230823-062227-ladsgroup.json [production]
06:21 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1178 (T344589)', diff saved to https://phabricator.wikimedia.org/P51026 and previous config saved to /var/cache/conftool/dbconfig/20230823-062136-ladsgroup.json [production]
06:21 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1178.eqiad.wmnet with reason: Maintenance [production]
06:21 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1178.eqiad.wmnet with reason: Maintenance [production]
06:21 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1177 (T344589)', diff saved to https://phabricator.wikimedia.org/P51025 and previous config saved to /var/cache/conftool/dbconfig/20230823-062112-ladsgroup.json [production]
06:20 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2163 (T344589)', diff saved to https://phabricator.wikimedia.org/P51024 and previous config saved to /var/cache/conftool/dbconfig/20230823-062038-ladsgroup.json [production]
06:20 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2163.codfw.wmnet with reason: Maintenance [production]
06:20 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2163.codfw.wmnet with reason: Maintenance [production]
06:20 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2162 (T344589)', diff saved to https://phabricator.wikimedia.org/P51023 and previous config saved to /var/cache/conftool/dbconfig/20230823-062013-ladsgroup.json [production]
06:10 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2127', diff saved to https://phabricator.wikimedia.org/P51022 and previous config saved to /var/cache/conftool/dbconfig/20230823-061007-ladsgroup.json [production]
06:07 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P51021 and previous config saved to /var/cache/conftool/dbconfig/20230823-060721-ladsgroup.json [production]
06:06 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P51020 and previous config saved to /var/cache/conftool/dbconfig/20230823-060606-ladsgroup.json [production]
06:05 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2162', diff saved to https://phabricator.wikimedia.org/P51019 and previous config saved to /var/cache/conftool/dbconfig/20230823-060506-ladsgroup.json [production]
05:55 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2127 (T343718)', diff saved to https://phabricator.wikimedia.org/P51018 and previous config saved to /var/cache/conftool/dbconfig/20230823-055500-ladsgroup.json [production]
05:52 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1175 (T343718)', diff saved to https://phabricator.wikimedia.org/P51017 and previous config saved to /var/cache/conftool/dbconfig/20230823-055215-ladsgroup.json [production]
05:51 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P51016 and previous config saved to /var/cache/conftool/dbconfig/20230823-055059-ladsgroup.json [production]
05:50 <zabe@deploy1002> Backport cancelled. [production]
05:50 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2162', diff saved to https://phabricator.wikimedia.org/P51015 and previous config saved to /var/cache/conftool/dbconfig/20230823-055000-ladsgroup.json [production]
05:41 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance es1031 (T344589)', diff saved to https://phabricator.wikimedia.org/P51014 and previous config saved to /var/cache/conftool/dbconfig/20230823-054144-ladsgroup.json [production]
05:41 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1175.eqiad.wmnet with reason: Maintenance [production]
05:41 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1175.eqiad.wmnet with reason: Maintenance [production]
05:41 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1166 (T343718)', diff saved to https://phabricator.wikimedia.org/P51013 and previous config saved to /var/cache/conftool/dbconfig/20230823-054124-ladsgroup.json [production]
05:35 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1177 (T344589)', diff saved to https://phabricator.wikimedia.org/P51012 and previous config saved to /var/cache/conftool/dbconfig/20230823-053553-ladsgroup.json [production]
05:34 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2162 (T344589)', diff saved to https://phabricator.wikimedia.org/P51011 and previous config saved to /var/cache/conftool/dbconfig/20230823-053454-ladsgroup.json [production]
05:29 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1177 (T344589)', diff saved to https://phabricator.wikimedia.org/P51010 and previous config saved to /var/cache/conftool/dbconfig/20230823-052939-ladsgroup.json [production]
05:29 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1177.eqiad.wmnet with reason: Maintenance [production]
05:29 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1177.eqiad.wmnet with reason: Maintenance [production]
05:29 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1172 (T344589)', diff saved to https://phabricator.wikimedia.org/P51009 and previous config saved to /var/cache/conftool/dbconfig/20230823-052915-ladsgroup.json [production]
05:28 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2162 (T344589)', diff saved to https://phabricator.wikimedia.org/P51008 and previous config saved to /var/cache/conftool/dbconfig/20230823-052834-ladsgroup.json [production]
05:28 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2162.codfw.wmnet with reason: Maintenance [production]
05:28 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2162.codfw.wmnet with reason: Maintenance [production]
05:28 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2154 (T344589)', diff saved to https://phabricator.wikimedia.org/P51007 and previous config saved to /var/cache/conftool/dbconfig/20230823-052809-ladsgroup.json [production]
05:26 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance es1031', diff saved to https://phabricator.wikimedia.org/P51006 and previous config saved to /var/cache/conftool/dbconfig/20230823-052637-ladsgroup.json [production]
05:26 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P51005 and previous config saved to /var/cache/conftool/dbconfig/20230823-052618-ladsgroup.json [production]
05:14 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P51004 and previous config saved to /var/cache/conftool/dbconfig/20230823-051409-ladsgroup.json [production]
05:13 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2127 (T343718)', diff saved to https://phabricator.wikimedia.org/P51003 and previous config saved to /var/cache/conftool/dbconfig/20230823-051312-ladsgroup.json [production]
05:13 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2127.codfw.wmnet with reason: Maintenance [production]
05:13 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P51002 and previous config saved to /var/cache/conftool/dbconfig/20230823-051303-ladsgroup.json [production]
05:12 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2127.codfw.wmnet with reason: Maintenance [production]
05:12 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2109 (T343718)', diff saved to https://phabricator.wikimedia.org/P51001 and previous config saved to /var/cache/conftool/dbconfig/20230823-051251-ladsgroup.json [production]
05:11 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance es1031', diff saved to https://phabricator.wikimedia.org/P51000 and previous config saved to /var/cache/conftool/dbconfig/20230823-051131-ladsgroup.json [production]
05:11 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P50999 and previous config saved to /var/cache/conftool/dbconfig/20230823-051112-ladsgroup.json [production]
04:59 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P50998 and previous config saved to /var/cache/conftool/dbconfig/20230823-045902-ladsgroup.json [production]
04:57 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P50997 and previous config saved to /var/cache/conftool/dbconfig/20230823-045757-ladsgroup.json [production]
04:57 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P50996 and previous config saved to /var/cache/conftool/dbconfig/20230823-045744-ladsgroup.json [production]
04:56 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance es1031 (T344589)', diff saved to https://phabricator.wikimedia.org/P50995 and previous config saved to /var/cache/conftool/dbconfig/20230823-045625-ladsgroup.json [production]
04:56 <ryankemper@cumin1001> START - Cookbook sre.wdqs.reboot [production]
04:56 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1166 (T343718)', diff saved to https://phabricator.wikimedia.org/P50994 and previous config saved to /var/cache/conftool/dbconfig/20230823-045606-ladsgroup.json [production]
04:53 <ryankemper@cumin1001> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster relforge: relforge cluster reboot - ryankemper@cumin1001 - T344587 [production]