651-700 of 10000 results (105ms)
2022-11-17 ยง
10:20 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2019.codfw.wmnet [production]
10:19 <hashar> gerrit1001: removed 5G of 2019's thread dumps in `/srv/home-cobalt.wikimedia.org/thcipriani/threaddumps` [production]
10:12 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti2019.codfw.wmnet [production]
10:03 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2018.codfw.wmnet [production]
09:56 <hashar> Stopped Gerrit and running offline reindexing [production]
09:55 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti2018.codfw.wmnet [production]
09:52 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2017.codfw.wmnet [production]
09:45 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti2017.codfw.wmnet [production]
09:42 <hashar> Cleaning gerrit1001.wikimedia.org `/` partition [production]
09:37 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2016.codfw.wmnet [production]
09:36 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2175 (T323214)', diff saved to https://phabricator.wikimedia.org/P40087 and previous config saved to /var/cache/conftool/dbconfig/20221117-093650-ladsgroup.json [production]
09:36 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2175.codfw.wmnet with reason: Maintenance [production]
09:36 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2175.codfw.wmnet with reason: Maintenance [production]
09:36 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2170:3312 (T323214)', diff saved to https://phabricator.wikimedia.org/P40086 and previous config saved to /var/cache/conftool/dbconfig/20221117-093628-ladsgroup.json [production]
09:29 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti2016.codfw.wmnet [production]
09:29 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1170:3312 (T323214)', diff saved to https://phabricator.wikimedia.org/P40085 and previous config saved to /var/cache/conftool/dbconfig/20221117-092902-ladsgroup.json [production]
09:28 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1170.eqiad.wmnet with reason: Maintenance [production]
09:28 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1170.eqiad.wmnet with reason: Maintenance [production]
09:28 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1162 (T323214)', diff saved to https://phabricator.wikimedia.org/P40084 and previous config saved to /var/cache/conftool/dbconfig/20221117-092841-ladsgroup.json [production]
09:27 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2014.codfw.wmnet [production]
09:21 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2170:3312', diff saved to https://phabricator.wikimedia.org/P40083 and previous config saved to /var/cache/conftool/dbconfig/20221117-092121-ladsgroup.json [production]
09:20 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti2014.codfw.wmnet [production]
09:13 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P40082 and previous config saved to /var/cache/conftool/dbconfig/20221117-091334-ladsgroup.json [production]
09:12 <hashar> Bringing back primary Gerrit on gerrit1001 [production]
09:11 <hashar@deploy1002> Finished deploy [gerrit/gerrit@39d9f06]: Gerrit to 3.5.4 on gerrit1001 (duration: 00m 08s) [production]
09:10 <hashar@deploy1002> Started deploy [gerrit/gerrit@39d9f06]: Gerrit to 3.5.4 on gerrit1001 [production]
09:09 <hashar> Upgrading Gerrit primary instance [production]
09:07 <hashar> Bringing back Gerrit on gerrit2002 [production]
09:06 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2170:3312', diff saved to https://phabricator.wikimedia.org/P40081 and previous config saved to /var/cache/conftool/dbconfig/20221117-090615-ladsgroup.json [production]
09:04 <hashar@deploy1002> Finished deploy [gerrit/gerrit@39d9f06]: Gerrit to 3.5.4 on gerrit2002 (duration: 00m 10s) [production]
09:04 <hashar@deploy1002> Started deploy [gerrit/gerrit@39d9f06]: Gerrit to 3.5.4 on gerrit2002 [production]
09:02 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubestagetcd2002.codfw.wmnet to plain [production]
09:02 <jmm@cumin2002> START - Cookbook sre.ganeti.changedisk for changing disk type of kubestagetcd2002.codfw.wmnet to plain [production]
08:58 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P40080 and previous config saved to /var/cache/conftool/dbconfig/20221117-085828-ladsgroup.json [production]
08:55 <krinkle@deploy1002> Finished deploy [integration/docroot@de83506]: (no justification provided) (duration: 00m 39s) [production]
08:55 <krinkle@deploy1002> Started deploy [integration/docroot@de83506]: (no justification provided) [production]
08:51 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2170:3312 (T323214)', diff saved to https://phabricator.wikimedia.org/P40079 and previous config saved to /var/cache/conftool/dbconfig/20221117-085108-ladsgroup.json [production]
08:50 <moritzm> draining ganeti1019 for eventual reimage T311687 [production]
08:43 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1162 (T323214)', diff saved to https://phabricator.wikimedia.org/P40078 and previous config saved to /var/cache/conftool/dbconfig/20221117-084321-ladsgroup.json [production]
08:31 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubestagetcd2002.codfw.wmnet to drbd [production]
08:21 <jmm@cumin2002> START - Cookbook sre.ganeti.changedisk for changing disk type of kubestagetcd2002.codfw.wmnet to drbd [production]
08:14 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1162 (T323214)', diff saved to https://phabricator.wikimedia.org/P40076 and previous config saved to /var/cache/conftool/dbconfig/20221117-081413-ladsgroup.json [production]
08:14 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1162.eqiad.wmnet with reason: Maintenance [production]
08:13 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1162.eqiad.wmnet with reason: Maintenance [production]
08:13 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1156 (T323214)', diff saved to https://phabricator.wikimedia.org/P40075 and previous config saved to /var/cache/conftool/dbconfig/20221117-081352-ladsgroup.json [production]
07:58 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P40074 and previous config saved to /var/cache/conftool/dbconfig/20221117-075845-ladsgroup.json [production]
07:47 <elukey> restart kube-apiserver on ml-serve-ctrl2002 - high LIST latencies for knative, attempt to clear them out [production]
07:47 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2170:3312 (T323214)', diff saved to https://phabricator.wikimedia.org/P40073 and previous config saved to /var/cache/conftool/dbconfig/20221117-074732-ladsgroup.json [production]
07:47 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2170.codfw.wmnet with reason: Maintenance [production]
07:47 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2170.codfw.wmnet with reason: Maintenance [production]