2801-2850 of 10000 results (83ms)
2022-11-17 ยง
11:20 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2022.codfw.wmnet [production]
11:17 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1182 (T323214)', diff saved to https://phabricator.wikimedia.org/P40095 and previous config saved to /var/cache/conftool/dbconfig/20221117-111745-ladsgroup.json [production]
11:17 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1182.eqiad.wmnet with reason: Maintenance [production]
11:17 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1182.eqiad.wmnet with reason: Maintenance [production]
11:17 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T323214)', diff saved to https://phabricator.wikimedia.org/P40094 and previous config saved to /var/cache/conftool/dbconfig/20221117-111712-ladsgroup.json [production]
11:13 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti2022.codfw.wmnet [production]
11:08 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P40093 and previous config saved to /var/cache/conftool/dbconfig/20221117-110801-ladsgroup.json [production]
11:02 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2021.codfw.wmnet [production]
11:02 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P40092 and previous config saved to /var/cache/conftool/dbconfig/20221117-110206-ladsgroup.json [production]
10:55 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti2021.codfw.wmnet [production]
10:52 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2175 (T323214)', diff saved to https://phabricator.wikimedia.org/P40091 and previous config saved to /var/cache/conftool/dbconfig/20221117-105254-ladsgroup.json [production]
10:47 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P40090 and previous config saved to /var/cache/conftool/dbconfig/20221117-104659-ladsgroup.json [production]
10:45 <moritzm> restarting apache/FPM on mw canaries to pick up gnutls security updates [production]
10:31 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T323214)', diff saved to https://phabricator.wikimedia.org/P40089 and previous config saved to /var/cache/conftool/dbconfig/20221117-103153-ladsgroup.json [production]
10:25 <vgutierrez> pool ats-be@cp2042 [production]
10:20 <moritzm> installing gnutls28 security updates on Buster [production]
10:20 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2019.codfw.wmnet [production]
10:19 <hashar> gerrit1001: removed 5G of 2019's thread dumps in `/srv/home-cobalt.wikimedia.org/thcipriani/threaddumps` [production]
10:12 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti2019.codfw.wmnet [production]
10:03 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2018.codfw.wmnet [production]
09:56 <hashar> Stopped Gerrit and running offline reindexing [production]
09:55 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti2018.codfw.wmnet [production]
09:52 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2017.codfw.wmnet [production]
09:45 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti2017.codfw.wmnet [production]
09:42 <hashar> Cleaning gerrit1001.wikimedia.org `/` partition [production]
09:37 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2016.codfw.wmnet [production]
09:36 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2175 (T323214)', diff saved to https://phabricator.wikimedia.org/P40087 and previous config saved to /var/cache/conftool/dbconfig/20221117-093650-ladsgroup.json [production]
09:36 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2175.codfw.wmnet with reason: Maintenance [production]
09:36 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2175.codfw.wmnet with reason: Maintenance [production]
09:36 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2170:3312 (T323214)', diff saved to https://phabricator.wikimedia.org/P40086 and previous config saved to /var/cache/conftool/dbconfig/20221117-093628-ladsgroup.json [production]
09:29 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti2016.codfw.wmnet [production]
09:29 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1170:3312 (T323214)', diff saved to https://phabricator.wikimedia.org/P40085 and previous config saved to /var/cache/conftool/dbconfig/20221117-092902-ladsgroup.json [production]
09:28 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1170.eqiad.wmnet with reason: Maintenance [production]
09:28 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1170.eqiad.wmnet with reason: Maintenance [production]
09:28 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1162 (T323214)', diff saved to https://phabricator.wikimedia.org/P40084 and previous config saved to /var/cache/conftool/dbconfig/20221117-092841-ladsgroup.json [production]
09:27 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2014.codfw.wmnet [production]
09:21 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2170:3312', diff saved to https://phabricator.wikimedia.org/P40083 and previous config saved to /var/cache/conftool/dbconfig/20221117-092121-ladsgroup.json [production]
09:20 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti2014.codfw.wmnet [production]
09:13 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P40082 and previous config saved to /var/cache/conftool/dbconfig/20221117-091334-ladsgroup.json [production]
09:12 <hashar> Bringing back primary Gerrit on gerrit1001 [production]
09:11 <hashar@deploy1002> Finished deploy [gerrit/gerrit@39d9f06]: Gerrit to 3.5.4 on gerrit1001 (duration: 00m 08s) [production]
09:10 <hashar@deploy1002> Started deploy [gerrit/gerrit@39d9f06]: Gerrit to 3.5.4 on gerrit1001 [production]
09:09 <hashar> Upgrading Gerrit primary instance [production]
09:07 <hashar> Bringing back Gerrit on gerrit2002 [production]
09:06 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2170:3312', diff saved to https://phabricator.wikimedia.org/P40081 and previous config saved to /var/cache/conftool/dbconfig/20221117-090615-ladsgroup.json [production]
09:04 <hashar@deploy1002> Finished deploy [gerrit/gerrit@39d9f06]: Gerrit to 3.5.4 on gerrit2002 (duration: 00m 10s) [production]
09:04 <hashar@deploy1002> Started deploy [gerrit/gerrit@39d9f06]: Gerrit to 3.5.4 on gerrit2002 [production]
09:02 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubestagetcd2002.codfw.wmnet to plain [production]
09:02 <jmm@cumin2002> START - Cookbook sre.ganeti.changedisk for changing disk type of kubestagetcd2002.codfw.wmnet to plain [production]
08:58 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P40080 and previous config saved to /var/cache/conftool/dbconfig/20221117-085828-ladsgroup.json [production]