8651-8700 of 10000 results (63ms)
2022-03-08 ยง
10:34 <vgutierrez> pool cp2035 with HAProxy as TLS termination layer - T290005 [production]
10:34 <marostegui@cumin1001> dbctl commit (dc=all): 'db1162 (re)pooling @ 10%: repooling after schema change', diff saved to https://phabricator.wikimedia.org/P22070 and previous config saved to /var/cache/conftool/dbconfig/20220308-103409-root.json [production]
10:32 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1098:3316 (T298294)', diff saved to https://phabricator.wikimedia.org/P22069 and previous config saved to /var/cache/conftool/dbconfig/20220308-103251-marostegui.json [production]
10:32 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1098.eqiad.wmnet with reason: Maintenance [production]
10:32 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on db1098.eqiad.wmnet with reason: Maintenance [production]
10:32 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1180 (T298294)', diff saved to https://phabricator.wikimedia.org/P22068 and previous config saved to /var/cache/conftool/dbconfig/20220308-103243-marostegui.json [production]
10:30 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1112 (T300381)', diff saved to https://phabricator.wikimedia.org/P22067 and previous config saved to /var/cache/conftool/dbconfig/20220308-103017-marostegui.json [production]
10:30 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
10:30 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
10:30 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1112.eqiad.wmnet with reason: Maintenance [production]
10:30 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1112.eqiad.wmnet with reason: Maintenance [production]
10:28 <elukey@cumin1001> START - Cookbook sre.hosts.reboot-single for host ml-serve2003.codfw.wmnet [production]
10:27 <klausman@cumin2002> START - Cookbook sre.hosts.reboot-single for host ml-staging2002.codfw.wmnet [production]
10:27 <klausman@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-staging2001.codfw.wmnet [production]
10:26 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve2002.codfw.wmnet [production]
10:22 <klausman@cumin2002> START - Cookbook sre.hosts.reboot-single for host ml-staging2001.codfw.wmnet [production]
10:19 <elukey@cumin1001> START - Cookbook sre.hosts.reboot-single for host ml-serve2002.codfw.wmnet [production]
10:19 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve2001.codfw.wmnet [production]
10:17 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P22066 and previous config saved to /var/cache/conftool/dbconfig/20220308-101739-marostegui.json [production]
10:17 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp2035.codfw.wmnet with reason: host reimage [production]
10:14 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cp2035.codfw.wmnet with reason: host reimage [production]
10:12 <jmm@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cumin2002.codfw.wmnet [production]
10:10 <elukey@cumin1001> START - Cookbook sre.hosts.reboot-single for host ml-serve2001.codfw.wmnet [production]
10:06 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1102.eqiad.wmnet with reason: Maintenance [production]
10:06 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1102.eqiad.wmnet with reason: Maintenance [production]
10:06 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1179 (T300381)', diff saved to https://phabricator.wikimedia.org/P22065 and previous config saved to /var/cache/conftool/dbconfig/20220308-100559-marostegui.json [production]
10:03 <jmm@cumin1001> START - Cookbook sre.hosts.reboot-single for host cumin2002.codfw.wmnet [production]
10:02 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P22064 and previous config saved to /var/cache/conftool/dbconfig/20220308-100234-marostegui.json [production]
09:56 <vgutierrez@cumin1001> START - Cookbook sre.hosts.reimage for host cp2035.codfw.wmnet with OS buster [production]
09:50 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P22063 and previous config saved to /var/cache/conftool/dbconfig/20220308-095055-marostegui.json [production]
09:47 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1180 (T298294)', diff saved to https://phabricator.wikimedia.org/P22062 and previous config saved to /var/cache/conftool/dbconfig/20220308-094730-marostegui.json [production]
09:46 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1180 (T298294)', diff saved to https://phabricator.wikimedia.org/P22061 and previous config saved to /var/cache/conftool/dbconfig/20220308-094613-marostegui.json [production]
09:46 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1180.eqiad.wmnet with reason: Maintenance [production]
09:46 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on db1180.eqiad.wmnet with reason: Maintenance [production]
09:46 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1165 (T298294)', diff saved to https://phabricator.wikimedia.org/P22060 and previous config saved to /var/cache/conftool/dbconfig/20220308-094605-marostegui.json [production]
09:43 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1174 (T300775)', diff saved to https://phabricator.wikimedia.org/P22059 and previous config saved to /var/cache/conftool/dbconfig/20220308-094354-marostegui.json [production]
09:43 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db1174.eqiad.wmnet with reason: Maintenance [production]
09:43 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db1174.eqiad.wmnet with reason: Maintenance [production]
09:41 <marostegui@cumin1001> dbctl commit (dc=all): 'db1158 (re)pooling @ 100%: After schema change', diff saved to https://phabricator.wikimedia.org/P22058 and previous config saved to /var/cache/conftool/dbconfig/20220308-094155-root.json [production]
09:35 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P22057 and previous config saved to /var/cache/conftool/dbconfig/20220308-093550-marostegui.json [production]
09:34 <jayme@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kubernetes2022.codfw.wmnet [production]
09:31 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P22056 and previous config saved to /var/cache/conftool/dbconfig/20220308-093101-marostegui.json [production]
09:27 <jayme@cumin1001> START - Cookbook sre.hosts.reboot-single for host kubernetes2022.codfw.wmnet [production]
09:26 <marostegui@cumin1001> dbctl commit (dc=all): 'db1158 (re)pooling @ 75%: After schema change', diff saved to https://phabricator.wikimedia.org/P22055 and previous config saved to /var/cache/conftool/dbconfig/20220308-092651-root.json [production]
09:26 <jayme@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kubernetes2021.codfw.wmnet [production]
09:20 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1179 (T300381)', diff saved to https://phabricator.wikimedia.org/P22054 and previous config saved to /var/cache/conftool/dbconfig/20220308-092045-marostegui.json [production]
09:18 <jayme@cumin1001> START - Cookbook sre.hosts.reboot-single for host kubernetes2021.codfw.wmnet [production]
09:15 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P22053 and previous config saved to /var/cache/conftool/dbconfig/20220308-091556-marostegui.json [production]
09:11 <marostegui@cumin1001> dbctl commit (dc=all): 'db1158 (re)pooling @ 50%: After schema change', diff saved to https://phabricator.wikimedia.org/P22052 and previous config saved to /var/cache/conftool/dbconfig/20220308-091147-root.json [production]
09:10 <jayme@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kubernetes2020.codfw.wmnet [production]