|
2023-02-08
ยง
|
| 18:42 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.reimage for host mw2420.codfw.wmnet with OS buster |
[production] |
| 18:39 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2170:3312', diff saved to https://phabricator.wikimedia.org/P43866 and previous config saved to /var/cache/conftool/dbconfig/20230208-183945-marostegui.json |
[production] |
| 18:39 |
<mutante> |
adding az.wikimedia.org to DNS - approved by affcom T306015 |
[production] |
| 18:35 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2105 (T328255)', diff saved to https://phabricator.wikimedia.org/P43865 and previous config saved to /var/cache/conftool/dbconfig/20230208-183556-ladsgroup.json |
[production] |
| 18:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P43864 and previous config saved to /var/cache/conftool/dbconfig/20230208-183336-marostegui.json |
[production] |
| 18:24 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2170:3312', diff saved to https://phabricator.wikimedia.org/P43863 and previous config saved to /var/cache/conftool/dbconfig/20230208-182439-marostegui.json |
[production] |
| 18:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1122 (T328817)', diff saved to https://phabricator.wikimedia.org/P43862 and previous config saved to /var/cache/conftool/dbconfig/20230208-181829-marostegui.json |
[production] |
| 18:12 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1122 (T328817)', diff saved to https://phabricator.wikimedia.org/P43861 and previous config saved to /var/cache/conftool/dbconfig/20230208-181222-marostegui.json |
[production] |
| 18:12 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1122.eqiad.wmnet with reason: Maintenance |
[production] |
| 18:12 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1122.eqiad.wmnet with reason: Maintenance |
[production] |
| 18:12 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T328817)', diff saved to https://phabricator.wikimedia.org/P43860 and previous config saved to /var/cache/conftool/dbconfig/20230208-181200-marostegui.json |
[production] |
| 18:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2170:3312 (T328817)', diff saved to https://phabricator.wikimedia.org/P43859 and previous config saved to /var/cache/conftool/dbconfig/20230208-180933-marostegui.json |
[production] |
| 18:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db2170:3312 (T328817)', diff saved to https://phabricator.wikimedia.org/P43858 and previous config saved to /var/cache/conftool/dbconfig/20230208-180216-marostegui.json |
[production] |
| 18:02 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2170.codfw.wmnet with reason: Maintenance |
[production] |
| 18:02 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2170.codfw.wmnet with reason: Maintenance |
[production] |
| 18:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2148 (T328817)', diff saved to https://phabricator.wikimedia.org/P43857 and previous config saved to /var/cache/conftool/dbconfig/20230208-180153-marostegui.json |
[production] |
| 17:56 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P43856 and previous config saved to /var/cache/conftool/dbconfig/20230208-175654-marostegui.json |
[production] |
| 17:46 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P43855 and previous config saved to /var/cache/conftool/dbconfig/20230208-174647-marostegui.json |
[production] |
| 17:41 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P43854 and previous config saved to /var/cache/conftool/dbconfig/20230208-174148-marostegui.json |
[production] |
| 17:20 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1105:3312 (T328817)', diff saved to https://phabricator.wikimedia.org/P43850 and previous config saved to /var/cache/conftool/dbconfig/20230208-172021-marostegui.json |
[production] |
| 17:20 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1105.eqiad.wmnet with reason: Maintenance |
[production] |
| 17:20 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1105.eqiad.wmnet with reason: Maintenance |
[production] |
| 17:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2148 (T328817)', diff saved to https://phabricator.wikimedia.org/P43849 and previous config saved to /var/cache/conftool/dbconfig/20230208-171634-marostegui.json |
[production] |
| 17:15 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1102.eqiad.wmnet with reason: Maintenance |
[production] |
| 17:15 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1102.eqiad.wmnet with reason: Maintenance |
[production] |
| 17:14 |
<cmooney@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on rpki2002.codfw.wmnet with reason: Restarting to increase VM RAM allocation |
[production] |
| 17:13 |
<cmooney@cumin1001> |
START - Cookbook sre.hosts.downtime for 0:15:00 on rpki2002.codfw.wmnet with reason: Restarting to increase VM RAM allocation |
[production] |
| 17:11 |
<oblivian@cumin2002> |
END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) status all services in codfw: maintenance |
[production] |
| 17:11 |
<oblivian@cumin2002> |
START - Cookbook sre.discovery.datacenter status all services in codfw: maintenance |
[production] |
| 17:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db2148 (T328817)', diff saved to https://phabricator.wikimedia.org/P43848 and previous config saved to /var/cache/conftool/dbconfig/20230208-171028-marostegui.json |
[production] |
| 17:10 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2148.codfw.wmnet with reason: Maintenance |
[production] |
| 17:10 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2148.codfw.wmnet with reason: Maintenance |
[production] |
| 17:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2138:3312 (T328817)', diff saved to https://phabricator.wikimedia.org/P43847 and previous config saved to /var/cache/conftool/dbconfig/20230208-171006-marostegui.json |
[production] |
| 17:09 |
<jynus> |
disable bacula job backup1002.eqiad.wmnet-Weekly-Thu-EsRwCodfw-mysql-srv-backups-dumps-latest |
[production] |
| 17:08 |
<oblivian@cumin2002> |
END (FAIL) - Cookbook sre.discovery.datacenter (exit_code=93) status all services in codfw: maintenance |
[production] |
| 17:08 |
<oblivian@cumin2002> |
START - Cookbook sre.discovery.datacenter status all services in codfw: maintenance |
[production] |
| 16:55 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2138:3312', diff saved to https://phabricator.wikimedia.org/P43844 and previous config saved to /var/cache/conftool/dbconfig/20230208-165500-marostegui.json |
[production] |
| 16:47 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1145.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:47 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1145.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:45 |
<ladsgroup@cumin1001> |
END (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 12:00:00 on db1145.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:45 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1145.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:44 |
<sukhe> |
[done] rolling restart of haproxy and trafficserver in A:cp |
[production] |
| 16:44 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1102.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:43 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1102.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:39 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2138:3312', diff saved to https://phabricator.wikimedia.org/P43842 and previous config saved to /var/cache/conftool/dbconfig/20230208-163954-marostegui.json |
[production] |
| 16:27 |
<sukhe> |
rolling restart of haproxy and trafficserver in A:cp |
[production] |
| 16:24 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2138:3312 (T328817)', diff saved to https://phabricator.wikimedia.org/P43841 and previous config saved to /var/cache/conftool/dbconfig/20230208-162447-marostegui.json |
[production] |
| 16:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db2138:3312 (T328817)', diff saved to https://phabricator.wikimedia.org/P43840 and previous config saved to /var/cache/conftool/dbconfig/20230208-161828-marostegui.json |
[production] |
| 16:18 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2138.codfw.wmnet with reason: Maintenance |
[production] |
| 16:18 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2138.codfw.wmnet with reason: Maintenance |
[production] |