|
2022-05-04
ยง
|
| 17:57 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1174 (T307525)', diff saved to https://phabricator.wikimedia.org/P27497 and previous config saved to /var/cache/conftool/dbconfig/20220504-175747-ladsgroup.json |
[production] |
| 17:51 |
<dduvall> |
created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789225 |
[releng] |
| 17:43 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1174 (T307525)', diff saved to https://phabricator.wikimedia.org/P27496 and previous config saved to /var/cache/conftool/dbconfig/20220504-174325-ladsgroup.json |
[production] |
| 17:43 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1174.eqiad.wmnet with reason: Maintenance |
[production] |
| 17:43 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1174.eqiad.wmnet with reason: Maintenance |
[production] |
| 17:43 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1158 (T307525)', diff saved to https://phabricator.wikimedia.org/P27495 and previous config saved to /var/cache/conftool/dbconfig/20220504-174317-ladsgroup.json |
[production] |
| 17:42 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P27494 and previous config saved to /var/cache/conftool/dbconfig/20220504-174244-ladsgroup.json |
[production] |
| 17:28 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P27493 and previous config saved to /var/cache/conftool/dbconfig/20220504-172810-ladsgroup.json |
[production] |
| 17:27 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 (T307525)', diff saved to https://phabricator.wikimedia.org/P27492 and previous config saved to /var/cache/conftool/dbconfig/20220504-172739-ladsgroup.json |
[production] |
| 17:22 |
<dduvall> |
Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789218 |
[releng] |
| 17:22 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1113:3315 (T307525)', diff saved to https://phabricator.wikimedia.org/P27491 and previous config saved to /var/cache/conftool/dbconfig/20220504-172214-ladsgroup.json |
[production] |
| 17:22 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance |
[production] |
| 17:22 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance |
[production] |
| 17:13 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P27490 and previous config saved to /var/cache/conftool/dbconfig/20220504-171305-ladsgroup.json |
[production] |
| 17:12 |
<dduvall> |
created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789217 |
[releng] |
| 16:58 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1158 (T307525)', diff saved to https://phabricator.wikimedia.org/P27489 and previous config saved to /var/cache/conftool/dbconfig/20220504-165800-ladsgroup.json |
[production] |
| 16:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1132 (re)pooling @ 100%: After optimizing a table', diff saved to https://phabricator.wikimedia.org/P27488 and previous config saved to /var/cache/conftool/dbconfig/20220504-164806-root.json |
[production] |
| 16:44 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1158 (T307525)', diff saved to https://phabricator.wikimedia.org/P27487 and previous config saved to /var/cache/conftool/dbconfig/20220504-164448-ladsgroup.json |
[production] |
| 16:44 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:44 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:44 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1158.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:44 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1158.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:44 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 (T307525)', diff saved to https://phabricator.wikimedia.org/P27486 and previous config saved to /var/cache/conftool/dbconfig/20220504-164434-ladsgroup.json |
[production] |
| 16:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1132 (re)pooling @ 75%: After optimizing a table', diff saved to https://phabricator.wikimedia.org/P27485 and previous config saved to /var/cache/conftool/dbconfig/20220504-163302-root.json |
[production] |
| 16:32 |
<razzi@cumin1001> |
START - Cookbook sre.kafka.reboot-workers for Kafka jumbo-eqiad cluster: Reboot kafka nodes |
[production] |
| 16:29 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P27484 and previous config saved to /var/cache/conftool/dbconfig/20220504-162929-ladsgroup.json |
[production] |
| 16:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1132 (re)pooling @ 50%: After optimizing a table', diff saved to https://phabricator.wikimedia.org/P27483 and previous config saved to /var/cache/conftool/dbconfig/20220504-161758-root.json |
[production] |
| 16:14 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P27482 and previous config saved to /var/cache/conftool/dbconfig/20220504-161424-ladsgroup.json |
[production] |
| 16:11 |
<dduvall> |
Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789204 |
[releng] |
| 16:03 |
<sukhe@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5002.eqsin.wmnet with OS buster |
[production] |
| 16:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1132 (re)pooling @ 25%: After optimizing a table', diff saved to https://phabricator.wikimedia.org/P27481 and previous config saved to /var/cache/conftool/dbconfig/20220504-160255-root.json |
[production] |
| 16:02 |
<bd808> |
Updated demo server to 6a179a |
[toolhub] |
| 16:01 |
<dduvall> |
created 2 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789203 |
[releng] |
| 16:01 |
<dduvall> |
Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789195 |
[releng] |
| 15:59 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 (T307525)', diff saved to https://phabricator.wikimedia.org/P27480 and previous config saved to /var/cache/conftool/dbconfig/20220504-155919-ladsgroup.json |
[production] |
| 15:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1132 (re)pooling @ 10%: After optimizing a table', diff saved to https://phabricator.wikimedia.org/P27479 and previous config saved to /var/cache/conftool/dbconfig/20220504-154751-root.json |
[production] |
| 15:42 |
<dduvall> |
created 2 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789194 |
[releng] |
| 15:35 |
<mvernon@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2048.codfw.wmnet with OS bullseye |
[production] |
| 15:35 |
<wm-bot> |
<root> Restart of webservice. Errors reported on IRC. Logs have lots of uwsgi queue exhaustion errors. |
[tools.whois-referral] |
| 15:32 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1132 (re)pooling @ 5%: After optimizing a table', diff saved to https://phabricator.wikimedia.org/P27478 and previous config saved to /var/cache/conftool/dbconfig/20220504-153247-root.json |
[production] |
| 15:16 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1170:3317 (T307525)', diff saved to https://phabricator.wikimedia.org/P27477 and previous config saved to /var/cache/conftool/dbconfig/20220504-151630-ladsgroup.json |
[production] |
| 15:16 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance |
[production] |
| 15:16 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance |
[production] |
| 15:16 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 (T307525)', diff saved to https://phabricator.wikimedia.org/P27476 and previous config saved to /var/cache/conftool/dbconfig/20220504-151606-ladsgroup.json |
[production] |
| 15:16 |
<sukhe@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5002.eqsin.wmnet with reason: host reimage |
[production] |
| 15:13 |
<mvernon@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2048.codfw.wmnet with reason: host reimage |
[production] |
| 15:11 |
<sukhe@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cp5002.eqsin.wmnet with reason: host reimage |
[production] |
| 15:10 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance |
[production] |
| 15:10 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance |
[production] |
| 15:10 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1110 (T307525)', diff saved to https://phabricator.wikimedia.org/P27475 and previous config saved to /var/cache/conftool/dbconfig/20220504-151048-ladsgroup.json |
[production] |