|
2021-11-22
§
|
| 07:55 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1131 (re)pooling @ 50%: Repool after HW maintenance', diff saved to https://phabricator.wikimedia.org/P17791 and previous config saved to /var/cache/conftool/dbconfig/20211122-075518-root.json |
[production] |
| 07:40 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1131 (re)pooling @ 40%: Repool after HW maintenance', diff saved to https://phabricator.wikimedia.org/P17790 and previous config saved to /var/cache/conftool/dbconfig/20211122-074015-root.json |
[production] |
| 07:30 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2095.codfw.wmnet with reason: Maintenance T296143 |
[production] |
| 07:30 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2095.codfw.wmnet with reason: Maintenance T296143 |
[production] |
| 07:28 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2147.codfw.wmnet with reason: Maintenance T296143 |
[production] |
| 07:28 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2147.codfw.wmnet with reason: Maintenance T296143 |
[production] |
| 07:28 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2140.codfw.wmnet with reason: Maintenance T296143 |
[production] |
| 07:28 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2140.codfw.wmnet with reason: Maintenance T296143 |
[production] |
| 07:28 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2138.codfw.wmnet with reason: Maintenance T296143 |
[production] |
| 07:28 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2138.codfw.wmnet with reason: Maintenance T296143 |
[production] |
| 07:28 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2137.codfw.wmnet with reason: Maintenance T296143 |
[production] |
| 07:28 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2137.codfw.wmnet with reason: Maintenance T296143 |
[production] |
| 07:28 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2136.codfw.wmnet with reason: Maintenance T296143 |
[production] |
| 07:28 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2136.codfw.wmnet with reason: Maintenance T296143 |
[production] |
| 07:28 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2119.codfw.wmnet with reason: Maintenance T296143 |
[production] |
| 07:28 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2119.codfw.wmnet with reason: Maintenance T296143 |
[production] |
| 07:28 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2106.codfw.wmnet with reason: Maintenance T296143 |
[production] |
| 07:28 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2106.codfw.wmnet with reason: Maintenance T296143 |
[production] |
| 07:28 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2090.codfw.wmnet with reason: Maintenance T296143 |
[production] |
| 07:27 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2090.codfw.wmnet with reason: Maintenance T296143 |
[production] |
| 07:27 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2073.codfw.wmnet with reason: Maintenance T296143 |
[production] |
| 07:27 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2073.codfw.wmnet with reason: Maintenance T296143 |
[production] |
| 07:27 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2110.codfw.wmnet with reason: Maintenance T296143 |
[production] |
| 07:27 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2110.codfw.wmnet with reason: Maintenance T296143 |
[production] |
| 07:25 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1131 (re)pooling @ 25%: Repool after HW maintenance', diff saved to https://phabricator.wikimedia.org/P17789 and previous config saved to /var/cache/conftool/dbconfig/20211122-072511-root.json |
[production] |
| 07:17 |
<Amir1> |
running optimize table on image table in commonswiki on codfw with replication enabled, it'll cause replication lag (T296143) |
[production] |
| 07:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1131 (re)pooling @ 20%: Repool after HW maintenance', diff saved to https://phabricator.wikimedia.org/P17788 and previous config saved to /var/cache/conftool/dbconfig/20211122-071006-root.json |
[production] |
| 06:55 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1131 (re)pooling @ 10%: Repool after HW maintenance', diff saved to https://phabricator.wikimedia.org/P17787 and previous config saved to /var/cache/conftool/dbconfig/20211122-065502-root.json |
[production] |
| 06:46 |
<marostegui> |
Revoke dump grants for scholarships database T296166 |
[production] |
| 06:39 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1131 (re)pooling @ 5%: Repool after HW maintenance', diff saved to https://phabricator.wikimedia.org/P17786 and previous config saved to /var/cache/conftool/dbconfig/20211122-063959-root.json |
[production] |
| 06:24 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1131 (re)pooling @ 1%: Repool after HW maintenance', diff saved to https://phabricator.wikimedia.org/P17785 and previous config saved to /var/cache/conftool/dbconfig/20211122-062455-root.json |
[production] |
| 03:30 |
<Amir1> |
run optimize table on db2140 for image table (T296143) |
[production] |
| 00:51 |
<wm-bot> |
<legoktm> Upgraded uatu to 0.1.7 |
[tools.mjolnir] |
| 00:50 |
<wm-bot> |
<legoktm> Upgraded mjolnir to 1.2.1 |
[tools.mjolnir] |
|
2021-11-20
§
|
| 10:01 |
<majavah> |
root@deployment-puppetdb03:~# cp /var/lib/puppet/ssl/certs/ca.pem /etc/ssl/certs/Puppet_Internal_CA.pem && systemctl restart puppetdb.service # T296125 |
[releng] |
| 08:36 |
<thcipriani> |
disabled beta-scap-sync-world |
[releng] |
| 01:02 |
<mutante> |
lists1001 - restarted apache, icinga alerts for the web UI, but recovered |
[production] |
| 00:27 |
<cdanis@cumin1001> |
END (PASS) - Cookbook sre.network.cf (exit_code=0) |
[production] |
| 00:26 |
<cdanis@cumin1001> |
START - Cookbook sre.network.cf |
[production] |
| 00:25 |
<bblack> |
lvs3005 - re-enabling puppet + pybal |
[production] |
| 00:25 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.network.cf (exit_code=0) |
[production] |
| 00:25 |
<legoktm@cumin1001> |
START - Cookbook sre.network.cf |
[production] |
| 00:24 |
<cdanis@cumin1001> |
END (PASS) - Cookbook sre.network.cf (exit_code=0) |
[production] |
| 00:23 |
<cdanis@cumin1001> |
START - Cookbook sre.network.cf |
[production] |
| 00:06 |
<bblack> |
lvs3005 - disabling puppet and stopping pybal (traffic will go to lvs3007) |
[production] |