|
2021-02-22
ยง
|
| 13:27 |
<filippo@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host prometheus3001.esams.wmnet |
[production] |
| 13:26 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host cuminunpriv1001.eqiad.wmnet |
[production] |
| 13:17 |
<dcaro@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on cloudvirt-wdqs[1001-1003].eqiad.wmnet with reason: Restarting cloudcanary instances |
[production] |
| 13:16 |
<dcaro@cumin1001> |
START - Cookbook sre.hosts.downtime for 5:00:00 on cloudvirt-wdqs[1001-1003].eqiad.wmnet with reason: Restarting cloudcanary instances |
[production] |
| 13:16 |
<dcaro@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on 27 hosts with reason: Restarting cloudcanary instances |
[production] |
| 13:16 |
<dcaro@cumin1001> |
START - Cookbook sre.hosts.downtime for 5:00:00 on 27 hosts with reason: Restarting cloudcanary instances |
[production] |
| 13:11 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1175 (re)pooling @ 100%: Slowly repool db1175', diff saved to https://phabricator.wikimedia.org/P14439 and previous config saved to /var/cache/conftool/dbconfig/20210222-131153-root.json |
[production] |
| 12:56 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1175 (re)pooling @ 75%: Slowly repool db1175', diff saved to https://phabricator.wikimedia.org/P14438 and previous config saved to /var/cache/conftool/dbconfig/20210222-125650-root.json |
[production] |
| 12:41 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1175 (re)pooling @ 50%: Slowly repool db1175', diff saved to https://phabricator.wikimedia.org/P14437 and previous config saved to /var/cache/conftool/dbconfig/20210222-124146-root.json |
[production] |
| 12:32 |
<akosiaris@cumin1001> |
conftool action : set/pooled=false; selector: name=eqiad,dnsdisc=sessionstore |
[production] |
| 12:28 |
<akosiaris@cumin1001> |
conftool action : set/pooled=true; selector: name=eqiad,dnsdisc=sessionstore |
[production] |
| 12:26 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1175 (re)pooling @ 25%: Slowly repool db1175', diff saved to https://phabricator.wikimedia.org/P14436 and previous config saved to /var/cache/conftool/dbconfig/20210222-122643-root.json |
[production] |
| 12:24 |
<urbanecm@deploy1001> |
Synchronized wmf-config//throttle.php: d806f3a986244f8027aba730e72d99babe3b37e9: Add a throttle rule for for edit-a-thon (T275237) (duration: 00m 54s) |
[production] |
| 12:22 |
<akosiaris> |
depool sessionstore in eqiad for sessionstore certificate refresh. T274564 |
[production] |
| 12:21 |
<akosiaris> |
repool sessionstore in codfw after sessionstore certificate refresh. T274564 |
[production] |
| 12:21 |
<akosiaris@cumin1001> |
conftool action : set/pooled=true; selector: name=codfw,dnsdisc=sessionstore |
[production] |
| 12:20 |
<urbanecm@deploy1001> |
Synchronized php-1.36.0-wmf.31/extensions/FlaggedRevs/extension.json: a4cd98e7a581fe18634da05ba04eaf8035023c26: Grant sysops review and unreviewed pages right by default (T275293) (duration: 00m 55s) |
[production] |
| 12:18 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: 7bd26dc6160a5bc3ba9235ce93c01e7ab9744487: Add inaturalist-open-data.s3.amazonaws.com to copyupload list (T275318) (duration: 00m 56s) |
[production] |
| 12:15 |
<urbanecm@deploy1001> |
Synchronized wmf-config/abusefilter.php: 391900b8db9ffdee8565d82c38c089843876a27b: ukwikivoyage: Enable block AbuseFilter action (T275271) (duration: 00m 55s) |
[production] |
| 12:11 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: a1f8ce48249ad457d79c57e27836ee492eb00427: Enable Section Translation on Bengali Wikipedia (T271397) (duration: 00m 56s) |
[production] |
| 12:11 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1175 (re)pooling @ 10%: Slowly repool db1175', diff saved to https://phabricator.wikimedia.org/P14435 and previous config saved to /var/cache/conftool/dbconfig/20210222-121139-root.json |
[production] |
| 12:07 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1175 for schema change', diff saved to https://phabricator.wikimedia.org/P14434 and previous config saved to /var/cache/conftool/dbconfig/20210222-120717-marostegui.json |
[production] |
| 12:05 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: 4775fb63e79501c3dba7ae4b9c3b1172d92dc0d0: Adjust CX MT threshold to 90 for Vietnamese Wikipedia (T275121) (duration: 00m 57s) |
[production] |
| 12:02 |
<moritzm> |
installing openldap security updates on serpens/seaborgium |
[production] |
| 11:59 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc1036.eqiad.wmnet |
[production] |
| 11:54 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host mc1036.eqiad.wmnet |
[production] |
| 11:53 |
<effie> |
upgrading memecached to 1.6 on mc1036 |
[production] |
| 11:50 |
<volans> |
upgrading python3-wmflib fleet wide to 0.0.7-1+deb10u1 |
[production] |
| 11:27 |
<dcaro@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 27 hosts with reason: Restarting cloudcanary instances |
[production] |
| 11:27 |
<dcaro@cumin1001> |
START - Cookbook sre.hosts.downtime for 4:00:00 on 27 hosts with reason: Restarting cloudcanary instances |
[production] |
| 11:26 |
<dcaro@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudvirt-wdqs[1001-1003].eqiad.wmnet with reason: Restarting cloudcanary instances |
[production] |
| 11:26 |
<dcaro@cumin1001> |
START - Cookbook sre.hosts.downtime for 4:00:00 on cloudvirt-wdqs[1001-1003].eqiad.wmnet with reason: Restarting cloudcanary instances |
[production] |
| 11:26 |
<dcaro@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudvirt-wdqs[1001-1003].eqiad.wmnet with reason: Restarting cloudcanary instances |
[production] |
| 11:26 |
<dcaro@cumin1001> |
START - Cookbook sre.hosts.downtime for 4:00:00 on cloudvirt-wdqs[1001-1003].eqiad.wmnet with reason: Restarting cloudcanary instances |
[production] |
| 11:22 |
<godog> |
roll restart prometheus on cloudmetrics* |
[production] |
| 11:21 |
<godog> |
roll restart prometheus on prometheus* |
[production] |
| 11:12 |
<godog> |
restart prometheus on prometheus2004 to apply changes - T273278 |
[production] |
| 11:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1166 (re)pooling @ 100%: Slowly repool db1166', diff saved to https://phabricator.wikimedia.org/P14433 and previous config saved to /var/cache/conftool/dbconfig/20210222-111032-root.json |
[production] |
| 10:55 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1166 (re)pooling @ 75%: Slowly repool db1166', diff saved to https://phabricator.wikimedia.org/P14432 and previous config saved to /var/cache/conftool/dbconfig/20210222-105528-root.json |
[production] |
| 10:49 |
<_joe_> |
removing stray old builds from compiler1003 |
[production] |
| 10:40 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1166 (re)pooling @ 50%: Slowly repool db1166', diff saved to https://phabricator.wikimedia.org/P14431 and previous config saved to /var/cache/conftool/dbconfig/20210222-104025-root.json |
[production] |
| 10:36 |
<_joe_> |
manually removed the restbase-http ipvs entry from the load balancers |
[production] |
| 10:30 |
<akosiaris@cumin1001> |
conftool action : set/pooled=false; selector: name=codfw,dnsdisc=sessionstore |
[production] |
| 10:29 |
<akosiaris> |
depool sessionstore in codfw for sessionstore certificate refresh. T274564 |
[production] |
| 10:25 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1166 (re)pooling @ 25%: Slowly repool db1166', diff saved to https://phabricator.wikimedia.org/P14430 and previous config saved to /var/cache/conftool/dbconfig/20210222-102521-root.json |
[production] |
| 10:16 |
<_joe_> |
restarting pybal on lvs1015 to pick up restbase http removal |
[production] |
| 10:12 |
<_joe_> |
restarting pybal on lvs1016 to pick up restbase http removal |
[production] |
| 10:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1166 (re)pooling @ 10%: Slowly repool db1166', diff saved to https://phabricator.wikimedia.org/P14429 and previous config saved to /var/cache/conftool/dbconfig/20210222-101018-root.json |
[production] |
| 10:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1166 for schema change', diff saved to https://phabricator.wikimedia.org/P14428 and previous config saved to /var/cache/conftool/dbconfig/20210222-100653-marostegui.json |
[production] |
| 09:51 |
<_joe_> |
restarting low-traffic pybals in codfw to remove the restbase http endpoint |
[production] |