|
2020-07-21
ยง
|
| 17:50 |
<volans@cumin1001> |
END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) |
[production] |
| 17:45 |
<volans@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 17:10 |
<jhuneidi@deploy1001> |
Pruned MediaWiki: 1.35.0-wmf.39 (duration: 16m 25s) |
[production] |
| 16:32 |
<ppchelko@deploy1001> |
Finished deploy [restbase/deploy@4f3cb41]: Add new wikis to RESTBase, take 2 (duration: 04m 54s) |
[production] |
| 16:27 |
<ppchelko@deploy1001> |
Started deploy [restbase/deploy@4f3cb41]: Add new wikis to RESTBase, take 2 |
[production] |
| 16:27 |
<ppchelko@deploy1001> |
Finished deploy [restbase/deploy@4f3cb41]: Add new wikis to RESTBase (duration: 10m 37s) |
[production] |
| 16:21 |
<longma> |
1.36.0-wmf.1 was branched at 3a1faac3764ecae8dde813bd67a5a8e8f4975a85 for T257969 |
[production] |
| 16:16 |
<ppchelko@deploy1001> |
Started deploy [restbase/deploy@4f3cb41]: Add new wikis to RESTBase |
[production] |
| 15:16 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 15:12 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 15:10 |
<moritzm> |
draining restbase1027 for eventual reboot for kernel security update |
[production] |
| 15:09 |
<godog> |
poweroff ms-be1024 for bbu replacement - T257949 |
[production] |
| 15:08 |
<filippo@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 15:08 |
<filippo@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 15:04 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 15:01 |
<vgutierrez> |
show a synthetic warning for traffic using ECDHE-RSA-AES128-SHA - T258405 |
[production] |
| 15:01 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 15:00 |
<moritzm> |
draining restbase1026 for eventual reboot for kernel security update |
[production] |
| 14:57 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:57 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:56 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 14:52 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 14:51 |
<moritzm> |
draining restbase1025 for eventual reboot for kernel security update |
[production] |
| 14:48 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 14:44 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 14:35 |
<akosiaris@cumin1001> |
conftool action : set/weight=10; selector: dc=codfw,service=mobileapps,name=scb.* |
[production] |
| 14:35 |
<akosiaris> |
decrease codfw mobileapps kubernetes traffic to 72% T218733. Weird latency patterns exhibited when 92% was reached. See https://grafana.wikimedia.org/d/5CmeRcnMz/mobileapps?panelId=34&fullscreen&orgId=1&from=1595338489749&to=1595342071227&var-dc=codfw%20prometheus%2Fk8s&var-service=mobileapps&var-container_name=All |
[production] |
| 14:35 |
<moritzm> |
draining restbase1024 for eventual reboot for kernel security update |
[production] |
| 14:32 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1119', diff saved to https://phabricator.wikimedia.org/P11994 and previous config saved to /var/cache/conftool/dbconfig/20200721-143204-marostegui.json |
[production] |
| 14:26 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1119', diff saved to https://phabricator.wikimedia.org/P11993 and previous config saved to /var/cache/conftool/dbconfig/20200721-142634-marostegui.json |
[production] |
| 14:24 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:24 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:23 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 14:19 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 14:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1119', diff saved to https://phabricator.wikimedia.org/P11992 and previous config saved to /var/cache/conftool/dbconfig/20200721-141813-marostegui.json |
[production] |
| 14:16 |
<moritzm> |
draining restbase1023 for eventual reboot for kernel security update |
[production] |
| 14:10 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 14:06 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 14:06 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:06 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:03 |
<moritzm> |
draining restbase1022 for eventual reboot for kernel security update |
[production] |
| 14:01 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 13:57 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 13:55 |
<moritzm> |
draining restbase1021 for eventual reboot for kernel security update |
[production] |
| 13:51 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 13:50 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1119', diff saved to https://phabricator.wikimedia.org/P11991 and previous config saved to /var/cache/conftool/dbconfig/20200721-135028-marostegui.json |
[production] |
| 13:48 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 13:46 |
<moritzm> |
draining restbase1020 for eventual reboot for kernel security update |
[production] |
| 13:42 |
<akosiaris@cumin1001> |
conftool action : set/weight=1; selector: dc=codfw,service=mobileapps,name=scb.* |
[production] |
| 13:41 |
<akosiaris> |
increase codfw mobileapps kubernetes traffic to 96% T218733 |
[production] |