|
2020-07-21
ยง
|
| 15:00 |
<moritzm> |
draining restbase1026 for eventual reboot for kernel security update |
[production] |
| 14:57 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:57 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:56 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 14:52 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 14:51 |
<moritzm> |
draining restbase1025 for eventual reboot for kernel security update |
[production] |
| 14:48 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 14:44 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 14:35 |
<akosiaris@cumin1001> |
conftool action : set/weight=10; selector: dc=codfw,service=mobileapps,name=scb.* |
[production] |
| 14:35 |
<akosiaris> |
decrease codfw mobileapps kubernetes traffic to 72% T218733. Weird latency patterns exhibited when 92% was reached. See https://grafana.wikimedia.org/d/5CmeRcnMz/mobileapps?panelId=34&fullscreen&orgId=1&from=1595338489749&to=1595342071227&var-dc=codfw%20prometheus%2Fk8s&var-service=mobileapps&var-container_name=All |
[production] |
| 14:35 |
<moritzm> |
draining restbase1024 for eventual reboot for kernel security update |
[production] |
| 14:32 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1119', diff saved to https://phabricator.wikimedia.org/P11994 and previous config saved to /var/cache/conftool/dbconfig/20200721-143204-marostegui.json |
[production] |
| 14:26 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1119', diff saved to https://phabricator.wikimedia.org/P11993 and previous config saved to /var/cache/conftool/dbconfig/20200721-142634-marostegui.json |
[production] |
| 14:24 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:24 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:23 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 14:19 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 14:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1119', diff saved to https://phabricator.wikimedia.org/P11992 and previous config saved to /var/cache/conftool/dbconfig/20200721-141813-marostegui.json |
[production] |
| 14:16 |
<moritzm> |
draining restbase1023 for eventual reboot for kernel security update |
[production] |
| 14:10 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 14:06 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 14:06 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:06 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:03 |
<moritzm> |
draining restbase1022 for eventual reboot for kernel security update |
[production] |
| 14:01 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 13:57 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 13:55 |
<moritzm> |
draining restbase1021 for eventual reboot for kernel security update |
[production] |
| 13:51 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 13:50 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1119', diff saved to https://phabricator.wikimedia.org/P11991 and previous config saved to /var/cache/conftool/dbconfig/20200721-135028-marostegui.json |
[production] |
| 13:48 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 13:46 |
<moritzm> |
draining restbase1020 for eventual reboot for kernel security update |
[production] |
| 13:42 |
<akosiaris@cumin1001> |
conftool action : set/weight=1; selector: dc=codfw,service=mobileapps,name=scb.* |
[production] |
| 13:41 |
<akosiaris> |
increase codfw mobileapps kubernetes traffic to 96% T218733 |
[production] |
| 13:41 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 13:41 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 13:15 |
<Amir1> |
end of ladsgroup@mwmaint1002:~$ foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https (T258472 T258473) |
[production] |
| 13:13 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 13:11 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 13:10 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 13:06 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 13:03 |
<moritzm> |
draining restbase1019 for eventual reboot for kernel security update |
[production] |
| 13:01 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 13:01 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 12:55 |
<Amir1> |
start of ladsgroup@mwmaint1002:~$ foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https (T258472 T258473) |
[production] |
| 12:54 |
<marostegui> |
Stop haproxy on dbproxy1012 - T255408 |
[production] |
| 12:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1087', diff saved to https://phabricator.wikimedia.org/P11988 and previous config saved to /var/cache/conftool/dbconfig/20200721-121302-marostegui.json |
[production] |
| 12:05 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 12:01 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 11:53 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 11:49 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |