|
2020-02-13
ยง
|
| 11:12 |
<ema> |
A:cp re-enable puppet, leave it to cron to apply wikimedia-common/wikimedia-frontend VCL merge T241239 |
[production] |
| 11:08 |
<vgutierrez> |
upload trafficserver 8.0.5-1wm16 to apt.wm.o (buster) - T244464 |
[production] |
| 11:02 |
<vgutierrez> |
pool cp10[81,82] and reimage as buster - T242093 |
[production] |
| 10:59 |
<ema> |
cp4021 (cache_upload): apply wikimedia-common/wikimedia-frontend VCL merge T241239 |
[production] |
| 10:49 |
<ema> |
cp4027 (cache_text): apply wikimedia-common/wikimedia-frontend VCL merge T241239 |
[production] |
| 10:46 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:44 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:44 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:41 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:24 |
<vgutierrez> |
depool cp10[81,82] and reimage as buster - T242093 |
[production] |
| 10:23 |
<vgutierrez> |
removing /root/.ssh/known_hosts in cumin1001 |
[production] |
| 10:21 |
<vgutierrez> |
pool cp10[83,84] running buster - T242093 |
[production] |
| 10:08 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:06 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:06 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:03 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:45 |
<vgutierrez> |
depool cp10[83,84] and reimage as buster - T242093 |
[production] |
| 09:45 |
<vgutierrez> |
pool cp10[85,86] running buster - T242093 |
[production] |
| 09:10 |
<moritzm> |
installing Java security updates on elastic* and relforge* |
[production] |
| 08:59 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Increase weight for db1107 50 -> 100 for 10.4 testing - T242702', diff saved to https://phabricator.wikimedia.org/P10403 and previous config saved to /var/cache/conftool/dbconfig/20200213-085957-marostegui.json |
[production] |
| 08:57 |
<gehel> |
restart elasticsearch on elastic2051 - JVM upgrade |
[production] |
| 08:21 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 08:18 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 08:17 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:15 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 07:57 |
<moritzm> |
installing Java security updates on Hadoop, Kafka/Jumbo, AQS and Druid canaries |
[production] |
| 07:57 |
<vgutierrez> |
depool cp10[85,86] and reimage as buster - T242093 |
[production] |
| 07:53 |
<moritzm> |
rolling restart of restbase-dev to pick up Java security update |
[production] |
| 07:49 |
<vgutierrez> |
pool cp10[87,88] running buster - T242093 |
[production] |
| 07:49 |
<vgutierrez> |
testing ATS 8.0.5-1wm16 + KA between ats-tls and varnish-fe in cp4031 - T244464 |
[production] |
| 07:47 |
<moritzm> |
installing Java security updates on stat/SWAP hosts |
[production] |
| 07:28 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1107 with weight 50 for 10.4 testing - T242702', diff saved to https://phabricator.wikimedia.org/P10402 and previous config saved to /var/cache/conftool/dbconfig/20200213-072839-marostegui.json |
[production] |
| 07:26 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 07:24 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 07:23 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 07:21 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 07:03 |
<vgutierrez> |
depool cp10[87,88] and reimage as buster - T242093 |
[production] |
| 07:01 |
<vgutierrez> |
pool cp10[89,90] running buster - T242093 |
[production] |
| 06:49 |
<vgutierrez> |
pool cp20[02,05] running buster - T242093 |
[production] |
| 06:36 |
<marostegui> |
Upgrade and compress db1087, this will generate lag on s8 on the wiki replicas - T232446 |
[production] |
| 06:35 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1087 for compression - T232446', diff saved to https://phabricator.wikimedia.org/P10401 and previous config saved to /var/cache/conftool/dbconfig/20200213-063535-marostegui.json |
[production] |
| 06:34 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 06:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Pool db1099:3318 into vslow for s8 T239453', diff saved to https://phabricator.wikimedia.org/P10400 and previous config saved to /var/cache/conftool/dbconfig/20200213-063334-marostegui.json |
[production] |
| 06:32 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 06:32 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1099:3318, db1099:3311 T239453', diff saved to https://phabricator.wikimedia.org/P10399 and previous config saved to /var/cache/conftool/dbconfig/20200213-063207-marostegui.json |
[production] |
| 06:30 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 06:28 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 06:26 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1099:3318, db1099:3311 T239453', diff saved to https://phabricator.wikimedia.org/P10398 and previous config saved to /var/cache/conftool/dbconfig/20200213-062642-marostegui.json |
[production] |
| 06:25 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 06:23 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |