|
2020-02-13
ยง
|
| 09:45 |
<vgutierrez> |
pool cp10[85,86] running buster - T242093 |
[production] |
| 09:10 |
<moritzm> |
installing Java security updates on elastic* and relforge* |
[production] |
| 08:59 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Increase weight for db1107 50 -> 100 for 10.4 testing - T242702', diff saved to https://phabricator.wikimedia.org/P10403 and previous config saved to /var/cache/conftool/dbconfig/20200213-085957-marostegui.json |
[production] |
| 08:57 |
<gehel> |
restart elasticsearch on elastic2051 - JVM upgrade |
[production] |
| 08:21 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 08:18 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 08:17 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:15 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 07:57 |
<moritzm> |
installing Java security updates on Hadoop, Kafka/Jumbo, AQS and Druid canaries |
[production] |
| 07:57 |
<vgutierrez> |
depool cp10[85,86] and reimage as buster - T242093 |
[production] |
| 07:53 |
<moritzm> |
rolling restart of restbase-dev to pick up Java security update |
[production] |
| 07:49 |
<vgutierrez> |
pool cp10[87,88] running buster - T242093 |
[production] |
| 07:49 |
<vgutierrez> |
testing ATS 8.0.5-1wm16 + KA between ats-tls and varnish-fe in cp4031 - T244464 |
[production] |
| 07:47 |
<moritzm> |
installing Java security updates on stat/SWAP hosts |
[production] |
| 07:28 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1107 with weight 50 for 10.4 testing - T242702', diff saved to https://phabricator.wikimedia.org/P10402 and previous config saved to /var/cache/conftool/dbconfig/20200213-072839-marostegui.json |
[production] |
| 07:26 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 07:24 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 07:23 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 07:21 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 07:03 |
<vgutierrez> |
depool cp10[87,88] and reimage as buster - T242093 |
[production] |
| 07:01 |
<vgutierrez> |
pool cp10[89,90] running buster - T242093 |
[production] |
| 06:49 |
<vgutierrez> |
pool cp20[02,05] running buster - T242093 |
[production] |
| 06:36 |
<marostegui> |
Upgrade and compress db1087, this will generate lag on s8 on the wiki replicas - T232446 |
[production] |
| 06:35 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1087 for compression - T232446', diff saved to https://phabricator.wikimedia.org/P10401 and previous config saved to /var/cache/conftool/dbconfig/20200213-063535-marostegui.json |
[production] |
| 06:34 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 06:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Pool db1099:3318 into vslow for s8 T239453', diff saved to https://phabricator.wikimedia.org/P10400 and previous config saved to /var/cache/conftool/dbconfig/20200213-063334-marostegui.json |
[production] |
| 06:32 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 06:32 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1099:3318, db1099:3311 T239453', diff saved to https://phabricator.wikimedia.org/P10399 and previous config saved to /var/cache/conftool/dbconfig/20200213-063207-marostegui.json |
[production] |
| 06:30 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 06:28 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 06:26 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1099:3318, db1099:3311 T239453', diff saved to https://phabricator.wikimedia.org/P10398 and previous config saved to /var/cache/conftool/dbconfig/20200213-062642-marostegui.json |
[production] |
| 06:25 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 06:23 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 06:22 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 06:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1099:3318, db1099:3311 T239453', diff saved to https://phabricator.wikimedia.org/P10397 and previous config saved to /var/cache/conftool/dbconfig/20200213-062148-marostegui.json |
[production] |
| 06:19 |
<vgutierrez> |
testing a new build of ATS 8.0.6 in cp40[26,32] |
[production] |
| 06:19 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 06:12 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1099:3318, db1099:3311 T239453', diff saved to https://phabricator.wikimedia.org/P10396 and previous config saved to /var/cache/conftool/dbconfig/20200213-061219-marostegui.json |
[production] |
| 06:11 |
<vgutierrez> |
depool cp10[89,90] and reimage as buster - T242093 |
[production] |
| 06:04 |
<vgutierrez> |
depool cp20[02,05] and reimage as buster - T242093 |
[production] |
| 06:03 |
<vgutierrez> |
pool cp20[01,08] running buster - T242093 |
[production] |
| 06:02 |
<twentyafterfour> |
set phabricator read-only to false |
[production] |
| 06:01 |
<twentyafterfour> |
set phabricator read-only |
[production] |
| 06:00 |
<marostegui> |
Start phabricator maintenance T244566 |
[production] |
| 05:55 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 05:53 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 05:53 |
<marostegui> |
Upgrade db1128 without restarting mysql - T244566 |
[production] |
| 05:52 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 05:50 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 05:47 |
<marostegui> |
Silence m3 hosts for maintenance - T244566 |
[production] |