|
2020-02-10
§
|
| 11:31 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 11:29 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 11:22 |
<vgutierrez> |
depooling cp5011 and cp5005 & reimage as buster - T242093 |
[production] |
| 11:07 |
<vgutierrez> |
depool cp4027 & reimage as buster - T242093 |
[production] |
| 11:07 |
<vgutierrez> |
pooling ncredir2001 with buster - T243391 |
[production] |
| 11:03 |
<vgutierrez> |
pooling cp4028 with buster - T242093 |
[production] |
| 11:03 |
<arturo> |
restarted webservice by hand for nothing, given we don't have here yet the webservice version I was expecting |
[tools.openstack-browser] |
| 10:50 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:48 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:47 |
<godog> |
remove old logs from /var/log/swift on swift hsots |
[production] |
| 10:31 |
<vgutierrez> |
depool ncredir2001 and reimage as buster - T243391 |
[production] |
| 10:26 |
<vgutierrez> |
depool cp4028 & reimage as buster - T242093 |
[production] |
| 10:14 |
<moritzm> |
installing sudo security updates for buster |
[production] |
| 09:42 |
<legoktm> |
reloading zuul for https://gerrit.wikimedia.org/r/571254 |
[releng] |
| 09:39 |
<legoktm> |
building php74-compile image https://gerrit.wikimedia.org/r/542071 |
[releng] |
| 09:25 |
<legoktm> |
building php74-compile image https://gerrit.wikimedia.org/r/571251 |
[releng] |
| 09:11 |
<legoktm> |
building php74-compile image https://gerrit.wikimedia.org/r/571247 |
[releng] |
| 09:02 |
<legoktm> |
reloading zuul for https://gerrit.wikimedia.org/r/571242 |
[releng] |
| 08:53 |
<vgutierrez> |
pooling cp4029 with buster - T242093 |
[production] |
| 08:44 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Increase weight from 1 to 5 for db1107 - T242702', diff saved to https://phabricator.wikimedia.org/P10364 and previous config saved to /var/cache/conftool/dbconfig/20200210-084446-marostegui.json |
[production] |
| 08:44 |
<legoktm> |
reloading zuul for https://gerrit.wikimedia.org/r/539724 |
[releng] |
| 08:43 |
<vgutierrez> |
pooling ncredir2002 with buster - T243391 |
[production] |
| 08:34 |
<effie> |
rolling restart php-fpm on labweb[1001-1002].wikimedia.org,mw*.eqiad.wmnet,scandium.eqiad.wmnet, wtp[1025-1048].eqiad.wmnet |
[production] |
| 08:32 |
<effie> |
update php-apcu on eqiad - T236800 |
[production] |
| 08:30 |
<legoktm> |
reloading zuul for https://gerrit.wikimedia.org/r/571236 |
[releng] |
| 08:29 |
<effie> |
rolling restart php-fpm on cloudweb2001-dev.wikimedia.org,mw[2135-2147,2150-2212,2214-2290].codfw.wmnet,wtp[2001-2020].codfw.wmnet |
[production] |
| 08:23 |
<effie> |
update php-apcu on codfw - T236800 |
[production] |
| 08:17 |
<legoktm> |
updating tox-mysqld docker-pkg image https://gerrit.wikimedia.org/r/571234 |
[releng] |
| 07:58 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 07:56 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 07:54 |
<moritzm> |
updating d-i netinst image for Stretch 9.12 point release (which bumped the kernel ABI) |
[production] |
| 07:43 |
<legoktm> |
updating tox docker-pkg image for gettext https://gerrit.wikimedia.org/r/571019 |
[releng] |
| 07:29 |
<moritzm> |
updating d-i netinst image for Buster 10.3 point release (which bumped the kernel ABI) |
[production] |
| 07:20 |
<legoktm> |
updating tox docker-pkg image (try #2) https://gerrit.wikimedia.org/r/571127 |
[releng] |
| 07:09 |
<elukey> |
restore mw1347's mcrouter settings to its default (proxy threads 10 -> 5) |
[production] |
| 07:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Place db1107 - MariaDB 10.4 on s1 with minimal weight - T242702', diff saved to https://phabricator.wikimedia.org/P10363 and previous config saved to /var/cache/conftool/dbconfig/20200210-070140-marostegui.json |
[production] |
| 07:00 |
<legoktm> |
updating docker-pkg for https://gerrit.wikimedia.org/r/571017 |
[releng] |
| 06:55 |
<vgutierrez> |
depool ncredir2002 and reimage as buster - T243391 |
[production] |
| 06:53 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool es1019', diff saved to https://phabricator.wikimedia.org/P10362 and previous config saved to /var/cache/conftool/dbconfig/20200210-065326-marostegui.json |
[production] |
| 06:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1091 T232446', diff saved to https://phabricator.wikimedia.org/P10361 and previous config saved to /var/cache/conftool/dbconfig/20200210-065135-marostegui.json |
[production] |
| 06:47 |
<vgutierrez> |
depool cp4029 & reimage as buster - T242093 |
[production] |
| 06:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool es1019', diff saved to https://phabricator.wikimedia.org/P10360 and previous config saved to /var/cache/conftool/dbconfig/20200210-064553-marostegui.json |
[production] |
| 06:44 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1091 T232446', diff saved to https://phabricator.wikimedia.org/P10359 and previous config saved to /var/cache/conftool/dbconfig/20200210-064458-marostegui.json |
[production] |
| 06:39 |
<marostegui> |
Compress db1124:3318 - this will generate lag on s8 wiki replicas - T232446 |
[production] |
| 06:37 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1091 T232446', diff saved to https://phabricator.wikimedia.org/P10358 and previous config saved to /var/cache/conftool/dbconfig/20200210-063716-marostegui.json |
[production] |
| 06:23 |
<marostegui> |
Remove partitions from db1099:3311, db1099:3318 T239453 |
[production] |
| 06:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1099:3318 T239453', diff saved to https://phabricator.wikimedia.org/P10357 and previous config saved to /var/cache/conftool/dbconfig/20200210-062112-marostegui.json |
[production] |
| 06:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool repool db1099:3311 T239453', diff saved to https://phabricator.wikimedia.org/P10356 and previous config saved to /var/cache/conftool/dbconfig/20200210-061822-marostegui.json |
[production] |
| 06:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1091 T232446', diff saved to https://phabricator.wikimedia.org/P10355 and previous config saved to /var/cache/conftool/dbconfig/20200210-061656-marostegui.json |
[production] |