|
2020-02-10
§
|
| 12:18 |
<_joe_> |
running puppet, scap pull on mwdebug1001 |
[production] |
| 12:17 |
<vgutierrez> |
upload trafficserver 8.0.5-1wm15 to apt.wm.o (buster) - T244538 |
[production] |
| 12:08 |
<vgutierrez@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 12:08 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 12:06 |
<vgutierrez> |
testing ats 8.0.5-1-wm15 on cp4032 - T244538 |
[production] |
| 12:06 |
<urbanecm@deploy1001> |
Synchronized wmf-config/throttle.php: SWAT: 014405a: Add throttle rules for OSU Editathon and workshop for cawiki, remove expired ones (T244608, T244645) (duration: 01m 03s) |
[production] |
| 11:57 |
<vgutierrez> |
depool ncredir1001 and reimage as buster - T243391 |
[production] |
| 11:57 |
<vgutierrez> |
pooling ncredir1002 with buster - T243391 |
[production] |
| 11:43 |
<vgutierrez> |
pooling cp4027 with buster - T242093 |
[production] |
| 11:38 |
<vgutierrez> |
depool ncredir1002 and reimage as buster - T243391 |
[production] |
| 11:31 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 11:29 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 11:22 |
<vgutierrez> |
depooling cp5011 and cp5005 & reimage as buster - T242093 |
[production] |
| 11:07 |
<vgutierrez> |
depool cp4027 & reimage as buster - T242093 |
[production] |
| 11:07 |
<vgutierrez> |
pooling ncredir2001 with buster - T243391 |
[production] |
| 11:03 |
<vgutierrez> |
pooling cp4028 with buster - T242093 |
[production] |
| 10:50 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:48 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:47 |
<godog> |
remove old logs from /var/log/swift on swift hsots |
[production] |
| 10:31 |
<vgutierrez> |
depool ncredir2001 and reimage as buster - T243391 |
[production] |
| 10:26 |
<vgutierrez> |
depool cp4028 & reimage as buster - T242093 |
[production] |
| 10:14 |
<moritzm> |
installing sudo security updates for buster |
[production] |
| 08:53 |
<vgutierrez> |
pooling cp4029 with buster - T242093 |
[production] |
| 08:44 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Increase weight from 1 to 5 for db1107 - T242702', diff saved to https://phabricator.wikimedia.org/P10364 and previous config saved to /var/cache/conftool/dbconfig/20200210-084446-marostegui.json |
[production] |
| 08:43 |
<vgutierrez> |
pooling ncredir2002 with buster - T243391 |
[production] |
| 08:34 |
<effie> |
rolling restart php-fpm on labweb[1001-1002].wikimedia.org,mw*.eqiad.wmnet,scandium.eqiad.wmnet, wtp[1025-1048].eqiad.wmnet |
[production] |
| 08:32 |
<effie> |
update php-apcu on eqiad - T236800 |
[production] |
| 08:29 |
<effie> |
rolling restart php-fpm on cloudweb2001-dev.wikimedia.org,mw[2135-2147,2150-2212,2214-2290].codfw.wmnet,wtp[2001-2020].codfw.wmnet |
[production] |
| 08:23 |
<effie> |
update php-apcu on codfw - T236800 |
[production] |
| 07:58 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 07:56 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 07:54 |
<moritzm> |
updating d-i netinst image for Stretch 9.12 point release (which bumped the kernel ABI) |
[production] |
| 07:29 |
<moritzm> |
updating d-i netinst image for Buster 10.3 point release (which bumped the kernel ABI) |
[production] |
| 07:09 |
<elukey> |
restore mw1347's mcrouter settings to its default (proxy threads 10 -> 5) |
[production] |
| 07:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Place db1107 - MariaDB 10.4 on s1 with minimal weight - T242702', diff saved to https://phabricator.wikimedia.org/P10363 and previous config saved to /var/cache/conftool/dbconfig/20200210-070140-marostegui.json |
[production] |
| 06:55 |
<vgutierrez> |
depool ncredir2002 and reimage as buster - T243391 |
[production] |
| 06:53 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool es1019', diff saved to https://phabricator.wikimedia.org/P10362 and previous config saved to /var/cache/conftool/dbconfig/20200210-065326-marostegui.json |
[production] |
| 06:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1091 T232446', diff saved to https://phabricator.wikimedia.org/P10361 and previous config saved to /var/cache/conftool/dbconfig/20200210-065135-marostegui.json |
[production] |
| 06:47 |
<vgutierrez> |
depool cp4029 & reimage as buster - T242093 |
[production] |
| 06:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool es1019', diff saved to https://phabricator.wikimedia.org/P10360 and previous config saved to /var/cache/conftool/dbconfig/20200210-064553-marostegui.json |
[production] |
| 06:44 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1091 T232446', diff saved to https://phabricator.wikimedia.org/P10359 and previous config saved to /var/cache/conftool/dbconfig/20200210-064458-marostegui.json |
[production] |
| 06:39 |
<marostegui> |
Compress db1124:3318 - this will generate lag on s8 wiki replicas - T232446 |
[production] |
| 06:37 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1091 T232446', diff saved to https://phabricator.wikimedia.org/P10358 and previous config saved to /var/cache/conftool/dbconfig/20200210-063716-marostegui.json |
[production] |
| 06:23 |
<marostegui> |
Remove partitions from db1099:3311, db1099:3318 T239453 |
[production] |
| 06:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1099:3318 T239453', diff saved to https://phabricator.wikimedia.org/P10357 and previous config saved to /var/cache/conftool/dbconfig/20200210-062112-marostegui.json |
[production] |
| 06:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool repool db1099:3311 T239453', diff saved to https://phabricator.wikimedia.org/P10356 and previous config saved to /var/cache/conftool/dbconfig/20200210-061822-marostegui.json |
[production] |
| 06:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1091 T232446', diff saved to https://phabricator.wikimedia.org/P10355 and previous config saved to /var/cache/conftool/dbconfig/20200210-061656-marostegui.json |
[production] |