|
2020-02-19
ยง
|
| 10:58 |
<marostegui@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 10:58 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:58 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:58 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:45 |
<jynus> |
upgrading mariadb client on cumin hosts |
[production] |
| 10:38 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db2089:3315, db2089:3316 after new package testing', diff saved to https://phabricator.wikimedia.org/P10457 and previous config saved to /var/cache/conftool/dbconfig/20200219-103806-marostegui.json |
[production] |
| 10:26 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:24 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:17 |
<jynus> |
stopping db2089 mariadb@s5 |
[production] |
| 10:12 |
<jiji@cumin1001> |
conftool action : set/weight=30; selector: dc=eqiad,cluster=appserver,service=apache2,name=mw135[0-5]*.eqiad.wmnet |
[production] |
| 10:12 |
<jiji@cumin1001> |
conftool action : set/weight=30; selector: dc=eqiad,cluster=appserver,service=nginx,name=mw135[0-5]*.eqiad.wmnet |
[production] |
| 10:11 |
<jiji@cumin1001> |
conftool action : set/weight=30; selector: dc=eqiad,cluster=appserver,service=nginx,name=mw1349.eqiad.wmnet |
[production] |
| 10:11 |
<jiji@cumin1001> |
conftool action : set/weight=30; selector: dc=eqiad,cluster=appserver,service=apache2,name=mw1349.eqiad.wmnet |
[production] |
| 10:09 |
<moritzm> |
updated tftpboot environment for stretch-bootif for the 9.12 point release T241359 |
[production] |
| 09:53 |
<jynus> |
stopping and upgrading db1140 instances |
[production] |
| 09:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2089:3315, db2089:3316 for new package testing', diff saved to https://phabricator.wikimedia.org/P10455 and previous config saved to /var/cache/conftool/dbconfig/20200219-095139-marostegui.json |
[production] |
| 09:51 |
<marostegui> |
Depool db2089:3315, db2089:3316 for new package testing |
[production] |
| 09:49 |
<akosiaris> |
T245516. Deploy mathoid chart version 0.0.27, removing logstash gelf configuration |
[production] |
| 09:46 |
<akosiaris@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'mathoid' for release 'production' . |
[production] |
| 09:43 |
<vgutierrez> |
test trafficserver 8.0.6-rc1 in cp40[26,32] |
[production] |
| 09:34 |
<_joe_> |
cleared opcache on mw1313 |
[production] |
| 09:34 |
<akosiaris@deploy1001> |
helmfile [CODFW] Ran 'sync' command on namespace 'mathoid' for release 'canary' . |
[production] |
| 09:34 |
<akosiaris@deploy1001> |
helmfile [CODFW] Ran 'sync' command on namespace 'mathoid' for release 'production' . |
[production] |
| 09:33 |
<akosiaris@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'mathoid' for release 'staging' . |
[production] |
| 08:54 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 08:53 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 08:50 |
<marostegui> |
Remove dbproxy1007 grants from m2 - T231280 |
[production] |
| 08:41 |
<marostegui> |
Remove wikiadmin2 user from s7 - T243512 |
[production] |
| 08:23 |
<Urbanecm> |
run mwscript deleteEqualMessages.php cswiki --delete |
[production] |
| 08:14 |
<godog> |
roll restart swift proxies - T244776 |
[production] |
| 07:02 |
<marostegui> |
Remove wikiadmin2 user from es2 - T243512 |
[production] |
| 06:57 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Increase API weight for db1107 50 -> 100 for 10.4 testing - T242702', diff saved to https://phabricator.wikimedia.org/P10454 and previous config saved to /var/cache/conftool/dbconfig/20200219-065726-marostegui.json |
[production] |
| 06:35 |
<marostegui> |
Compress watchlist_expiry table on s3 (this will take hours as I have left a 60 seconds sleep between tables) - T245358 |
[production] |
| 06:17 |
<marostegui> |
Compress new and empty watchlist_expiry table - T245358 |
[production] |
| 01:34 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 01:32 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 01:28 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1353.eqiad.wmnet |
[production] |
| 01:27 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 01:24 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 01:23 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1354.eqiad.wmnet |
[production] |
| 01:22 |
<mutante> |
mw1353 - restarted apache (some race condition on new installs, 5 other servers did not have the issue) |
[production] |
| 01:17 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1355.eqiad.wmnet |
[production] |
| 01:16 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1350.eqiad.wmnet |
[production] |
| 01:16 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1351.eqiad.wmnet |
[production] |
| 01:15 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1352.eqiad.wmnet |
[production] |
| 01:14 |
<dzahn@cumin1001> |
conftool action : set/weight=10; selector: name=mw1355.eqiad.wmnet |
[production] |
| 01:14 |
<dzahn@cumin1001> |
conftool action : set/weight=10; selector: name=mw1354.eqiad.wmnet |
[production] |
| 01:14 |
<dzahn@cumin1001> |
conftool action : set/weight=10; selector: name=mw1350.eqiad.wmnet |
[production] |
| 01:14 |
<dzahn@cumin1001> |
conftool action : set/weight=10; selector: name=mw1353.eqiad.wmnet |
[production] |
| 01:14 |
<dzahn@cumin1001> |
conftool action : set/weight=10; selector: name=mw1351.eqiad.wmnet |
[production] |