|
2019-07-16
§
|
| 15:37 |
<elukey> |
reboot analytics1072 as attempt to force the raid controller to set a drive failed - T226467 |
[production] |
| 15:12 |
<elukey> |
start mariadb on db1107 and re-enable mysql consumers on eventlog1002 and replication on db1108 |
[production] |
| 14:53 |
<elukey> |
stop mariadb on db1107 to allow maintenance |
[production] |
| 14:53 |
<elukey> |
stop eventlogging mysql consumers on eventlog1002 and eventlogging_sync on db1108 to allow db1107 maintenance |
[production] |
| 14:52 |
<jbond42> |
will restart redis on oresdb at 16:00 UTC - T228045 |
[production] |
| 14:51 |
<jbond42> |
enable puppet accross the fleat |
[production] |
| 14:50 |
<jbond@cumin1001> |
conftool action : set/pooled=yes; selector: name=dns1001.wikimedia.org |
[production] |
| 14:43 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:43 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:43 |
<jbond@cumin1001> |
conftool action : set/pooled=no; selector: name=dns1001.wikimedia.org |
[production] |
| 14:40 |
<jbond42> |
disable puppet accross the fleat to make a change to the hiera |
[production] |
| 14:29 |
<jijiki> |
Enable puppet and rolling restart thumbor* in codfw - T224572 |
[production] |
| 14:16 |
<jijiki> |
Depool thumbor2001 and pool back - T224572 |
[production] |
| 14:13 |
<jijiki> |
Disabling puppet on thumbor*codfw.wmnet - T224572 |
[production] |
| 14:08 |
<liw> |
group0 to 1.34.0-wmf.14 |
[production] |
| 14:06 |
<liw@deploy1001> |
rebuilt and synchronized wikiversions files: group0 to php-1.34.0-wmf.14 |
[production] |
| 13:41 |
<liw@deploy1001> |
Finished scap: testwiki to php-1.34.0-wmf.14 and rebuild l10n cache (duration: 26m 45s) |
[production] |
| 13:24 |
<vgutierrez> |
restarting pybal on lvs2001 and lvs1013 |
[production] |
| 13:20 |
<vgutierrez> |
restarting pybal on lvs2004 and lvs1016 |
[production] |
| 13:14 |
<liw@deploy1001> |
Started scap: testwiki to php-1.34.0-wmf.14 and rebuild l10n cache |
[production] |
| 12:59 |
<liw@deploy1001> |
Pruned MediaWiki: 1.34.0-wmf.8 (duration: 01m 46s) |
[production] |
| 12:57 |
<liw@deploy1001> |
Pruned MediaWiki: 1.34.0-wmf.7 (duration: 02m 01s) |
[production] |
| 12:54 |
<liw@deploy1001> |
Pruned MediaWiki: 1.34.0-wmf.6 (duration: 02m 04s) |
[production] |
| 12:52 |
<liw@deploy1001> |
Pruned MediaWiki: 1.34.0-wmf.4 (duration: 02m 11s) |
[production] |
| 12:49 |
<liw@deploy1001> |
Pruned MediaWiki: 1.34.0-wmf.5 (duration: 07m 42s) |
[production] |
| 12:42 |
<dcausse> |
deleting stale wikidata indices (elastic@eqiad) T227136 |
[production] |
| 12:11 |
<jijiki> |
Depool mw1293 and pool back |
[production] |
| 11:57 |
<moritzm> |
synched docker-ce, docker-ce-cli, containerd.io to thirdparty/ci for stretch-wikimedia (T226236) |
[production] |
| 11:12 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 11:12 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 11:12 |
<moritzm> |
rebooting remaining swift frontends in eqiad to pick up a kernel with SACK fixed (T228086) |
[production] |
| 10:29 |
<moritzm> |
rebooting ms-fe1005 to pick up kernel with SACK fixed (T228086) |
[production] |
| 10:28 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:28 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:17 |
<vgutierrez> |
restart pybal on lvs1013 |
[production] |
| 10:15 |
<vgutierrez> |
restart pybal on lvs2001 |
[production] |
| 10:11 |
<vgutierrez> |
restarting pybal on lvs1016 |
[production] |
| 10:08 |
<vgutierrez> |
restarting pybal on lvs2004 |
[production] |
| 10:04 |
<vgutierrez@puppetmaster1001> |
conftool action : set/pooled=yes; selector: cluster=ncredir,service=nginx |
[production] |
| 09:24 |
<elukey> |
apply mcrouter async replication settings to mw1276 - T225642 |
[production] |
| 09:23 |
<elukey> |
pool mw1261 back with mcrouter async replication settings - T225642 |
[production] |
| 08:50 |
<fsero> |
upload coredns docker image into registry T226516 |
[production] |
| 08:44 |
<jynus> |
droping servermon accounts from m1 dbs T198939 |
[production] |
| 08:12 |
<fsero> |
uploading coredns_1.5.2 for buster and stretch - T226516 |
[production] |
| 08:11 |
<fsero> |
uploading coredns_1.5.2 for buster and stretch |
[production] |
| 07:45 |
<elukey> |
depool mw1261 to test mcrouter changes |
[production] |
| 00:24 |
<krinkle@deploy1001> |
Synchronized php-1.34.0-wmf.13/includes/cache/LinkCache.php: 4a5f4ca2fd788 (duration: 00m 51s) |
[production] |
| 00:05 |
<catrope@deploy1001> |
Synchronized php-1.34.0-wmf.13/skins/MinervaNeue/: Restrict AMC scripts and styles to AMC mode (T227929) (duration: 00m 52s) |
[production] |
| 00:03 |
<shdubsh> |
restart logstash to revert mitigations - T228089 |
[production] |