|
2019-04-15
§
|
| 14:21 |
<cdanis> |
cdanis@cumin1001.eqiad.wmnet ~ % sudo cumin 'maps1*' "disable-puppet 'bad permissions - T220982 - cdanis'" |
[production] |
| 14:18 |
<cdanis> |
cdanis@cumin1001.eqiad.wmnet ~ % sudo cumin 'maps*' 'sudo chmod -R a+r /srv/deployment/tilerator /srv/deployment/kartotherian' |
[production] |
| 14:18 |
<gehel> |
reseting permissions on maps server fir /srv/deployment/kartotherian and /srv/deplyoment/tilerator |
[production] |
| 14:04 |
<moritzm> |
rebooting ms-fe1005 for combined kernel/glibc/OpenSSL update |
[production] |
| 13:57 |
<jbond42> |
upgrading puppet 4 -> 5 and facter 2 -> 3 on mediawiki::canary_appserver, mediawiki::appserver::canary_api and cache::cache roles |
[production] |
| 13:56 |
<gehel> |
restart tilerator / kartotherian on all maps servers for openssl update |
[production] |
| 13:55 |
<godog> |
start ms-be1013 decom - T220590 |
[production] |
| 13:42 |
<godog> |
reboot ms-be1013 |
[production] |
| 13:09 |
<moritzm> |
installing wget security updates on trusty hosts |
[production] |
| 12:59 |
<moritzm> |
restarting archiva on archiva1001 for OpenJDK security update |
[production] |
| 12:50 |
<moritzm> |
restarting Apache on matomo1001 to pick up OpenSSL update |
[production] |
| 12:14 |
<moritzm> |
rolling restart of HHVM/Apache on deployment servers to pick up OpenSSL update |
[production] |
| 11:59 |
<fsero> |
pointing boron docker builds to the new registry temporarily (docker builds on boron might fail) |
[production] |
| 11:35 |
<Amir1> |
EU swat is done |
[production] |
| 11:26 |
<moritzm> |
rolling restart of HHVM/Apache on labweb* to pick up OpenSSL update |
[production] |
| 09:58 |
<moritzm> |
installing openssl1.0 security updates |
[production] |
| 09:18 |
<gehel> |
unbanning elastic1029 from cluster |
[production] |
| 08:58 |
<moritzm> |
updating mediawiki servers in eqiad to version 1.8.1 of the PHP extension for wikidiff |
[production] |
| 08:29 |
<onimisionipe> |
increase wal_keep_segments on codfw maps master |
[production] |
| 08:19 |
<moritzm> |
updating mediawiki servers in codfw to version 1.8.1 of the PHP extension for wikidiff |
[production] |
| 07:50 |
<Amir1> |
ladsgroup@mwmaint1002:~$ mwscript maintenance/initSiteStats.php --wiki=hywwiki --active (T220936) |
[production] |
| 05:31 |
<marostegui> |
Upgrade db1100 |
[production] |
| 05:07 |
<marostegui> |
powercycle mw1280 (crashed) |
[production] |
|
2019-04-12
§
|
| 21:16 |
<Krinkle> |
scap was unable to sync to 1 apache (connect to host cloudweb2001-dev.wikimedia.org port 22: Connection timed out) |
[production] |
| 21:10 |
<krinkle@deploy1001> |
Synchronized php-1.33.0-wmf.25/extensions/ImageMap/includes/ImageMap.php: I0ee84f059da / T217087 (duration: 05m 12s) |
[production] |
| 19:27 |
<dzahn@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) |
[production] |
| 19:27 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 19:24 |
<dzahn@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) |
[production] |
| 19:24 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 18:59 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 18:59 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 17:17 |
<onimisionipe> |
depooling maps2002 for postgres init |
[production] |
| 17:16 |
<onimisionipe> |
repooling maps2001 - postgres init is complete |
[production] |
| 16:14 |
<elukey> |
install ifstat on all the mc1* hosts for network bandwidth investigation |
[production] |
| 15:56 |
<gehel> |
starting data trasnfer from wdqs1008 to wdqs1009 - T220830 |
[production] |
| 15:32 |
<thcipriani> |
gerrit back |
[production] |
| 15:29 |
<thcipriani> |
gerrit restart incoming |
[production] |
| 14:29 |
<onimisionipe> |
depool maps2001 for postgres initialization |
[production] |
| 13:24 |
<akosiaris> |
re-enable puppet across the fleet. Patch merged, recovery storm coming |
[production] |