|
2019-07-17
ยง
|
| 13:37 |
<gehel@cumin1001> |
START - Cookbook sre.wdqs.data-transfer |
[production] |
| 13:26 |
<moritzm> |
disabled puppet on Icinga hosts in preparation of adding the LDAP replicas/codfw to LVS |
[production] |
| 13:10 |
<ema> |
cp-codfw: varnish frontend rolling restarts for 5.1.3-1wm11 upgrades T227672 |
[production] |
| 13:06 |
<ema> |
prometheus servers: remove varnish-upload_$dc_backend.yaml, replaced by ATS equivalent T227668 |
[production] |
| 12:57 |
<gehel@cumin1001> |
END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) |
[production] |
| 12:36 |
<godog> |
upgrade hp raid firmware on ms-be1 hosts - T141756 |
[production] |
| 12:15 |
<Urbanecm> |
Running foreachwiki extensions/AbuseFilter/maintenance/normalizeThrottleParameters.php in tmux session on mwmaint1002 (T209565) |
[production] |
| 12:11 |
<Urbanecm> |
Ran extensions/AbuseFilter/maintenance/normalizeThrottleParameters.php for cawiki and viwiki (T209565) |
[production] |
| 11:58 |
<gehel@cumin1001> |
START - Cookbook sre.wdqs.data-transfer |
[production] |
| 11:30 |
<mlitn@deploy1001> |
Synchronized php-1.34.0-wmf.14/extensions/WikibaseMediaInfo: [WikibaseMediaInfo] Revert "Add Wikidata links to statement UI elements" (duration: 00m 56s) |
[production] |
| 11:16 |
<dcausse> |
reindexing wikidata (elastic@eqiad) T227136 |
[production] |
| 11:08 |
<dcausse@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T227136: [cirrus] switch search traffic (except completion) to codfw (duration: 00m 54s) |
[production] |
| 10:53 |
<moritzm> |
re-enabled icinga1001 in meta monitoring |
[production] |
| 10:41 |
<godog> |
install updated linux-image-4.9.0-9-amd64 on ms-be hosts |
[production] |
| 10:30 |
<godog> |
start rolling reboot of ms-be eqiad hosts - T225713 |
[production] |
| 10:30 |
<gehel@cumin1001> |
END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) |
[production] |
| 10:23 |
<moritzm> |
rebooting icinga1001 for kernel update |
[production] |
| 10:20 |
<moritzm> |
disabled icinga1001 in meta monitoring |
[production] |
| 10:18 |
<jmm@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 10:18 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:08 |
<moritzm> |
rebooting lithium for kernel update |
[production] |
| 10:04 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:04 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:33 |
<gehel@cumin1001> |
START - Cookbook sre.wdqs.data-transfer |
[production] |
| 09:33 |
<gehel@cumin1001> |
END (ERROR) - Cookbook sre.wdqs.data-transfer (exit_code=97) |
[production] |
| 09:23 |
<moritzm> |
rebooting grafana1001 to pick up MDS-enabled qemu |
[production] |
| 09:21 |
<ema> |
cp-ats: upgrade fifo-log-demux to 0.3 T227668 |
[production] |
| 09:21 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Depool and clarify db2045 status T227862 (duration: 00m 55s) |
[production] |
| 09:19 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:19 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:15 |
<gehel@cumin1001> |
START - Cookbook sre.wdqs.data-transfer |
[production] |
| 09:07 |
<ema> |
upload fifo-log-demux 0.3 to stretch-wikimedia T227668 |
[production] |
| 08:51 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 08:51 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:36 |
<jijiki> |
Disable puppet on thumbor* in eqiad, depool and pool back to apply 523728 - T224572 |
[production] |
| 08:17 |
<jijiki> |
Pool mw1239 - T227867 |
[production] |
| 07:48 |
<godog> |
swift eqiad-prod: put back ms-be1043 sdk1 - T218544 |
[production] |
| 07:46 |
<ema> |
cp-esams: varnish frontend rolling restarts for 5.1.3-1wm11 upgrades T227672 |
[production] |
| 07:33 |
<moritzm> |
reimaging sarin for some tests |
[production] |
| 06:59 |
<elukey> |
apply mcrouter async replication to mw2224 - T225642 |
[production] |
| 06:25 |
<elukey> |
reboot analytics1072 as attempt to clear the megacli's config (and add a new disk) |
[production] |
| 06:20 |
<elukey> |
sudo -i /usr/local/sbin/restart-php7.2-fpm on mwdebug* to reset opcache |
[production] |
| 05:26 |
<marostegui> |
Stop MySQL on db1065 for decommissioning - T227560 |
[production] |
| 05:24 |
<marostegui> |
Remove db1065 from tendril and zarcillo - T227560 |
[production] |
| 03:46 |
<tstarling@deploy1001> |
Synchronized php-1.34.0-wmf.14/extensions/CentralAuth/includes/specials/SpecialMultiLock.php: T227772 (duration: 00m 54s) |
[production] |
| 03:42 |
<tstarling@deploy1001> |
Synchronized php-1.34.0-wmf.13/extensions/CentralAuth/includes/specials/SpecialMultiLock.php: T227772 (duration: 00m 56s) |
[production] |
| 03:00 |
<tstarling@deploy1001> |
Synchronized php-1.34.0-wmf.13/includes/Permissions/PermissionManager.php: (no justification provided) (duration: 00m 54s) |
[production] |
| 02:58 |
<tstarling@deploy1001> |
Synchronized php-1.34.0-wmf.14/includes/Permissions/PermissionManager.php: (no justification provided) (duration: 00m 57s) |
[production] |
| 00:50 |
<mutante> |
wikitech-static commented out cert renewal cron job out of caution - still needs fixing but continue tomorrow |
[production] |
| 00:12 |
<mutante> |
wikitech-static - adding (undocumented!) option webroot-map to certbot config to use webroot authenticator with different document roots per domain while using the config file and not cli params (T214640) |
[production] |