2019-07-17
ยง
|
14:41 |
<otto@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
14:35 |
<moritzm> |
restart pybal on lvs2002 (codfw primary) T227778 |
[production] |
14:32 |
<gehel@cumin1001> |
START - Cookbook sre.postgresql.postgres-init |
[production] |
14:30 |
<gehel> |
repool maps1004 - T218097 |
[production] |
14:11 |
<liw@deploy1001> |
Synchronized php: group1 wikis to 1.34.0-wmf.14 (duration: 00m 54s) |
[production] |
14:10 |
<liw@deploy1001> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.34.0-wmf.14 |
[production] |
14:09 |
<moritzm> |
restarting pybal on backup LVSes in codfw |
[production] |
14:02 |
<liw@deploy1001> |
Synchronized php-1.34.0-wmf.14/extensions/CirrusSearch/includes/Searcher.php: Do not serialize ResultsType instance T228276 (duration: 00m 55s) |
[production] |
13:37 |
<gehel@cumin1001> |
START - Cookbook sre.wdqs.data-transfer |
[production] |
13:26 |
<moritzm> |
disabled puppet on Icinga hosts in preparation of adding the LDAP replicas/codfw to LVS |
[production] |
13:10 |
<ema> |
cp-codfw: varnish frontend rolling restarts for 5.1.3-1wm11 upgrades T227672 |
[production] |
13:06 |
<ema> |
prometheus servers: remove varnish-upload_$dc_backend.yaml, replaced by ATS equivalent T227668 |
[production] |
12:57 |
<gehel@cumin1001> |
END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) |
[production] |
12:36 |
<godog> |
upgrade hp raid firmware on ms-be1 hosts - T141756 |
[production] |
12:15 |
<Urbanecm> |
Running foreachwiki extensions/AbuseFilter/maintenance/normalizeThrottleParameters.php in tmux session on mwmaint1002 (T209565) |
[production] |
12:11 |
<Urbanecm> |
Ran extensions/AbuseFilter/maintenance/normalizeThrottleParameters.php for cawiki and viwiki (T209565) |
[production] |
11:58 |
<gehel@cumin1001> |
START - Cookbook sre.wdqs.data-transfer |
[production] |
11:30 |
<mlitn@deploy1001> |
Synchronized php-1.34.0-wmf.14/extensions/WikibaseMediaInfo: [WikibaseMediaInfo] Revert "Add Wikidata links to statement UI elements" (duration: 00m 56s) |
[production] |
11:16 |
<dcausse> |
reindexing wikidata (elastic@eqiad) T227136 |
[production] |
11:08 |
<dcausse@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T227136: [cirrus] switch search traffic (except completion) to codfw (duration: 00m 54s) |
[production] |
10:53 |
<moritzm> |
re-enabled icinga1001 in meta monitoring |
[production] |
10:41 |
<godog> |
install updated linux-image-4.9.0-9-amd64 on ms-be hosts |
[production] |
10:30 |
<godog> |
start rolling reboot of ms-be eqiad hosts - T225713 |
[production] |
10:30 |
<gehel@cumin1001> |
END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) |
[production] |
10:23 |
<moritzm> |
rebooting icinga1001 for kernel update |
[production] |
10:20 |
<moritzm> |
disabled icinga1001 in meta monitoring |
[production] |
10:18 |
<jmm@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
10:18 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
10:08 |
<moritzm> |
rebooting lithium for kernel update |
[production] |
10:04 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
10:04 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
09:33 |
<gehel@cumin1001> |
START - Cookbook sre.wdqs.data-transfer |
[production] |
09:33 |
<gehel@cumin1001> |
END (ERROR) - Cookbook sre.wdqs.data-transfer (exit_code=97) |
[production] |
09:23 |
<moritzm> |
rebooting grafana1001 to pick up MDS-enabled qemu |
[production] |
09:21 |
<ema> |
cp-ats: upgrade fifo-log-demux to 0.3 T227668 |
[production] |
09:21 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Depool and clarify db2045 status T227862 (duration: 00m 55s) |
[production] |
09:19 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
09:19 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
09:15 |
<gehel@cumin1001> |
START - Cookbook sre.wdqs.data-transfer |
[production] |
09:07 |
<ema> |
upload fifo-log-demux 0.3 to stretch-wikimedia T227668 |
[production] |
08:51 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
08:51 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
08:36 |
<jijiki> |
Disable puppet on thumbor* in eqiad, depool and pool back to apply 523728 - T224572 |
[production] |
08:17 |
<jijiki> |
Pool mw1239 - T227867 |
[production] |
07:48 |
<godog> |
swift eqiad-prod: put back ms-be1043 sdk1 - T218544 |
[production] |
07:46 |
<ema> |
cp-esams: varnish frontend rolling restarts for 5.1.3-1wm11 upgrades T227672 |
[production] |
07:33 |
<moritzm> |
reimaging sarin for some tests |
[production] |
06:59 |
<elukey> |
apply mcrouter async replication to mw2224 - T225642 |
[production] |
06:25 |
<elukey> |
reboot analytics1072 as attempt to clear the megacli's config (and add a new disk) |
[production] |
06:20 |
<elukey> |
sudo -i /usr/local/sbin/restart-php7.2-fpm on mwdebug* to reset opcache |
[production] |