4351-4400 of 10000 results (73ms)
2019-07-17 ยง
14:41 <otto@cumin1001> START - Cookbook sre.hosts.decommission [production]
14:35 <moritzm> restart pybal on lvs2002 (codfw primary) T227778 [production]
14:32 <gehel@cumin1001> START - Cookbook sre.postgresql.postgres-init [production]
14:30 <gehel> repool maps1004 - T218097 [production]
14:11 <liw@deploy1001> Synchronized php: group1 wikis to 1.34.0-wmf.14 (duration: 00m 54s) [production]
14:10 <liw@deploy1001> rebuilt and synchronized wikiversions files: group1 wikis to 1.34.0-wmf.14 [production]
14:09 <moritzm> restarting pybal on backup LVSes in codfw [production]
14:02 <liw@deploy1001> Synchronized php-1.34.0-wmf.14/extensions/CirrusSearch/includes/Searcher.php: Do not serialize ResultsType instance T228276 (duration: 00m 55s) [production]
13:37 <gehel@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
13:26 <moritzm> disabled puppet on Icinga hosts in preparation of adding the LDAP replicas/codfw to LVS [production]
13:10 <ema> cp-codfw: varnish frontend rolling restarts for 5.1.3-1wm11 upgrades T227672 [production]
13:06 <ema> prometheus servers: remove varnish-upload_$dc_backend.yaml, replaced by ATS equivalent T227668 [production]
12:57 <gehel@cumin1001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
12:36 <godog> upgrade hp raid firmware on ms-be1 hosts - T141756 [production]
12:15 <Urbanecm> Running foreachwiki extensions/AbuseFilter/maintenance/normalizeThrottleParameters.php in tmux session on mwmaint1002 (T209565) [production]
12:11 <Urbanecm> Ran extensions/AbuseFilter/maintenance/normalizeThrottleParameters.php for cawiki and viwiki (T209565) [production]
11:58 <gehel@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
11:30 <mlitn@deploy1001> Synchronized php-1.34.0-wmf.14/extensions/WikibaseMediaInfo: [WikibaseMediaInfo] Revert "Add Wikidata links to statement UI elements" (duration: 00m 56s) [production]
11:16 <dcausse> reindexing wikidata (elastic@eqiad) T227136 [production]
11:08 <dcausse@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T227136: [cirrus] switch search traffic (except completion) to codfw (duration: 00m 54s) [production]
10:53 <moritzm> re-enabled icinga1001 in meta monitoring [production]
10:41 <godog> install updated linux-image-4.9.0-9-amd64 on ms-be hosts [production]
10:30 <godog> start rolling reboot of ms-be eqiad hosts - T225713 [production]
10:30 <gehel@cumin1001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
10:23 <moritzm> rebooting icinga1001 for kernel update [production]
10:20 <moritzm> disabled icinga1001 in meta monitoring [production]
10:18 <jmm@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
10:18 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
10:08 <moritzm> rebooting lithium for kernel update [production]
10:04 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:04 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
09:33 <gehel@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
09:33 <gehel@cumin1001> END (ERROR) - Cookbook sre.wdqs.data-transfer (exit_code=97) [production]
09:23 <moritzm> rebooting grafana1001 to pick up MDS-enabled qemu [production]
09:21 <ema> cp-ats: upgrade fifo-log-demux to 0.3 T227668 [production]
09:21 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Depool and clarify db2045 status T227862 (duration: 00m 55s) [production]
09:19 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:19 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
09:15 <gehel@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
09:07 <ema> upload fifo-log-demux 0.3 to stretch-wikimedia T227668 [production]
08:51 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:51 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
08:36 <jijiki> Disable puppet on thumbor* in eqiad, depool and pool back to apply 523728 - T224572 [production]
08:17 <jijiki> Pool mw1239 - T227867 [production]
07:48 <godog> swift eqiad-prod: put back ms-be1043 sdk1 - T218544 [production]
07:46 <ema> cp-esams: varnish frontend rolling restarts for 5.1.3-1wm11 upgrades T227672 [production]
07:33 <moritzm> reimaging sarin for some tests [production]
06:59 <elukey> apply mcrouter async replication to mw2224 - T225642 [production]
06:25 <elukey> reboot analytics1072 as attempt to clear the megacli's config (and add a new disk) [production]
06:20 <elukey> sudo -i /usr/local/sbin/restart-php7.2-fpm on mwdebug* to reset opcache [production]