3101-3150 of 10000 results (55ms)
2018-06-25 §
13:05 <marostegui> Deploy schema change on db2035 (codfw primary master for s2) with replication, this will generate lag on s2 codfw T191316 T192926 T89737 T195193 [production]
13:05 <godog> refresh cassandra certificates for 'restbase' cluster [production]
13:00 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1106 after alter table (duration: 00m 57s) [production]
12:29 <kartik@deploy1001> Finished deploy [cxserver/deploy@cc6dc61]: Update cxserver to ece5e7a (T191874, T196354, T195768, T195768) (duration: 03m 33s) [production]
12:26 <kartik@deploy1001> Started deploy [cxserver/deploy@cc6dc61]: Update cxserver to ece5e7a (T191874, T196354, T195768, T195768) [production]
12:04 <mobrovac> Restbase: Add the "headers" field to Cassandra schemae in production for T197789 [production]
12:00 <mobrovac@deploy1001> Finished deploy [restbase/deploy@f521e7e] (dev-cluster): Dev cluster: Include pagelanguage in title_revisions (duration: 04m 05s) [production]
12:00 <gehel> repooling wdqs1005 it has catched up on updates - T198042 [production]
11:56 <mobrovac@deploy1001> Started deploy [restbase/deploy@f521e7e] (dev-cluster): Dev cluster: Include pagelanguage in title_revisions [production]
10:32 <akosiaris> increase CPU count for proton machines from 2 to 10. T197862 [production]
08:22 <marostegui> Deploy schema change on db1106 with replication, this will generate lag on s1 labs T191316 T192926 T89737 T195193 [production]
08:16 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1106 for alter table (duration: 00m 56s) [production]
08:16 <marostegui> Stop replication on db1106 to change triggers for db1124 and db1095 - T192926 [production]
08:11 <gehel> rolling restart of wdqs to enable async logging - T198051 [production]
08:11 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1080 after alter table (duration: 00m 58s) [production]
07:46 <gehel> depooling wdqs1005 to allow it to catch up on updates - T198042 [production]
07:33 <moritzm> slow rollout of most of the remaining debmonitor client installations [production]
05:43 <marostegui> Deploy schema change on db1080 T191316 T192926 T89737 T195193 [production]
05:41 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1080 for alter table (duration: 01m 17s) [production]
02:39 <l10nupdate@deploy1001> scap sync-l10n completed (1.32.0-wmf.999) (duration: 07m 37s) [production]
02:21 <l10nupdate@deploy1001> scap sync-l10n completed (1.32.0-wmf.8) (duration: 08m 40s) [production]
2018-06-24 §
22:26 <gehel> restarting wdqs1004 - T198042 [production]
21:51 <gehel> restarting wdqs1005 after taking a few thread dumps - T198042 [production]
19:41 <gehel> restart blazegraph on wdqs1004 [production]
16:09 <godog> restart wdqs-updater on wdqs1005 [production]
15:40 <godog> restart wdqs-blazegraph on wdqs1005 [production]
15:40 <jynus> restart graphite2001 [production]
15:35 <godog> systemctl restart wdqs-blazegraph on wdqs1004 [production]
15:33 <godog> restart-wdqs on wdqs1004 [production]
2018-06-22 §
19:54 <jynus> applying transaction manually on dbstore1002 due to weird bug with savepoint on tokudb+image table [production]
2018-06-21 §
21:38 <ebernhardson> banned elastic1036 from search cluster, waited for all load to shift away, and unbanned [production]
16:38 <ejegg> updated CiviCRM from 349d43eb8b to e2c8ddd70e [production]
13:28 <vgutierrez> Bump AES128-SHA pageview replacement to 4% [production]
12:49 <chasemp> remove labvirt1019 canary to start debug of network [production]
11:08 <hashar> Refreshed operations-puppet-tests-docker jenkins job to a new Docker container build that includes isc-dhcp-server | https://gerrit.wikimedia.org/r/c/integration/config/+/441367 [production]
02:51 <l10nupdate@deploy1001> scap sync-l10n completed (1.32.0-wmf.999) (duration: 07m 40s) [production]
02:33 <l10nupdate@deploy1001> scap sync-l10n completed (1.32.0-wmf.8) (duration: 14m 12s) [production]
2018-06-20 §
21:18 <ejegg> updated 'domain' settings in CiviCRM so name is now 'Wikimedia Foundation' and description is 'donate.wikimedia.org' [production]
20:08 <bearND> rolled back "Update mobileapps to 9d856ec" (was just on canary) [production]
20:07 <bsitzmann@deploy1001> Finished deploy [mobileapps/deploy@3420e67]: Update mobileapps to 9d856ec (duration: 03m 51s) [production]
20:03 <bsitzmann@deploy1001> Started deploy [mobileapps/deploy@3420e67]: Update mobileapps to 9d856ec [production]
19:54 <ottomata> removed Kafka MirrorMaker from kafka10(12|13|14) [production]
16:01 <sbisson@deploy1001> Finished deploy [kartotherian/deploy@9af9191]: Kartotherian: make Pl fallback to EN (duration: 03m 08s) [production]
15:58 <sbisson@deploy1001> Started deploy [kartotherian/deploy@9af9191]: Kartotherian: make Pl fallback to EN [production]
15:57 <sbisson@deploy1001> Finished deploy [kartotherian/deploy@9af9191]: Kartotherian: make Pl fallback to EN (duration: 00m 27s) [production]
15:56 <sbisson@deploy1001> Started deploy [kartotherian/deploy@9af9191]: Kartotherian: make Pl fallback to EN [production]
15:52 <sbisson@deploy1001> Finished deploy [kartotherian/deploy@9af9191]: (no justification provided) (duration: 03m 36s) [production]
15:49 <sbisson@deploy1001> Started deploy [kartotherian/deploy@9af9191]: (no justification provided) [production]
14:59 <dcausse> disk space issue on elastic1020 is due to shard rebalancing (currently receiving 2 enwiki_general shards but removing one wikidatawiki_content) [production]
14:07 <imarlier@deploy1001> Finished deploy [performance/navtiming@742edb0]: (no justification provided) (duration: 00m 04s) [production]