851-900 of 10000 results (32ms)
2017-04-26 ยง
19:24 <twentyafterfour> begin deployment train: group1 wikis to 1.29.0-wmf.21 refs T161733 [production]
19:22 <bblack> initiating cumin-based restart of all varnish backends for cache_upload in codfw to downgrade from experimental package. 30 minute spacing, 10 hosts, ~5h to completion... [production]
19:17 <thcipriani@naos> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:350442|Disable collectionsaveascommunitypage right on es.wikipedia]] T163767 (duration: 00m 49s) [production]
19:05 <bblack> restarting varnish frontend and backend on cp3033 to downgrade [production]
19:03 <bblack> restaring varnish-frontend on cp2014 to downgrade [production]
18:58 <thcipriani@naos> Synchronized wmf-config/CommonSettings.php: SWAT: [[gerrit:350459|Workaround issue of overriding whitelist config variable]] T163114 (duration: 00m 53s) [production]
18:56 <bblack> downgrading varnish back to 4.1.5-wm1 on all -wm2 hosts [production]
18:50 <thcipriani@naos> Synchronized php-1.29.0-wmf.21/extensions/CirrusSearch: SWAT: [[gerrit:350453|Provide a way to blacklist a set of wikis for crosswiki search]] T163546 (duration: 01m 02s) [production]
18:44 <thcipriani@naos> Synchronized wmf-config/CirrusSearch-common.php: SWAT: [[gerrit:350456|Adjust sistersearch against wikivoyage to require title matching]] T163547 (duration: 01m 11s) [production]
18:38 <thcipriani@naos> Synchronized wmf-config/CirrusSearch-common.php: SWAT: [[gerrit:350452|Configure multimedia search template boosting]] T163223 (duration: 00m 53s) [production]
18:30 <thcipriani@naos> Synchronized php-1.29.0-wmf.20/extensions/SecurePoll: SWAT: [[gerrit:350444|Add voter scripts for board/fdc election 2017]] T163854 (duration: 00m 57s) [production]
18:26 <thcipriani@naos> Synchronized php-1.29.0-wmf.21/extensions/SecurePoll: SWAT: [[gerrit:350443|Add voter scripts for board/fdc election 2017]] T163854 (duration: 01m 00s) [production]
18:23 <thcipriani@naos> Synchronized dblists/commonsuploads.dblist: SWAT: [[gerrit:350439|Enable local uploads on knwiki]] T133137 (duration: 01m 06s) [production]
18:16 <ema> start varnish-frontend on cp2014 [production]
18:14 <jynus> running alter table on all wikis of s3 T163912 [production]
17:49 <jynus> rebooting es1019 for upgrading and to fix race condition on services [production]
17:46 <elukey> restart nutcracker on the eqiad mw hosts to pick up the new shard config (spamming elasticsearch memcached and triggering alarms) [production]
17:44 <elukey> unmasking and starting daemons on restbase-dev1003 [production]
17:41 <reedy@naos> Synchronized wmf-config/InitialiseSettings.php: touch (duration: 01m 23s) [production]
17:02 <mobrovac@naos> Started restart [trending-edits/deploy@7112062]: Restart for ICU lib update [production]
17:01 <mobrovac@naos> Started restart [mobileapps/deploy@5c2b9a9]: Restart for ICU lib update [production]
17:00 <mobrovac@naos> Started restart [mathoid/deploy@7eb4092]: Restart for ICU lib update [production]
16:43 <mobrovac@naos> Started restart [electron-render/deploy@9156760]: Restart for ICU lib update [production]
16:39 <mobrovac@naos> Started restart [graphoid/deploy@128206b]: Restart for ICU lib update [production]
16:37 <mobrovac@naos> Started restart [eventstreams/deploy@05bcc8f]: Restart for ICU lib update [production]
16:37 <mobrovac@naos> Started restart [electron-render/deploy@9156760]: Restart for ICU lib update [production]
16:36 <mobrovac@naos> Started restart [cxserver/deploy@6899032]: Restart for ICU lib update [production]
16:34 <mobrovac@naos> Started restart [citoid/deploy@b8c4cb2]: Restart for ICU lib update [production]
16:14 <elukey> stop and mask cassandra and restbase on restbase-dev1003 for row-d maintenance [production]
16:07 <_joe_> disabled and masked strongswan, memcached, redis on mc1013-17 for decommissioning [production]
15:43 <XioNoX> VRRP priority removed, interfaces cr2/asw2 renamed - T148506 [production]
15:40 <_joe_> shutting down conf1003 T148506 [production]
15:33 <XioNoX> "cr2-eqiad# delete interfaces ae4 disable" done, confirmed links and LACP are up - T148506 [production]
15:33 <XioNoX> "cr2-eqiad# delete interfaces ae4 disable" done, confirmed links and LACP are up [production]
15:24 <marostegui> Shutdown es2019 for maintenance with papaul and Dell - T149526 [production]
15:12 <XioNoX> switch ports for rack D7 and D8 configured - T148506 [production]
14:47 <marostegui> Stop MySQL db1070 (just in case) to test drac cold restart [production]
14:47 <bblack@neodymium> conftool action : set/pooled=no; selector: dc=eqiad,cluster=cache_upload,name=cp107[1234].eqiad.wmnet [production]
14:26 <elukey> depooling aqs100[69] from AQS for network maintenance [production]
14:20 <elukey> stop zookeeper on conf1003 for row-d maintenance (Hadoop, Kafka related) [production]
14:04 <XioNoX> "cr2-eqiad# set interfaces ae4 disable" done, (1 ping loss) - T148506 [production]
14:00 <marostegui@naos> Synchronized wmf-config/db-eqiad.php: Repool db1026, depool db1045 - T162539 T163548 (duration: 00m 53s) [production]
13:59 <XioNoX> lowered VRRP priority for T148506 [production]
13:58 <andrewbogott> put labservices1001 into downtime to minimize (but probably not totally eliminate) alert spam [production]
13:56 <andrewbogott> disabled instance creation on Horizon via https://gerrit.wikimedia.org/r/#/c/350414/ and on wikitech via a strategic edit in extensions/OpenStackManager/special/SpecialNovaInstance.php [production]
13:56 <godog> downtime and poweroff ms-be 21 26 27 37 38 39 before switch relocation - T148506 [production]
13:54 <gehel> downtime "ElasticSearch health check for shards" checks for logstash and elasticsearch eqiad - T148506 [production]
13:53 <elukey> stop kafka on kafka1020 and kafka1018 for row-d extended maintenance (D2) [production]
13:44 <_joe_> shutting down mc1013-18 for row D maintenance [production]
13:40 <aude@naos> Synchronized wmf-config/CommonSettings-labs.php: (no justification provided) (duration: 00m 57s) [production]