|
2017-04-26
ยง
|
| 19:50 |
<elukey> |
restart kafka nodes (kafka1018 and kafka1020) after network maintenance |
[production] |
| 19:45 |
<twentyafterfour@naos> |
rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.29.0-wmf.20 |
[production] |
| 19:42 |
<twentyafterfour> |
rolling back group1 to wmf.20 due to T163896 refs T161733 |
[production] |
| 19:31 |
<twentyafterfour@naos> |
rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.29.0-wmf.21 |
[production] |
| 19:24 |
<twentyafterfour> |
begin deployment train: group1 wikis to 1.29.0-wmf.21 refs T161733 |
[production] |
| 19:22 |
<bblack> |
initiating cumin-based restart of all varnish backends for cache_upload in codfw to downgrade from experimental package. 30 minute spacing, 10 hosts, ~5h to completion... |
[production] |
| 19:17 |
<thcipriani@naos> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:350442|Disable collectionsaveascommunitypage right on es.wikipedia]] T163767 (duration: 00m 49s) |
[production] |
| 19:05 |
<bblack> |
restarting varnish frontend and backend on cp3033 to downgrade |
[production] |
| 19:03 |
<bblack> |
restaring varnish-frontend on cp2014 to downgrade |
[production] |
| 18:58 |
<thcipriani@naos> |
Synchronized wmf-config/CommonSettings.php: SWAT: [[gerrit:350459|Workaround issue of overriding whitelist config variable]] T163114 (duration: 00m 53s) |
[production] |
| 18:56 |
<bblack> |
downgrading varnish back to 4.1.5-wm1 on all -wm2 hosts |
[production] |
| 18:50 |
<thcipriani@naos> |
Synchronized php-1.29.0-wmf.21/extensions/CirrusSearch: SWAT: [[gerrit:350453|Provide a way to blacklist a set of wikis for crosswiki search]] T163546 (duration: 01m 02s) |
[production] |
| 18:44 |
<thcipriani@naos> |
Synchronized wmf-config/CirrusSearch-common.php: SWAT: [[gerrit:350456|Adjust sistersearch against wikivoyage to require title matching]] T163547 (duration: 01m 11s) |
[production] |
| 18:38 |
<thcipriani@naos> |
Synchronized wmf-config/CirrusSearch-common.php: SWAT: [[gerrit:350452|Configure multimedia search template boosting]] T163223 (duration: 00m 53s) |
[production] |
| 18:30 |
<thcipriani@naos> |
Synchronized php-1.29.0-wmf.20/extensions/SecurePoll: SWAT: [[gerrit:350444|Add voter scripts for board/fdc election 2017]] T163854 (duration: 00m 57s) |
[production] |
| 18:26 |
<thcipriani@naos> |
Synchronized php-1.29.0-wmf.21/extensions/SecurePoll: SWAT: [[gerrit:350443|Add voter scripts for board/fdc election 2017]] T163854 (duration: 01m 00s) |
[production] |
| 18:23 |
<thcipriani@naos> |
Synchronized dblists/commonsuploads.dblist: SWAT: [[gerrit:350439|Enable local uploads on knwiki]] T133137 (duration: 01m 06s) |
[production] |
| 18:16 |
<ema> |
start varnish-frontend on cp2014 |
[production] |
| 18:14 |
<jynus> |
running alter table on all wikis of s3 T163912 |
[production] |
| 17:49 |
<jynus> |
rebooting es1019 for upgrading and to fix race condition on services |
[production] |
| 17:46 |
<elukey> |
restart nutcracker on the eqiad mw hosts to pick up the new shard config (spamming elasticsearch memcached and triggering alarms) |
[production] |
| 17:44 |
<elukey> |
unmasking and starting daemons on restbase-dev1003 |
[production] |
| 17:41 |
<reedy@naos> |
Synchronized wmf-config/InitialiseSettings.php: touch (duration: 01m 23s) |
[production] |
| 17:02 |
<mobrovac@naos> |
Started restart [trending-edits/deploy@7112062]: Restart for ICU lib update |
[production] |
| 17:01 |
<mobrovac@naos> |
Started restart [mobileapps/deploy@5c2b9a9]: Restart for ICU lib update |
[production] |
| 17:00 |
<mobrovac@naos> |
Started restart [mathoid/deploy@7eb4092]: Restart for ICU lib update |
[production] |
| 16:43 |
<mobrovac@naos> |
Started restart [electron-render/deploy@9156760]: Restart for ICU lib update |
[production] |
| 16:39 |
<mobrovac@naos> |
Started restart [graphoid/deploy@128206b]: Restart for ICU lib update |
[production] |
| 16:37 |
<mobrovac@naos> |
Started restart [eventstreams/deploy@05bcc8f]: Restart for ICU lib update |
[production] |
| 16:37 |
<mobrovac@naos> |
Started restart [electron-render/deploy@9156760]: Restart for ICU lib update |
[production] |
| 16:36 |
<mobrovac@naos> |
Started restart [cxserver/deploy@6899032]: Restart for ICU lib update |
[production] |
| 16:34 |
<mobrovac@naos> |
Started restart [citoid/deploy@b8c4cb2]: Restart for ICU lib update |
[production] |
| 16:14 |
<elukey> |
stop and mask cassandra and restbase on restbase-dev1003 for row-d maintenance |
[production] |
| 16:07 |
<_joe_> |
disabled and masked strongswan, memcached, redis on mc1013-17 for decommissioning |
[production] |
| 15:43 |
<XioNoX> |
VRRP priority removed, interfaces cr2/asw2 renamed - T148506 |
[production] |
| 15:40 |
<_joe_> |
shutting down conf1003 T148506 |
[production] |
| 15:33 |
<XioNoX> |
"cr2-eqiad# delete interfaces ae4 disable" done, confirmed links and LACP are up - T148506 |
[production] |
| 15:33 |
<XioNoX> |
"cr2-eqiad# delete interfaces ae4 disable" done, confirmed links and LACP are up |
[production] |
| 15:24 |
<marostegui> |
Shutdown es2019 for maintenance with papaul and Dell - T149526 |
[production] |
| 15:12 |
<XioNoX> |
switch ports for rack D7 and D8 configured - T148506 |
[production] |
| 14:47 |
<marostegui> |
Stop MySQL db1070 (just in case) to test drac cold restart |
[production] |
| 14:47 |
<bblack@neodymium> |
conftool action : set/pooled=no; selector: dc=eqiad,cluster=cache_upload,name=cp107[1234].eqiad.wmnet |
[production] |
| 14:26 |
<elukey> |
depooling aqs100[69] from AQS for network maintenance |
[production] |
| 14:20 |
<elukey> |
stop zookeeper on conf1003 for row-d maintenance (Hadoop, Kafka related) |
[production] |
| 14:04 |
<XioNoX> |
"cr2-eqiad# set interfaces ae4 disable" done, (1 ping loss) - T148506 |
[production] |
| 14:00 |
<marostegui@naos> |
Synchronized wmf-config/db-eqiad.php: Repool db1026, depool db1045 - T162539 T163548 (duration: 00m 53s) |
[production] |
| 13:59 |
<XioNoX> |
lowered VRRP priority for T148506 |
[production] |
| 13:58 |
<andrewbogott> |
put labservices1001 into downtime to minimize (but probably not totally eliminate) alert spam |
[production] |
| 13:56 |
<andrewbogott> |
disabled instance creation on Horizon via https://gerrit.wikimedia.org/r/#/c/350414/ and on wikitech via a strategic edit in extensions/OpenStackManager/special/SpecialNovaInstance.php |
[production] |
| 13:56 |
<godog> |
downtime and poweroff ms-be 21 26 27 37 38 39 before switch relocation - T148506 |
[production] |