|
2017-04-26
§
|
| 17:49 |
<jynus> |
rebooting es1019 for upgrading and to fix race condition on services |
[production] |
| 17:46 |
<elukey> |
restart nutcracker on the eqiad mw hosts to pick up the new shard config (spamming elasticsearch memcached and triggering alarms) |
[production] |
| 17:44 |
<elukey> |
unmasking and starting daemons on restbase-dev1003 |
[production] |
| 17:41 |
<reedy@naos> |
Synchronized wmf-config/InitialiseSettings.php: touch (duration: 01m 23s) |
[production] |
| 17:02 |
<mobrovac@naos> |
Started restart [trending-edits/deploy@7112062]: Restart for ICU lib update |
[production] |
| 17:01 |
<mobrovac@naos> |
Started restart [mobileapps/deploy@5c2b9a9]: Restart for ICU lib update |
[production] |
| 17:00 |
<mobrovac@naos> |
Started restart [mathoid/deploy@7eb4092]: Restart for ICU lib update |
[production] |
| 16:43 |
<mobrovac@naos> |
Started restart [electron-render/deploy@9156760]: Restart for ICU lib update |
[production] |
| 16:39 |
<mobrovac@naos> |
Started restart [graphoid/deploy@128206b]: Restart for ICU lib update |
[production] |
| 16:37 |
<mobrovac@naos> |
Started restart [eventstreams/deploy@05bcc8f]: Restart for ICU lib update |
[production] |
| 16:37 |
<mobrovac@naos> |
Started restart [electron-render/deploy@9156760]: Restart for ICU lib update |
[production] |
| 16:36 |
<mobrovac@naos> |
Started restart [cxserver/deploy@6899032]: Restart for ICU lib update |
[production] |
| 16:34 |
<mobrovac@naos> |
Started restart [citoid/deploy@b8c4cb2]: Restart for ICU lib update |
[production] |
| 16:14 |
<elukey> |
stop and mask cassandra and restbase on restbase-dev1003 for row-d maintenance |
[production] |
| 16:07 |
<_joe_> |
disabled and masked strongswan, memcached, redis on mc1013-17 for decommissioning |
[production] |
| 15:43 |
<XioNoX> |
VRRP priority removed, interfaces cr2/asw2 renamed - T148506 |
[production] |
| 15:40 |
<_joe_> |
shutting down conf1003 T148506 |
[production] |
| 15:33 |
<XioNoX> |
"cr2-eqiad# delete interfaces ae4 disable" done, confirmed links and LACP are up - T148506 |
[production] |
| 15:33 |
<XioNoX> |
"cr2-eqiad# delete interfaces ae4 disable" done, confirmed links and LACP are up |
[production] |
| 15:24 |
<marostegui> |
Shutdown es2019 for maintenance with papaul and Dell - T149526 |
[production] |
| 15:12 |
<XioNoX> |
switch ports for rack D7 and D8 configured - T148506 |
[production] |
| 14:47 |
<marostegui> |
Stop MySQL db1070 (just in case) to test drac cold restart |
[production] |
| 14:47 |
<bblack@neodymium> |
conftool action : set/pooled=no; selector: dc=eqiad,cluster=cache_upload,name=cp107[1234].eqiad.wmnet |
[production] |
| 14:26 |
<elukey> |
depooling aqs100[69] from AQS for network maintenance |
[production] |
| 14:20 |
<elukey> |
stop zookeeper on conf1003 for row-d maintenance (Hadoop, Kafka related) |
[production] |
| 14:04 |
<XioNoX> |
"cr2-eqiad# set interfaces ae4 disable" done, (1 ping loss) - T148506 |
[production] |
| 14:00 |
<marostegui@naos> |
Synchronized wmf-config/db-eqiad.php: Repool db1026, depool db1045 - T162539 T163548 (duration: 00m 53s) |
[production] |
| 13:59 |
<XioNoX> |
lowered VRRP priority for T148506 |
[production] |
| 13:58 |
<andrewbogott> |
put labservices1001 into downtime to minimize (but probably not totally eliminate) alert spam |
[production] |
| 13:56 |
<andrewbogott> |
disabled instance creation on Horizon via https://gerrit.wikimedia.org/r/#/c/350414/ and on wikitech via a strategic edit in extensions/OpenStackManager/special/SpecialNovaInstance.php |
[production] |
| 13:56 |
<godog> |
downtime and poweroff ms-be 21 26 27 37 38 39 before switch relocation - T148506 |
[production] |
| 13:54 |
<gehel> |
downtime "ElasticSearch health check for shards" checks for logstash and elasticsearch eqiad - T148506 |
[production] |
| 13:53 |
<elukey> |
stop kafka on kafka1020 and kafka1018 for row-d extended maintenance (D2) |
[production] |
| 13:44 |
<_joe_> |
shutting down mc1013-18 for row D maintenance |
[production] |
| 13:40 |
<aude@naos> |
Synchronized wmf-config/CommonSettings-labs.php: (no justification provided) (duration: 00m 57s) |
[production] |
| 13:32 |
<aude@naos> |
Synchronized wmf-config/Wikibase-production.php: disable tabular-data for now on wikidata and enable echo notification on test wikis (duration: 01m 06s) |
[production] |
| 13:29 |
<marostegui> |
Deploy alter table on db1069 (wikidatawiki) https://phabricator.wikimedia.org/T162539 https://phabricator.wikimedia.org/T163548 |
[production] |
| 13:27 |
<marostegui> |
Deploy alter table labsdb1001 https://phabricator.wikimedia.org/T162539 https://phabricator.wikimedia.org/T163548 |
[production] |
| 13:23 |
<marostegui> |
Deploy alter table db1045 - https://phabricator.wikimedia.org/T162539 https://phabricator.wikimedia.org/T163548 |
[production] |
| 13:22 |
<elukey> |
restart HDFS on analytics100[12] (Hadoop master nodes) to pick up recent topology changes for the cluster |
[production] |
| 13:10 |
<aude@naos> |
Synchronized wmf-config/throttle.php: (no justification provided) (duration: 01m 23s) |
[production] |
| 13:02 |
<ema@neodymium> |
conftool action : set/pooled=yes; selector: name=cp2014.codfw.wmnet,service=varnish-be |
[production] |
| 13:00 |
<ema> |
cp2017: restart varnish-be |
[production] |
| 12:56 |
<marostegui> |
Shutdown db1092 for maintenance - https://phabricator.wikimedia.org/T162681 |
[production] |
| 12:55 |
<gehel> |
restart elasticsearch on relforge1001 to validate new config - T161830 |
[production] |
| 12:46 |
<moritzm> |
installing mysql security updates (5.5 as packaged in Debian jessie) |
[production] |
| 12:43 |
<ema@neodymium> |
conftool action : set/pooled=no; selector: name=cp2014.codfw.wmnet,service=varnish-be |
[production] |
| 11:32 |
<jynus> |
applying new events_coredb_slave.sql on db2055 T160984 |
[production] |
| 11:31 |
<moritzm> |
rebooting mwlog2001 for update to Linux 4.9 |
[production] |
| 10:47 |
<ladsgroup@naos> |
Synchronized wmf-config/Wikibase-labs.php: T142104, part II (duration: 00m 56s) |
[production] |