|
2017-04-26
§
|
| 16:07 |
<_joe_> |
disabled and masked strongswan, memcached, redis on mc1013-17 for decommissioning |
[production] |
| 15:43 |
<XioNoX> |
VRRP priority removed, interfaces cr2/asw2 renamed - T148506 |
[production] |
| 15:40 |
<_joe_> |
shutting down conf1003 T148506 |
[production] |
| 15:33 |
<XioNoX> |
"cr2-eqiad# delete interfaces ae4 disable" done, confirmed links and LACP are up - T148506 |
[production] |
| 15:33 |
<XioNoX> |
"cr2-eqiad# delete interfaces ae4 disable" done, confirmed links and LACP are up |
[production] |
| 15:24 |
<marostegui> |
Shutdown es2019 for maintenance with papaul and Dell - T149526 |
[production] |
| 15:12 |
<XioNoX> |
switch ports for rack D7 and D8 configured - T148506 |
[production] |
| 14:47 |
<marostegui> |
Stop MySQL db1070 (just in case) to test drac cold restart |
[production] |
| 14:47 |
<bblack@neodymium> |
conftool action : set/pooled=no; selector: dc=eqiad,cluster=cache_upload,name=cp107[1234].eqiad.wmnet |
[production] |
| 14:26 |
<elukey> |
depooling aqs100[69] from AQS for network maintenance |
[production] |
| 14:20 |
<elukey> |
stop zookeeper on conf1003 for row-d maintenance (Hadoop, Kafka related) |
[production] |
| 14:04 |
<XioNoX> |
"cr2-eqiad# set interfaces ae4 disable" done, (1 ping loss) - T148506 |
[production] |
| 14:00 |
<marostegui@naos> |
Synchronized wmf-config/db-eqiad.php: Repool db1026, depool db1045 - T162539 T163548 (duration: 00m 53s) |
[production] |
| 13:59 |
<XioNoX> |
lowered VRRP priority for T148506 |
[production] |
| 13:58 |
<andrewbogott> |
put labservices1001 into downtime to minimize (but probably not totally eliminate) alert spam |
[production] |
| 13:56 |
<andrewbogott> |
disabled instance creation on Horizon via https://gerrit.wikimedia.org/r/#/c/350414/ and on wikitech via a strategic edit in extensions/OpenStackManager/special/SpecialNovaInstance.php |
[production] |
| 13:56 |
<godog> |
downtime and poweroff ms-be 21 26 27 37 38 39 before switch relocation - T148506 |
[production] |
| 13:54 |
<gehel> |
downtime "ElasticSearch health check for shards" checks for logstash and elasticsearch eqiad - T148506 |
[production] |
| 13:53 |
<elukey> |
stop kafka on kafka1020 and kafka1018 for row-d extended maintenance (D2) |
[production] |
| 13:44 |
<_joe_> |
shutting down mc1013-18 for row D maintenance |
[production] |
| 13:40 |
<aude@naos> |
Synchronized wmf-config/CommonSettings-labs.php: (no justification provided) (duration: 00m 57s) |
[production] |
| 13:32 |
<aude@naos> |
Synchronized wmf-config/Wikibase-production.php: disable tabular-data for now on wikidata and enable echo notification on test wikis (duration: 01m 06s) |
[production] |
| 13:29 |
<marostegui> |
Deploy alter table on db1069 (wikidatawiki) https://phabricator.wikimedia.org/T162539 https://phabricator.wikimedia.org/T163548 |
[production] |
| 13:27 |
<marostegui> |
Deploy alter table labsdb1001 https://phabricator.wikimedia.org/T162539 https://phabricator.wikimedia.org/T163548 |
[production] |
| 13:23 |
<marostegui> |
Deploy alter table db1045 - https://phabricator.wikimedia.org/T162539 https://phabricator.wikimedia.org/T163548 |
[production] |
| 13:22 |
<elukey> |
restart HDFS on analytics100[12] (Hadoop master nodes) to pick up recent topology changes for the cluster |
[production] |
| 13:10 |
<aude@naos> |
Synchronized wmf-config/throttle.php: (no justification provided) (duration: 01m 23s) |
[production] |
| 13:02 |
<ema@neodymium> |
conftool action : set/pooled=yes; selector: name=cp2014.codfw.wmnet,service=varnish-be |
[production] |
| 13:00 |
<ema> |
cp2017: restart varnish-be |
[production] |
| 12:56 |
<marostegui> |
Shutdown db1092 for maintenance - https://phabricator.wikimedia.org/T162681 |
[production] |
| 12:55 |
<gehel> |
restart elasticsearch on relforge1001 to validate new config - T161830 |
[production] |
| 12:46 |
<moritzm> |
installing mysql security updates (5.5 as packaged in Debian jessie) |
[production] |
| 12:43 |
<ema@neodymium> |
conftool action : set/pooled=no; selector: name=cp2014.codfw.wmnet,service=varnish-be |
[production] |
| 11:32 |
<jynus> |
applying new events_coredb_slave.sql on db2055 T160984 |
[production] |
| 11:31 |
<moritzm> |
rebooting mwlog2001 for update to Linux 4.9 |
[production] |
| 10:47 |
<ladsgroup@naos> |
Synchronized wmf-config/Wikibase-labs.php: T142104, part II (duration: 00m 56s) |
[production] |
| 10:45 |
<ladsgroup@naos> |
Synchronized static/images/wikibase/echoIcon.svg: T142104, part I (duration: 01m 04s) |
[production] |
| 10:44 |
<marostegui> |
Deploy alter table on s5, on db1063 (eqiad master) for tables: change_tag and tag_summary - https://phabricator.wikimedia.org/T147166 |
[production] |
| 10:39 |
<jynus@naos> |
Synchronized wmf-config/db-eqiad.php: switch s5 eqiad master from db1049 to db1063 (duration: 01m 24s) |
[production] |
| 09:48 |
<jynus> |
migrating s5 eqiad replicas under db1063 |
[production] |
| 09:42 |
<jynus> |
restarting mariadb at db1063 |
[production] |
| 09:24 |
<marostegui> |
Shutdown db1094, db1093, db1091 for maintenance - T162681 |
[production] |
| 09:16 |
<marostegui> |
Shutdown es1019 for maintenance - T162681 |
[production] |
| 08:32 |
<elukey> |
Gracefully stopping hadoop daemons on Hadoop nodes affected by Row-D maintenance |
[production] |
| 08:29 |
<marostegui> |
Deploy alter table on change_tag and tag_summary on silver and labtestweb2001 - T147166 |
[production] |
| 08:27 |
<marostegui@naos> |
Synchronized wmf-config/db-eqiad.php: Depool hosts that need to be moved for the network maintenance - T162681 (duration: 02m 25s) |
[production] |
| 08:22 |
<moritzm> |
reimaging terbium to jessie |
[production] |
| 07:59 |
<jynus> |
shutting down mariadb on db1040 as a backup before decommissioning |
[production] |
| 07:48 |
<marostegui> |
Deploy alter table on s1, on db1052 (eqiad master) for tables: change_tag and tag_summary - https://phabricator.wikimedia.org/T147166 |
[production] |
| 07:30 |
<marostegui> |
Deploy alter table on s7, on db1062 (eqiad master) for tables: change_tag and tag_summary - https://phabricator.wikimedia.org/T147166 |
[production] |