|
2017-04-12
ยง
|
| 14:31 |
<andrewbogott> |
running maintain-meta_p on labsdb1001/1003/1009/1010/1011 |
[production] |
| 14:23 |
<hashar> |
Restarting Jenkins for git/scm plugins updates |
[production] |
| 14:06 |
<hashar> |
European SWAT complete |
[production] |
| 13:51 |
<switchdc> |
(volans@neodymium) END TASK - switchdc.stages.t05_switch_traffic(codfw, eqiad) Successfully completed |
[production] |
| 13:48 |
<switchdc> |
(volans@neodymium) START TASK - switchdc.stages.t05_switch_traffic(codfw, eqiad) Switch traffic flow to the appservers in the new datacenter |
[production] |
| 13:42 |
<volans> |
testing t05_switch_traffic of the switchdc |
[production] |
| 13:41 |
<elukey> |
apply SLOWLOG RESET and CONFIG SET slowlog-max-len 100000 (prev value 10000, 10ms) to rdb1005:6380 to track down slow reqs - T125735 |
[production] |
| 13:37 |
<hoo@tin> |
Synchronized php-1.29.0-wmf.20/extensions/Wikidata: Update Wikibase/ ArticlePlaceholder (duration: 02m 19s) |
[production] |
| 13:33 |
<hoo@tin> |
Synchronized php-1.29.0-wmf.19/extensions/Wikidata: Update Wikibase/ ArticlePlaceholder (duration: 02m 16s) |
[production] |
| 13:33 |
<elukey> |
restored slowlog-log-slower-than 10000 on rdb2005 |
[production] |
| 13:25 |
<elukey> |
applied CONFIG SET slowlog-log-slower-than 300000 to Redis 6379 on rdb2005 and reset slowlog history to play with the stats |
[production] |
| 13:10 |
<addshore@tin> |
Synchronized php-1.29.0-wmf.20/extensions/WikimediaEvents/extension.json: [[gerrit:347773|WMDE Spring campaign]] PT2/2 (duration: 00m 45s) |
[production] |
| 13:09 |
<addshore@tin> |
Synchronized php-1.29.0-wmf.20/extensions/WikimediaEvents/WikimediaEventsHooks.php: [[gerrit:347773|WMDE Spring campaign]] PT1/2 (duration: 00m 45s) |
[production] |
| 13:08 |
<hashar@tin> |
Synchronized wmf-config/InitialiseSettings.php: Revert "Temporarily enable change dispatch logging on testwikidata" - T159828 (duration: 00m 47s) |
[production] |
| 12:23 |
<elukey> |
restart HDFS datanode daemons on all the Hadoop worker node to pick up the new JVM settings |
[production] |
| 12:18 |
<kartik@tin> |
Finished deploy [cxserver/deploy@2842efa]: Update cxserver to 56a012d (duration: 03m 58s) |
[production] |
| 12:14 |
<kartik@tin> |
Started deploy [cxserver/deploy@2842efa]: Update cxserver to 56a012d |
[production] |
| 11:57 |
<elukey> |
restart Yarn nodemanager daemons on all the Hadoop worker node to pick up the new JVM settings |
[production] |
| 11:05 |
<_joe_> |
downgrading python-urllib3 on puppetmaster1001 |
[production] |
| 11:02 |
<akosiaris> |
upgrade puppet across the trusty fleet to 3.8. T162462 |
[production] |
| 10:34 |
<hashar> |
Upgrading Jenkins "Email Extension" plugin 2.57.1..2.57.2 and restarting Jenkins |
[production] |
| 10:07 |
<hashar> |
Upgrading Jenkins "Git client" plugin 2.3.0..2.4.1 and restarting Jenkins |
[production] |
| 09:58 |
<switchdc> |
(volans@neodymium) END TASK - switchdc.stages.t07_coredb_masters_readwrite(codfw, eqiad) Successfully completed |
[production] |
| 09:58 |
<switchdc> |
(volans@neodymium) START TASK - switchdc.stages.t07_coredb_masters_readwrite(codfw, eqiad) set core DB masters in read-write mode |
[production] |
| 09:56 |
<switchdc> |
(volans@neodymium) END TASK - switchdc.stages.t03_coredb_masters_readonly(codfw, eqiad) Failed to execute |
[production] |
| 09:56 |
<switchdc> |
(volans@neodymium) START TASK - switchdc.stages.t03_coredb_masters_readonly(codfw, eqiad) set core DB masters in read-only mode |
[production] |
| 09:53 |
<_joe_> |
removing the old directory of data from ocg1003 |
[production] |
| 09:52 |
<volans> |
testing t03 and t07 DB-RO/RW stages of switchdc (codfw->eqiad), we are already in that situation, t03 will fail the verfication, is expected |
[production] |
| 09:52 |
<godog> |
swift codfw-prod: ms-be2001 - ms-be2012 initial decom - T162785 |
[production] |
| 09:47 |
<_joe_> |
remounting the new partition under /srv/deployment/ocg/output, cleaning out the old dir. Will cause a service interruption for requests to ocg1003 for a few minutes. T162780 |
[production] |
| 09:42 |
<gehel> |
starting load on elastic2020 - T149006 |
[production] |
| 09:41 |
<addshore@tin> |
Synchronized wmf-config/InitialiseSettings.php: [[gerrit:347819|wmgUseGettingStarted false for dewiki]] (duration: 00m 45s) |
[production] |
| 09:26 |
<addshore@tin> |
Synchronized wmf-config/InitialiseSettings.php: [[gerrit:347817|WMDE Spring campaign - Add logging from WikimediaEvent]] (duration: 00m 46s) |
[production] |
| 09:22 |
<hashar> |
Restarting Jenkins for Matrix related plugins updates (3) |
[production] |
| 09:12 |
<_joe_> |
copying data from / to the neww partition on ocg1003 T162462 |
[production] |
| 09:10 |
<hashar> |
Restarting Jenkins for plugins update (2) |
[production] |
| 09:06 |
<_joe_> |
creating a LVM volume on ocg1003 |
[production] |
| 09:05 |
<hashar> |
Restarting Jenkins for plugins update |
[production] |
| 08:59 |
<addshore@tin> |
Synchronized php-1.29.0-wmf.19/extensions/WikimediaEvents/extension.json: [[gerrit:347815|patch1]] & [[gerrit:347774|patch2]] WMDE Spring campaign PT2/2 (duration: 00m 45s) |
[production] |
| 08:58 |
<addshore@tin> |
Synchronized php-1.29.0-wmf.19/extensions/WikimediaEvents/WikimediaEventsHooks.php: [[gerrit:347815|patch1]] & [[gerrit:347774|patch2]] WMDE Spring campaign PT1/2 (duration: 00m 47s) |
[production] |
| 08:52 |
<ema> |
upgrade cache_upload to linux 4.9 T162029 |
[production] |
| 08:44 |
<gehel> |
reimaging elastic2020 for testing - T149006 |
[production] |
| 08:24 |
<switchdc> |
(oblivian@sarin) END TASK - switchdc.stages.t09_start_maintenance(codfw, eqiad) Successfully completed |
[production] |
| 08:22 |
<switchdc> |
(oblivian@sarin) START TASK - switchdc.stages.t09_start_maintenance(codfw, eqiad) Start MediaWiki maintenance in the new master DC |
[production] |
| 08:14 |
<switchdc> |
(oblivian@sarin) END TASK - switchdc.stages.t08_stop_mediawiki_readonly(codfw, eqiad) Failed to execute |
[production] |
| 08:14 |
<root@tin> |
Synchronized wmf-config/db-eqiad.php: Set MediaWiki in read-write mode in datacenter eqiad (duration: 00m 35s) |
[production] |
| 08:13 |
<switchdc> |
(oblivian@sarin) START TASK - switchdc.stages.t08_stop_mediawiki_readonly(codfw, eqiad) Set MediaWiki in read-write mode (db_to config already merged and git pulled) |
[production] |
| 08:09 |
<switchdc> |
(oblivian@sarin) END TASK - switchdc.stages.t06_redis(codfw, eqiad) Successfully completed |
[production] |
| 08:09 |
<switchdc> |
(oblivian@sarin) START TASK - switchdc.stages.t06_redis(codfw, eqiad) Switch the Redis replication |
[production] |
| 08:02 |
<switchdc> |
(oblivian@sarin) END TASK - switchdc.stages.t05_switch_datacenter(codfw, eqiad) Successfully completed |
[production] |