|
2017-04-12
ยง
|
| 11:02 |
<akosiaris> |
upgrade puppet across the trusty fleet to 3.8. T162462 |
[production] |
| 10:34 |
<hashar> |
Upgrading Jenkins "Email Extension" plugin 2.57.1..2.57.2 and restarting Jenkins |
[production] |
| 10:07 |
<hashar> |
Upgrading Jenkins "Git client" plugin 2.3.0..2.4.1 and restarting Jenkins |
[production] |
| 09:58 |
<switchdc> |
(volans@neodymium) END TASK - switchdc.stages.t07_coredb_masters_readwrite(codfw, eqiad) Successfully completed |
[production] |
| 09:58 |
<switchdc> |
(volans@neodymium) START TASK - switchdc.stages.t07_coredb_masters_readwrite(codfw, eqiad) set core DB masters in read-write mode |
[production] |
| 09:56 |
<switchdc> |
(volans@neodymium) END TASK - switchdc.stages.t03_coredb_masters_readonly(codfw, eqiad) Failed to execute |
[production] |
| 09:56 |
<switchdc> |
(volans@neodymium) START TASK - switchdc.stages.t03_coredb_masters_readonly(codfw, eqiad) set core DB masters in read-only mode |
[production] |
| 09:53 |
<_joe_> |
removing the old directory of data from ocg1003 |
[production] |
| 09:52 |
<volans> |
testing t03 and t07 DB-RO/RW stages of switchdc (codfw->eqiad), we are already in that situation, t03 will fail the verfication, is expected |
[production] |
| 09:52 |
<godog> |
swift codfw-prod: ms-be2001 - ms-be2012 initial decom - T162785 |
[production] |
| 09:47 |
<_joe_> |
remounting the new partition under /srv/deployment/ocg/output, cleaning out the old dir. Will cause a service interruption for requests to ocg1003 for a few minutes. T162780 |
[production] |
| 09:42 |
<gehel> |
starting load on elastic2020 - T149006 |
[production] |
| 09:41 |
<addshore@tin> |
Synchronized wmf-config/InitialiseSettings.php: [[gerrit:347819|wmgUseGettingStarted false for dewiki]] (duration: 00m 45s) |
[production] |
| 09:26 |
<addshore@tin> |
Synchronized wmf-config/InitialiseSettings.php: [[gerrit:347817|WMDE Spring campaign - Add logging from WikimediaEvent]] (duration: 00m 46s) |
[production] |
| 09:22 |
<hashar> |
Restarting Jenkins for Matrix related plugins updates (3) |
[production] |
| 09:12 |
<_joe_> |
copying data from / to the neww partition on ocg1003 T162462 |
[production] |
| 09:10 |
<hashar> |
Restarting Jenkins for plugins update (2) |
[production] |
| 09:06 |
<_joe_> |
creating a LVM volume on ocg1003 |
[production] |
| 09:05 |
<hashar> |
Restarting Jenkins for plugins update |
[production] |
| 08:59 |
<addshore@tin> |
Synchronized php-1.29.0-wmf.19/extensions/WikimediaEvents/extension.json: [[gerrit:347815|patch1]] & [[gerrit:347774|patch2]] WMDE Spring campaign PT2/2 (duration: 00m 45s) |
[production] |
| 08:58 |
<addshore@tin> |
Synchronized php-1.29.0-wmf.19/extensions/WikimediaEvents/WikimediaEventsHooks.php: [[gerrit:347815|patch1]] & [[gerrit:347774|patch2]] WMDE Spring campaign PT1/2 (duration: 00m 47s) |
[production] |
| 08:52 |
<ema> |
upgrade cache_upload to linux 4.9 T162029 |
[production] |
| 08:44 |
<gehel> |
reimaging elastic2020 for testing - T149006 |
[production] |
| 08:24 |
<switchdc> |
(oblivian@sarin) END TASK - switchdc.stages.t09_start_maintenance(codfw, eqiad) Successfully completed |
[production] |
| 08:22 |
<switchdc> |
(oblivian@sarin) START TASK - switchdc.stages.t09_start_maintenance(codfw, eqiad) Start MediaWiki maintenance in the new master DC |
[production] |
| 08:14 |
<switchdc> |
(oblivian@sarin) END TASK - switchdc.stages.t08_stop_mediawiki_readonly(codfw, eqiad) Failed to execute |
[production] |
| 08:14 |
<root@tin> |
Synchronized wmf-config/db-eqiad.php: Set MediaWiki in read-write mode in datacenter eqiad (duration: 00m 35s) |
[production] |
| 08:13 |
<switchdc> |
(oblivian@sarin) START TASK - switchdc.stages.t08_stop_mediawiki_readonly(codfw, eqiad) Set MediaWiki in read-write mode (db_to config already merged and git pulled) |
[production] |
| 08:09 |
<switchdc> |
(oblivian@sarin) END TASK - switchdc.stages.t06_redis(codfw, eqiad) Successfully completed |
[production] |
| 08:09 |
<switchdc> |
(oblivian@sarin) START TASK - switchdc.stages.t06_redis(codfw, eqiad) Switch the Redis replication |
[production] |
| 08:02 |
<switchdc> |
(oblivian@sarin) END TASK - switchdc.stages.t05_switch_datacenter(codfw, eqiad) Successfully completed |
[production] |
| 08:02 |
<switchdc> |
(oblivian@sarin) START TASK - switchdc.stages.t05_switch_datacenter(codfw, eqiad) Switch MediaWiki configuration to the new datacenter |
[production] |
| 08:00 |
<switchdc> |
(oblivian@sarin) END TASK - switchdc.stages.t09_restore_ttl(codfw, eqiad) Successfully completed |
[production] |
| 07:59 |
<switchdc> |
(oblivian@sarin) START TASK - switchdc.stages.t09_restore_ttl(codfw, eqiad) Restore the TTL of all the MediaWiki discovery records |
[production] |
| 07:58 |
<switchdc> |
(oblivian@sarin) END TASK - switchdc.stages.t05_switch_traffic(codfw, eqiad) Successfully completed |
[production] |
| 07:55 |
<switchdc> |
(oblivian@sarin) START TASK - switchdc.stages.t05_switch_traffic(codfw, eqiad) Switch traffic flow to the appservers in the new datacenter |
[production] |
| 07:55 |
<_joe_> |
resuming non-dry run tests of switchdc, all logs from switchdc by me are just tests |
[production] |
| 06:57 |
<_joe_> |
the last messages are just a test and nothing was really done, as codfw is already in read-only mode right now |
[production] |
| 06:57 |
<switchdc> |
(oblivian@sarin) END TASK - switchdc.stages.t02_start_mediawiki_readonly(codfw, eqiad) Failed to execute |
[production] |
| 06:57 |
<root@tin> |
Synchronized wmf-config/db-codfw.php: Set MediaWiki in read-only mode in datacenter codfw (duration: 00m 23s) |
[production] |
| 06:57 |
<switchdc> |
(oblivian@sarin) MediaWiki read-only period starts at: 2017-04-12 06:56:53.822926 |
[production] |
| 06:56 |
<switchdc> |
(oblivian@sarin) START TASK - switchdc.stages.t02_start_mediawiki_readonly(codfw, eqiad) Set MediaWiki in read-only mode (db_from config already merged and git pulled) |
[production] |
| 06:53 |
<switchdc> |
(oblivian@sarin) END TASK - switchdc.stages.t01_stop_maintenance(codfw, eqiad) Failed to execute |
[production] |
| 06:53 |
<switchdc> |
(oblivian@sarin) START TASK - switchdc.stages.t01_stop_maintenance(codfw, eqiad) Stop MediaWiki maintenance in the old master DC |
[production] |
| 06:50 |
<_joe_> |
testing switchover codfw => eqiad, no destructive actions will be taken |
[production] |
| 06:42 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1093 - T17441 (duration: 00m 46s) |
[production] |
| 06:37 |
<elukey> |
reimage mw2246.codfw.wmnet mw2152.codfw.wmnet to remove the /tmp partition (codfw videoscalers, switchover prep) |
[production] |
| 06:32 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1072 - T132416 (duration: 00m 46s) |
[production] |
| 06:28 |
<_joe_> |
killing long-running puppet-agent on db2058 too |
[production] |
| 06:20 |
<_joe_> |
killing badly-started puppet agents on mc1010, tempdb2001,db1090, db2058, hydrogen, possibly others later |
[production] |