|
2017-04-12
§
|
| 09:12 |
<_joe_> |
copying data from / to the neww partition on ocg1003 T162462 |
[production] |
| 09:10 |
<hashar> |
Restarting Jenkins for plugins update (2) |
[production] |
| 09:06 |
<_joe_> |
creating a LVM volume on ocg1003 |
[production] |
| 09:05 |
<hashar> |
Restarting Jenkins for plugins update |
[production] |
| 08:59 |
<addshore@tin> |
Synchronized php-1.29.0-wmf.19/extensions/WikimediaEvents/extension.json: [[gerrit:347815|patch1]] & [[gerrit:347774|patch2]] WMDE Spring campaign PT2/2 (duration: 00m 45s) |
[production] |
| 08:58 |
<addshore@tin> |
Synchronized php-1.29.0-wmf.19/extensions/WikimediaEvents/WikimediaEventsHooks.php: [[gerrit:347815|patch1]] & [[gerrit:347774|patch2]] WMDE Spring campaign PT1/2 (duration: 00m 47s) |
[production] |
| 08:52 |
<ema> |
upgrade cache_upload to linux 4.9 T162029 |
[production] |
| 08:44 |
<gehel> |
reimaging elastic2020 for testing - T149006 |
[production] |
| 08:24 |
<switchdc> |
(oblivian@sarin) END TASK - switchdc.stages.t09_start_maintenance(codfw, eqiad) Successfully completed |
[production] |
| 08:22 |
<switchdc> |
(oblivian@sarin) START TASK - switchdc.stages.t09_start_maintenance(codfw, eqiad) Start MediaWiki maintenance in the new master DC |
[production] |
| 08:14 |
<switchdc> |
(oblivian@sarin) END TASK - switchdc.stages.t08_stop_mediawiki_readonly(codfw, eqiad) Failed to execute |
[production] |
| 08:14 |
<root@tin> |
Synchronized wmf-config/db-eqiad.php: Set MediaWiki in read-write mode in datacenter eqiad (duration: 00m 35s) |
[production] |
| 08:13 |
<switchdc> |
(oblivian@sarin) START TASK - switchdc.stages.t08_stop_mediawiki_readonly(codfw, eqiad) Set MediaWiki in read-write mode (db_to config already merged and git pulled) |
[production] |
| 08:09 |
<switchdc> |
(oblivian@sarin) END TASK - switchdc.stages.t06_redis(codfw, eqiad) Successfully completed |
[production] |
| 08:09 |
<switchdc> |
(oblivian@sarin) START TASK - switchdc.stages.t06_redis(codfw, eqiad) Switch the Redis replication |
[production] |
| 08:02 |
<switchdc> |
(oblivian@sarin) END TASK - switchdc.stages.t05_switch_datacenter(codfw, eqiad) Successfully completed |
[production] |
| 08:02 |
<switchdc> |
(oblivian@sarin) START TASK - switchdc.stages.t05_switch_datacenter(codfw, eqiad) Switch MediaWiki configuration to the new datacenter |
[production] |
| 08:00 |
<switchdc> |
(oblivian@sarin) END TASK - switchdc.stages.t09_restore_ttl(codfw, eqiad) Successfully completed |
[production] |
| 07:59 |
<switchdc> |
(oblivian@sarin) START TASK - switchdc.stages.t09_restore_ttl(codfw, eqiad) Restore the TTL of all the MediaWiki discovery records |
[production] |
| 07:58 |
<switchdc> |
(oblivian@sarin) END TASK - switchdc.stages.t05_switch_traffic(codfw, eqiad) Successfully completed |
[production] |
| 07:55 |
<switchdc> |
(oblivian@sarin) START TASK - switchdc.stages.t05_switch_traffic(codfw, eqiad) Switch traffic flow to the appservers in the new datacenter |
[production] |
| 07:55 |
<_joe_> |
resuming non-dry run tests of switchdc, all logs from switchdc by me are just tests |
[production] |
| 06:57 |
<_joe_> |
the last messages are just a test and nothing was really done, as codfw is already in read-only mode right now |
[production] |
| 06:57 |
<switchdc> |
(oblivian@sarin) END TASK - switchdc.stages.t02_start_mediawiki_readonly(codfw, eqiad) Failed to execute |
[production] |
| 06:57 |
<root@tin> |
Synchronized wmf-config/db-codfw.php: Set MediaWiki in read-only mode in datacenter codfw (duration: 00m 23s) |
[production] |
| 06:57 |
<switchdc> |
(oblivian@sarin) MediaWiki read-only period starts at: 2017-04-12 06:56:53.822926 |
[production] |
| 06:56 |
<switchdc> |
(oblivian@sarin) START TASK - switchdc.stages.t02_start_mediawiki_readonly(codfw, eqiad) Set MediaWiki in read-only mode (db_from config already merged and git pulled) |
[production] |
| 06:53 |
<switchdc> |
(oblivian@sarin) END TASK - switchdc.stages.t01_stop_maintenance(codfw, eqiad) Failed to execute |
[production] |
| 06:53 |
<switchdc> |
(oblivian@sarin) START TASK - switchdc.stages.t01_stop_maintenance(codfw, eqiad) Stop MediaWiki maintenance in the old master DC |
[production] |
| 06:50 |
<_joe_> |
testing switchover codfw => eqiad, no destructive actions will be taken |
[production] |
| 06:42 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1093 - T17441 (duration: 00m 46s) |
[production] |
| 06:37 |
<elukey> |
reimage mw2246.codfw.wmnet mw2152.codfw.wmnet to remove the /tmp partition (codfw videoscalers, switchover prep) |
[production] |
| 06:32 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1072 - T132416 (duration: 00m 46s) |
[production] |
| 06:28 |
<_joe_> |
killing long-running puppet-agent on db2058 too |
[production] |
| 06:20 |
<_joe_> |
killing badly-started puppet agents on mc1010, tempdb2001,db1090, db2058, hydrogen, possibly others later |
[production] |
| 06:13 |
<marostegui> |
Deploy alter table on db1075 eqiad master (s3, image table) - T160415 |
[production] |
| 06:04 |
<marostegui> |
Deploy schema change on s6 - db1093 - T17441 |
[production] |
| 06:04 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1093 (duration: 02m 00s) |
[production] |
| 05:56 |
<marostegui> |
Deploy alter table on db2108 codfw master (s3, image table) - T160415 |
[production] |
| 04:53 |
<legoktm> |
started `mwscriptwikiset refreshLinks.php small.dblist` on terbium |
[production] |
|
2017-04-11
§
|
| 23:58 |
<thcipriani@tin> |
Synchronized wmf-config/CirrusSearch-production.php: SWAT: [[gerrit:347782|Enable deleted archive indexing & searching]] T109561 PART II (duration: 00m 45s) |
[production] |
| 23:56 |
<thcipriani@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:347782|Enable deleted archive indexing & searching]] T109561 PART I (duration: 00m 45s) |
[production] |
| 23:29 |
<ejegg> |
updated fundraising-tools from 0a42db3ab4e79b5ce5569b698068d8676d575ef1 to a8b8d7242799b61dd2a48ef4e804164cd1818bc9 |
[production] |
| 23:27 |
<thcipriani@tin> |
Synchronized portals: SWAT: [[gerrit:347679|Bumping portals to master]] T128546 (duration: 00m 46s) |
[production] |
| 23:26 |
<thcipriani@tin> |
Synchronized portals/prod/wikipedia.org/assets: SWAT: [[gerrit:347679|Bumping portals to master]] T128546 (duration: 00m 46s) |
[production] |
| 23:23 |
<mutante> |
ocg: clearing host cache for ocg1001 which is shutdown for hardware repair. (on ocg1003: sudo -u ocg -g ocg nodejs-ocg /srv/deployment/ocg/ocg/mw-ocg-service/scripts/clear-host-cache.js -c /etc/ocg/mw-ocg-service.js ocg1001) T161158 |
[production] |
| 23:15 |
<thcipriani@tin> |
Synchronized docroot/noc/conf/pageassessments.dblist: SWAT: [[gerrit:347468|Adding pageassessments.dblist for maintanence script]] T159438 PART II (duration: 00m 45s) |
[production] |
| 23:14 |
<thcipriani@tin> |
Synchronized dblists/pageassessments.dblist: SWAT: [[gerrit:347468|Adding pageassessments.dblist for maintanence script]] T159438 PART I (duration: 00m 45s) |
[production] |
| 23:11 |
<mutante> |
ocg1001 - scheduled downtime in icinga for host and all services, confirmed it's not actively doign things anymore, shutting down for hardware replacement (T161158) |
[production] |
| 23:10 |
<thcipriani@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:347419|Enable Flow beta feature on frwikiversity]] T162022 (duration: 00m 46s) |
[production] |