|
2017-04-12
§
|
| 07:59 |
<switchdc> |
(oblivian@sarin) START TASK - switchdc.stages.t09_restore_ttl(codfw, eqiad) Restore the TTL of all the MediaWiki discovery records |
[production] |
| 07:58 |
<switchdc> |
(oblivian@sarin) END TASK - switchdc.stages.t05_switch_traffic(codfw, eqiad) Successfully completed |
[production] |
| 07:55 |
<switchdc> |
(oblivian@sarin) START TASK - switchdc.stages.t05_switch_traffic(codfw, eqiad) Switch traffic flow to the appservers in the new datacenter |
[production] |
| 07:55 |
<_joe_> |
resuming non-dry run tests of switchdc, all logs from switchdc by me are just tests |
[production] |
| 06:57 |
<_joe_> |
the last messages are just a test and nothing was really done, as codfw is already in read-only mode right now |
[production] |
| 06:57 |
<switchdc> |
(oblivian@sarin) END TASK - switchdc.stages.t02_start_mediawiki_readonly(codfw, eqiad) Failed to execute |
[production] |
| 06:57 |
<root@tin> |
Synchronized wmf-config/db-codfw.php: Set MediaWiki in read-only mode in datacenter codfw (duration: 00m 23s) |
[production] |
| 06:57 |
<switchdc> |
(oblivian@sarin) MediaWiki read-only period starts at: 2017-04-12 06:56:53.822926 |
[production] |
| 06:56 |
<switchdc> |
(oblivian@sarin) START TASK - switchdc.stages.t02_start_mediawiki_readonly(codfw, eqiad) Set MediaWiki in read-only mode (db_from config already merged and git pulled) |
[production] |
| 06:53 |
<switchdc> |
(oblivian@sarin) END TASK - switchdc.stages.t01_stop_maintenance(codfw, eqiad) Failed to execute |
[production] |
| 06:53 |
<switchdc> |
(oblivian@sarin) START TASK - switchdc.stages.t01_stop_maintenance(codfw, eqiad) Stop MediaWiki maintenance in the old master DC |
[production] |
| 06:50 |
<_joe_> |
testing switchover codfw => eqiad, no destructive actions will be taken |
[production] |
| 06:42 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1093 - T17441 (duration: 00m 46s) |
[production] |
| 06:37 |
<elukey> |
reimage mw2246.codfw.wmnet mw2152.codfw.wmnet to remove the /tmp partition (codfw videoscalers, switchover prep) |
[production] |
| 06:32 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1072 - T132416 (duration: 00m 46s) |
[production] |
| 06:28 |
<_joe_> |
killing long-running puppet-agent on db2058 too |
[production] |
| 06:20 |
<_joe_> |
killing badly-started puppet agents on mc1010, tempdb2001,db1090, db2058, hydrogen, possibly others later |
[production] |
| 06:13 |
<marostegui> |
Deploy alter table on db1075 eqiad master (s3, image table) - T160415 |
[production] |
| 06:04 |
<marostegui> |
Deploy schema change on s6 - db1093 - T17441 |
[production] |
| 06:04 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1093 (duration: 02m 00s) |
[production] |
| 05:56 |
<marostegui> |
Deploy alter table on db2108 codfw master (s3, image table) - T160415 |
[production] |
| 04:53 |
<legoktm> |
started `mwscriptwikiset refreshLinks.php small.dblist` on terbium |
[production] |
|
2017-04-11
§
|
| 23:58 |
<thcipriani@tin> |
Synchronized wmf-config/CirrusSearch-production.php: SWAT: [[gerrit:347782|Enable deleted archive indexing & searching]] T109561 PART II (duration: 00m 45s) |
[production] |
| 23:56 |
<thcipriani@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:347782|Enable deleted archive indexing & searching]] T109561 PART I (duration: 00m 45s) |
[production] |
| 23:29 |
<ejegg> |
updated fundraising-tools from 0a42db3ab4e79b5ce5569b698068d8676d575ef1 to a8b8d7242799b61dd2a48ef4e804164cd1818bc9 |
[production] |
| 23:27 |
<thcipriani@tin> |
Synchronized portals: SWAT: [[gerrit:347679|Bumping portals to master]] T128546 (duration: 00m 46s) |
[production] |
| 23:26 |
<thcipriani@tin> |
Synchronized portals/prod/wikipedia.org/assets: SWAT: [[gerrit:347679|Bumping portals to master]] T128546 (duration: 00m 46s) |
[production] |
| 23:23 |
<mutante> |
ocg: clearing host cache for ocg1001 which is shutdown for hardware repair. (on ocg1003: sudo -u ocg -g ocg nodejs-ocg /srv/deployment/ocg/ocg/mw-ocg-service/scripts/clear-host-cache.js -c /etc/ocg/mw-ocg-service.js ocg1001) T161158 |
[production] |
| 23:15 |
<thcipriani@tin> |
Synchronized docroot/noc/conf/pageassessments.dblist: SWAT: [[gerrit:347468|Adding pageassessments.dblist for maintanence script]] T159438 PART II (duration: 00m 45s) |
[production] |
| 23:14 |
<thcipriani@tin> |
Synchronized dblists/pageassessments.dblist: SWAT: [[gerrit:347468|Adding pageassessments.dblist for maintanence script]] T159438 PART I (duration: 00m 45s) |
[production] |
| 23:11 |
<mutante> |
ocg1001 - scheduled downtime in icinga for host and all services, confirmed it's not actively doign things anymore, shutting down for hardware replacement (T161158) |
[production] |
| 23:10 |
<thcipriani@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:347419|Enable Flow beta feature on frwikiversity]] T162022 (duration: 00m 46s) |
[production] |
| 23:04 |
<mutante> |
ocg1001 - apt-get clean for disk space |
[production] |
| 22:36 |
<mutante> |
ocg1003 started picking up jobs (mw-ocg-latexer) after it was enabled with gerrit:347781, ocg1001 was disabled in the same change. Also ganglia graphs confirm it. T84723 T161158 |
[production] |
| 22:22 |
<addshore@tin> |
Synchronized wmf-config/InitialiseSettings.php: [[gerrit:347583|Enable alternate RevSlider slider on group0 T160410]] (duration: 00m 45s) |
[production] |
| 22:19 |
<dzahn@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=ocg1001.eqiad.wmnet |
[production] |
| 22:17 |
<addshore@tin> |
Synchronized wmf-config/InitialiseSettings.php: [[gerrit:347582|Enable TwoColConflict BetaFeature on fiwiki]] (duration: 00m 46s) |
[production] |
| 21:23 |
<mobrovac@tin> |
Finished deploy [restbase/deploy@a4042a6]: Update the legal text in the API docs (duration: 06m 49s) |
[production] |
| 21:17 |
<mobrovac@tin> |
Started deploy [restbase/deploy@a4042a6]: Update the legal text in the API docs |
[production] |
| 21:16 |
<mobrovac@tin> |
Finished deploy [restbase/deploy@a4042a6]: Staging: Update the legal text in the API docs (duration: 03m 55s) |
[production] |
| 21:12 |
<mobrovac@tin> |
Started deploy [restbase/deploy@a4042a6]: Staging: Update the legal text in the API docs |
[production] |
| 21:12 |
<mobrovac@tin> |
Finished deploy [restbase/deploy@a4042a6]: Dev cluster: Update the legal text in the API docs (duration: 01m 37s) |
[production] |
| 21:11 |
<mobrovac@tin> |
Started deploy [restbase/deploy@a4042a6]: Dev cluster: Update the legal text in the API docs |
[production] |
| 20:51 |
<_joe_> |
killed running 'puppet agent t-v' on ruthenium |
[production] |
| 19:20 |
<ppchelko@tin> |
Finished deploy [electron-render/deploy@5492cdb]: Update to latest upstream, full deploy, attempt#2 T160764 (duration: 01m 25s) |
[production] |
| 19:18 |
<ppchelko@tin> |
Started deploy [electron-render/deploy@5492cdb]: Update to latest upstream, full deploy, attempt#2 T160764 |
[production] |
| 19:11 |
<ppchelko@tin> |
Finished deploy [electron-render/deploy@5492cdb]: Update to latest upstream, full deploy, T160764 (duration: 03m 38s) |
[production] |
| 19:08 |
<ppchelko@tin> |
Started deploy [electron-render/deploy@5492cdb]: Update to latest upstream, full deploy, T160764 |
[production] |
| 19:08 |
<demon@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.20 |
[production] |
| 19:01 |
<ppchelko@tin> |
Finished deploy [electron-render/deploy@5492cdb]: Update to latest upstream, canary on scb2001, attempt#3 T160764 (duration: 00m 52s) |
[production] |