1001-1050 of 10000 results (36ms)
2017-04-12 §
08:14 <root@tin> Synchronized wmf-config/db-eqiad.php: Set MediaWiki in read-write mode in datacenter eqiad (duration: 00m 35s) [production]
08:13 <switchdc> (oblivian@sarin) START TASK - switchdc.stages.t08_stop_mediawiki_readonly(codfw, eqiad) Set MediaWiki in read-write mode (db_to config already merged and git pulled) [production]
08:09 <switchdc> (oblivian@sarin) END TASK - switchdc.stages.t06_redis(codfw, eqiad) Successfully completed [production]
08:09 <switchdc> (oblivian@sarin) START TASK - switchdc.stages.t06_redis(codfw, eqiad) Switch the Redis replication [production]
08:02 <switchdc> (oblivian@sarin) END TASK - switchdc.stages.t05_switch_datacenter(codfw, eqiad) Successfully completed [production]
08:02 <switchdc> (oblivian@sarin) START TASK - switchdc.stages.t05_switch_datacenter(codfw, eqiad) Switch MediaWiki configuration to the new datacenter [production]
08:00 <switchdc> (oblivian@sarin) END TASK - switchdc.stages.t09_restore_ttl(codfw, eqiad) Successfully completed [production]
07:59 <switchdc> (oblivian@sarin) START TASK - switchdc.stages.t09_restore_ttl(codfw, eqiad) Restore the TTL of all the MediaWiki discovery records [production]
07:58 <switchdc> (oblivian@sarin) END TASK - switchdc.stages.t05_switch_traffic(codfw, eqiad) Successfully completed [production]
07:55 <switchdc> (oblivian@sarin) START TASK - switchdc.stages.t05_switch_traffic(codfw, eqiad) Switch traffic flow to the appservers in the new datacenter [production]
07:55 <_joe_> resuming non-dry run tests of switchdc, all logs from switchdc by me are just tests [production]
06:57 <_joe_> the last messages are just a test and nothing was really done, as codfw is already in read-only mode right now [production]
06:57 <switchdc> (oblivian@sarin) END TASK - switchdc.stages.t02_start_mediawiki_readonly(codfw, eqiad) Failed to execute [production]
06:57 <root@tin> Synchronized wmf-config/db-codfw.php: Set MediaWiki in read-only mode in datacenter codfw (duration: 00m 23s) [production]
06:57 <switchdc> (oblivian@sarin) MediaWiki read-only period starts at: 2017-04-12 06:56:53.822926 [production]
06:56 <switchdc> (oblivian@sarin) START TASK - switchdc.stages.t02_start_mediawiki_readonly(codfw, eqiad) Set MediaWiki in read-only mode (db_from config already merged and git pulled) [production]
06:53 <switchdc> (oblivian@sarin) END TASK - switchdc.stages.t01_stop_maintenance(codfw, eqiad) Failed to execute [production]
06:53 <switchdc> (oblivian@sarin) START TASK - switchdc.stages.t01_stop_maintenance(codfw, eqiad) Stop MediaWiki maintenance in the old master DC [production]
06:50 <_joe_> testing switchover codfw => eqiad, no destructive actions will be taken [production]
06:42 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1093 - T17441 (duration: 00m 46s) [production]
06:37 <elukey> reimage mw2246.codfw.wmnet mw2152.codfw.wmnet to remove the /tmp partition (codfw videoscalers, switchover prep) [production]
06:32 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1072 - T132416 (duration: 00m 46s) [production]
06:28 <_joe_> killing long-running puppet-agent on db2058 too [production]
06:20 <_joe_> killing badly-started puppet agents on mc1010, tempdb2001,db1090, db2058, hydrogen, possibly others later [production]
06:13 <marostegui> Deploy alter table on db1075 eqiad master (s3, image table) - T160415 [production]
06:04 <marostegui> Deploy schema change on s6 - db1093 - T17441 [production]
06:04 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1093 (duration: 02m 00s) [production]
05:56 <marostegui> Deploy alter table on db2108 codfw master (s3, image table) - T160415 [production]
04:53 <legoktm> started `mwscriptwikiset refreshLinks.php small.dblist` on terbium [production]
2017-04-11 §
23:58 <thcipriani@tin> Synchronized wmf-config/CirrusSearch-production.php: SWAT: [[gerrit:347782|Enable deleted archive indexing & searching]] T109561 PART II (duration: 00m 45s) [production]
23:56 <thcipriani@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:347782|Enable deleted archive indexing & searching]] T109561 PART I (duration: 00m 45s) [production]
23:29 <ejegg> updated fundraising-tools from 0a42db3ab4e79b5ce5569b698068d8676d575ef1 to a8b8d7242799b61dd2a48ef4e804164cd1818bc9 [production]
23:27 <thcipriani@tin> Synchronized portals: SWAT: [[gerrit:347679|Bumping portals to master]] T128546 (duration: 00m 46s) [production]
23:26 <thcipriani@tin> Synchronized portals/prod/wikipedia.org/assets: SWAT: [[gerrit:347679|Bumping portals to master]] T128546 (duration: 00m 46s) [production]
23:23 <mutante> ocg: clearing host cache for ocg1001 which is shutdown for hardware repair. (on ocg1003: sudo -u ocg -g ocg nodejs-ocg /srv/deployment/ocg/ocg/mw-ocg-service/scripts/clear-host-cache.js -c /etc/ocg/mw-ocg-service.js ocg1001) T161158 [production]
23:15 <thcipriani@tin> Synchronized docroot/noc/conf/pageassessments.dblist: SWAT: [[gerrit:347468|Adding pageassessments.dblist for maintanence script]] T159438 PART II (duration: 00m 45s) [production]
23:14 <thcipriani@tin> Synchronized dblists/pageassessments.dblist: SWAT: [[gerrit:347468|Adding pageassessments.dblist for maintanence script]] T159438 PART I (duration: 00m 45s) [production]
23:11 <mutante> ocg1001 - scheduled downtime in icinga for host and all services, confirmed it's not actively doign things anymore, shutting down for hardware replacement (T161158) [production]
23:10 <thcipriani@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:347419|Enable Flow beta feature on frwikiversity]] T162022 (duration: 00m 46s) [production]
23:04 <mutante> ocg1001 - apt-get clean for disk space [production]
22:36 <mutante> ocg1003 started picking up jobs (mw-ocg-latexer) after it was enabled with gerrit:347781, ocg1001 was disabled in the same change. Also ganglia graphs confirm it. T84723 T161158 [production]
22:22 <addshore@tin> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:347583|Enable alternate RevSlider slider on group0 T160410]] (duration: 00m 45s) [production]
22:19 <dzahn@puppetmaster1001> conftool action : set/pooled=no; selector: name=ocg1001.eqiad.wmnet [production]
22:17 <addshore@tin> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:347582|Enable TwoColConflict BetaFeature on fiwiki]] (duration: 00m 46s) [production]
21:23 <mobrovac@tin> Finished deploy [restbase/deploy@a4042a6]: Update the legal text in the API docs (duration: 06m 49s) [production]
21:17 <mobrovac@tin> Started deploy [restbase/deploy@a4042a6]: Update the legal text in the API docs [production]
21:16 <mobrovac@tin> Finished deploy [restbase/deploy@a4042a6]: Staging: Update the legal text in the API docs (duration: 03m 55s) [production]
21:12 <mobrovac@tin> Started deploy [restbase/deploy@a4042a6]: Staging: Update the legal text in the API docs [production]
21:12 <mobrovac@tin> Finished deploy [restbase/deploy@a4042a6]: Dev cluster: Update the legal text in the API docs (duration: 01m 37s) [production]
21:11 <mobrovac@tin> Started deploy [restbase/deploy@a4042a6]: Dev cluster: Update the legal text in the API docs [production]