| 2016-01-20
      
      § | 
    
  | 13:57 | <reedy@tin> | reedy@tin Synchronized php-1.27.0-wmf.10/extensions/SemanticResultFormats/: noop for wikitech deploy (duration: 00m 33s) | [production] | 
            
  | 13:55 | <reedy@tin> | reedy@tin Synchronized php-1.27.0-wmf.10/extensions/SemanticMediaWiki/: noop for wikitech deploy (duration: 00m 33s) | [production] | 
            
  | 13:54 | <reedy@tin> | reedy@tin Synchronized php-1.27.0-wmf.10/extensions/Validator/: noop for wikitech deploy (duration: 00m 33s) | [production] | 
            
  | 13:52 | <jynus@tin> | jynus@tin Synchronized wmf-config/db-eqiad.php: Setting db1049 as API node as db1045 is lagged (duration: 00m 33s) | [production] | 
            
  | 13:41 | <ema> | Revert migration of mobile traffic to text cluster in codfw https://phabricator.wikimedia.org/T109286 | [production] | 
            
  | 12:55 | <akosiaris> | restart hhvm on mw1130 | [production] | 
            
  | 12:43 | <jynus> | performing alter table on db1026 (ETA: 5 hours) | [production] | 
            
  | 12:20 | <jynus@tin> | jynus@tin Synchronized wmf-config/db-eqiad.php: Setting s5 master as recentchanges role (duration: 00m 32s) | [production] | 
            
  | 12:04 | <jynus> | trying schema change on wikidata (wb_terms) | [production] | 
            
  | 09:36 | <akosiaris> | gnt-instance modify -H disk_aio=native cygnus.codfw.wmnet | [production] | 
            
  | 09:18 | <akosiaris> | offline fr_archive volume on nas1001-a | [production] | 
            
  | 09:15 | <akosiaris> | unexport /vol/fr_archive on nas1001-a | [production] | 
            
  | 07:56 | <_joe_> | powercycling mw1162, unable to login from console, memory exhaustion | [production] | 
            
  | 07:24 | <ebernhardson@tin> | ebernhardson@tin Synchronized php-1.27.0-wmf.10/extensions/CirrusSearch/includes/DataSender.php: stop checking for frozen indices while codfw elasticsearch recovers (duration: 01m 42s) | [production] | 
            
  | 06:24 | <ebernhardson> | codfw elasticsearch cluster stopped responding during load test, idling test to see if it recovers | [production] | 
            
  | 03:44 | <l10nupdate@tin> | l10nupdate@tin ResourceLoader cache refresh completed at Wed Jan 20 03:44:48 UTC 2016 (duration 7m 29s) | [production] | 
            
  | 03:37 | <mwdeploy@tin> | mwdeploy@tin sync-l10n completed (1.27.0-wmf.11) (duration: 16m 21s) | [production] | 
            
  | 03:02 | <mwdeploy@tin> | mwdeploy@tin sync-l10n completed (1.27.0-wmf.10) (duration: 10m 06s) | [production] | 
            
  | 02:35 | <mwdeploy@tin> | mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 11m 20s) | [production] | 
            
  | 01:27 | <aaron@tin> | aaron@tin Synchronized wmf-config: Configure $wgCdnReboundPurgeDelay (duration: 00m 32s) | [production] | 
            
  | 01:01 | <mobrovac> | restbase deploy end of d621b76 | [production] | 
            
  | 00:57 | <krenair@tin> | krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/264917/ (duration: 00m 32s) | [production] | 
            
  | 00:56 | <legoktm> | delete from localuser where lu_name ="Αντώνης Μανιός" and lu_wiki ="mediawikiwiki" limit 1 on centralauth db for T119736 | [production] | 
            
  | 00:53 | <krenair@tin> | krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/264920/ (duration: 00m 33s) | [production] | 
            
  | 00:49 | <krenair@tin> | krenair@tin Synchronized php-1.27.0-wmf.10/extensions/MobileFrontend/includes/api/ApiMobileView.php: https://gerrit.wikimedia.org/r/#/c/264973/ (duration: 00m 32s) | [production] | 
            
  | 00:49 | <mobrovac> | restbase deploy start of d621b76 | [production] | 
            
  | 00:38 | <krenair@tin> | krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/264961/ (duration: 00m 31s) | [production] | 
            
  | 00:37 | <krenair@tin> | krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/264961/ (duration: 00m 33s) | [production] | 
            
  | 00:22 | <krenair@tin> | krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/264260/ (duration: 00m 32s) | [production] | 
            
  | 00:21 | <krenair@tin> | krenair@tin Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/264260/ (duration: 00m 32s) | [production] | 
            
  | 00:17 | <krenair@tin> | krenair@tin Synchronized php-1.27.0-wmf.10/extensions/CirrusSearch: https://gerrit.wikimedia.org/r/#/c/265146/ (duration: 00m 33s) | [production] | 
            
  | 00:10 | <krenair@tin> | krenair@tin Synchronized php-1.27.0-wmf.10/extensions/CirrusSearch/includes/ElasticsearchIntermediary.php: https://gerrit.wikimedia.org/r/#/c/264989/ (duration: 00m 32s) | [production] | 
            
  
    | 2016-01-19
      
      § | 
    
  | 23:33 | <aaron@tin> | aaron@tin Synchronized wmf-config/CommonSettings.php: Bump $wgJobBackoffThrottling to lower the htmlcacheupdate backlog (duration: 00m 32s) | [production] | 
            
  | 23:22 | <krenair@tin> | krenair@tin Synchronized wmf-config/wikitech.php:  https://gerrit.wikimedia.org/r/265145 (duration: 02m 24s) | [production] | 
            
  | 23:18 | <dduvall@tin> | dduvall@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.27.0-wmf.11 | [production] | 
            
  | 23:13 | <dduvall@tin> | dduvall@tin Finished scap: testwiki to php-1.27.0-wmf.11 and rebuild l10n cache (duration: 72m 03s) | [production] | 
            
  | 22:01 | <dduvall@tin> | dduvall@tin Started scap: testwiki to php-1.27.0-wmf.11 and rebuild l10n cache | [production] | 
            
  | 21:35 | <krenair@tin> | krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/265135 (duration: 00m 32s) | [production] | 
            
  | 21:33 | <krenair@tin> | krenair@tin Synchronized dblists/nonglobal.dblist: https://gerrit.wikimedia.org/r/265135 (duration: 03m 21s) | [production] | 
            
  | 21:33 | <ema> | Finished migrating mobile traffic to text cluster in codfw (Mexico + green US states on this map https://phabricator.wikimedia.org/T114659) | [production] | 
            
  | 21:14 | <dduvall@tin> | dduvall@tin scap failed: CalledProcessError Command '/usr/local/bin/mwscript mergeMessageFileList.php --wiki="testwiki" --list-file="/srv/mediawiki-staging/wmf-config/extension-list" --output="/tmp/tmp.qyk48j8kem" ' returned non-zero exit status 1 (duration: 16m 11s) | [production] | 
            
  | 20:59 | <Krenair> | sync-common on labtestweb2001 | [production] | 
            
  | 20:58 | <dduvall@tin> | dduvall@tin Started scap: testwiki to php-1.27.0-wmf.11 and rebuild l10n cache | [production] | 
            
  | 20:48 | <mutante> | tin: deleted unused things from /srv/deployment (T120157) | [production] | 
            
  | 20:46 | <catrope@tin> | catrope@tin Synchronized wmf-config/InitialiseSettings.php: Disable global AbuseFilters on non-global wikis (duration: 02m 04s) | [production] | 
            
  | 20:25 | <dduvall@tin> | dduvall@tin scap failed: CalledProcessError Command '/usr/local/bin/mwscript mergeMessageFileList.php --wiki="labtestwiki" --list-file="/srv/mediawiki-staging/wmf-config/extension-list" --output="/tmp/tmp.jRNpeW67FO" ' returned non-zero exit status 1 (duration: 01m 31s) | [production] | 
            
  | 20:23 | <dduvall@tin> | dduvall@tin Started scap: testwiki to php-1.27.0-wmf.11 and rebuild l10n cache | [production] | 
            
  | 20:13 | <mutante> | ruthenium: disable puppet, copy data over to osmium (screen) | [production] | 
            
  | 20:12 | <mutante> | ruthenium: service mysql stop | [production] | 
            
  | 19:15 | <catrope@tin> | catrope@tin Synchronized wmf-config/CommonSettings.php: EventBus plumbing (duration: 00m 30s) | [production] |