| 2016-09-14
      
      § | 
    
  | 10:25 | <ema> | varnish-be restarted on cp4005 | [production] | 
            
  | 10:03 | <marostegui> | alter localuser table in db2054 - T141951 | [production] | 
            
  | 09:47 | <marostegui> | Renaming tables before dropping them T54924 | [production] | 
            
  | 08:43 | <marostegui> | alter localuser table in db2047 - T141951 | [production] | 
            
  | 07:35 | <legoktm@tin> | Synchronized php-1.28.0-wmf.19/includes/MediaWiki.php: Use cpPosTime cookie for same-domain redirects on DB change - https://gerrit.wikimedia.org/r/#/c/310494/ (duration: 00m 45s) | [production] | 
            
  | 07:33 | <legoktm@tin> | Synchronized php-1.28.0-wmf.19/includes/db/ChronologyProtector.php: Use cpPosTime cookie for same-domain redirects on DB change - https://gerrit.wikimedia.org/r/#/c/310494/ (duration: 00m 47s) | [production] | 
            
  | 07:32 | <legoktm@tin> | Synchronized php-1.28.0-wmf.19/includes/db/loadbalancer/LBFactory.php: Use cpPosTime cookie for same-domain redirects on DB change - https://gerrit.wikimedia.org/r/#/c/310494/ (duration: 00m 46s) | [production] | 
            
  | 03:22 | <l10nupdate@tin> | ResourceLoader cache refresh completed at Wed Sep 14 03:22:13 UTC 2016 (duration 7m 6s) | [production] | 
            
  | 03:15 | <mwdeploy@tin> | scap sync-l10n completed (1.28.0-wmf.19) (duration: 18m 08s) | [production] | 
            
  | 02:46 | <mutante> | restarted grrrrit-wm | [production] | 
            
  | 02:44 | <mutante> | gerrit back to normal | [production] | 
            
  | 02:42 | <mutante> | gerrit restarting to apply config changes 256663 and 308885 | [production] | 
            
  | 02:40 | <mwdeploy@tin> | scap sync-l10n completed (1.28.0-wmf.18) (duration: 17m 56s) | [production] | 
            
  | 01:56 | <aaron@tin> | Synchronized wmf-config: Set $wgAPIMaxLagThreshold => 3 and "max lag" => 6 (duration: 00m 51s) | [production] | 
            
  
    | 2016-09-13
      
      § | 
    
  | 23:32 | <catrope@tin> | Synchronized php-1.28.0-wmf.18/resources/lib/moment/locale: T145382 (duration: 00m 47s) | [production] | 
            
  | 23:19 | <catrope@tin> | Synchronized php-1.28.0-wmf.19/resources/lib/moment/locale: T145382 (duration: 00m 49s) | [production] | 
            
  | 22:44 | <awight> | update paymentswiki config to da01ae9045af7b8b5f3d34a5f81419fde3bde1d5 | [production] | 
            
  | 22:13 | <demon@tin> | Synchronized php-1.28.0-wmf.19/includes/MediaWiki.php: Avoid stupid warnings on url parsing (duration: 00m 47s) | [production] | 
            
  | 21:45 | <demon@tin> | Synchronized php-1.28.0-wmf.19/includes/DefaultSettings.php: for lego <3 (duration: 00m 47s) | [production] | 
            
  | 20:46 | <demon@tin> | Synchronized php-1.28.0-wmf.19/extensions/Echo: For Roan <3 (duration: 00m 54s) | [production] | 
            
  | 20:23 | <Krenair> | restarted designate-api on labtestservices2001, now designate in labtest is working again | [production] | 
            
  | 20:21 | <Krenair> | restart rabbitmq-server on labtestcontrol2001 | [production] | 
            
  | 19:01 | <demon@tin> | rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.19 | [production] | 
            
  | 18:35 | <gehel> | starting data import on wdqs200? | [production] | 
            
  | 18:24 | <demon@tin> | Finished scap: testwiki to wmf.19 + l10n bootstrap (try 2) (duration: 47m 21s) | [production] | 
            
  | 18:08 | <gehel> | moving to scap deployed configuration for wdqs - T144380 | [production] | 
            
  | 17:58 | <mutante> | wtp2019 - powercycled, back up without the error, services started | [production] | 
            
  | 17:55 | <mutante> | wtp2019 Uncorrectable Memory Error | [production] | 
            
  | 17:55 | <mutante> | wtp2019 - down sinc a couple days. console says "Alert!  System fatal error during previous boot" | [production] | 
            
  | 17:53 | <volans> | reimaging mw2198 that failed early today | [production] | 
            
  | 17:36 | <demon@tin> | Started scap: testwiki to wmf.19 + l10n bootstrap (try 2) | [production] | 
            
  | 17:24 | <demon@tin> | scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="testwiki" --outdir="/tmp/scap_l10n_4282891950" --threads=4 --lang en  --quiet' returned non-zero exit status 1 (duration: 09m 23s) | [production] | 
            
  | 17:15 | <demon@tin> | Started scap: testwiki to wmf.19 + l10n bootstrap | [production] | 
            
  | 16:59 | <jynus> | beta dbs back in rw mode | [production] | 
            
  | 16:54 | <mobrovac> | restbase deploy end of d10d759 | [production] | 
            
  | 16:53 | <demon@tin> | Synchronized multiversion/updateWikiversions: unbreak myself (duration: 00m 48s) | [production] | 
            
  | 16:41 | <mobrovac> | restbase deploy start of d10d759 | [production] | 
            
  | 16:25 | <mobrovac> | change-prop deploying d701a69 | [production] | 
            
  | 16:24 | <godog> | dump hhvm backtrace on mw1162 and restart hhvm, apache gets connection refused | [production] | 
            
  | 16:17 | <urandom> | T144826: Removing compaction rate limit, increasing compactor threads (from 10 to 20), and beginning scrub of local_group_wikipedia_T_parsoid_html.data (restbase2004-b.codfw.wmnet) | [production] | 
            
  | 16:06 | <jynus> | power resetting db1082 | [production] | 
            
  | 16:02 | <jynus@tin> | Synchronized wmf-config/db-eqiad.php: Depool db1082 (duration: 01m 00s) | [production] | 
            
  | 15:46 | <gehel> | test_* indices remvoed on relforge cluster, cluster restarted | [production] | 
            
  | 15:44 | <jynus> | setting deployment-db1 and deployment-db1 mysqls in read only mode | [production] | 
            
  | 15:40 | <gehel> | shutting down relforge cluster for indices cleanup | [production] | 
            
  | 02:46 | <l10nupdate@tin> | ResourceLoader cache refresh completed at Tue Sep 13 02:46:54 UTC 2016 (duration 7m 12s) | [production] | 
            
  | 02:39 | <mwdeploy@tin> | scap sync-l10n completed (1.28.0-wmf.18) (duration: 17m 41s) | [production] | 
            
  | 01:58 | <aaron@tin> | Synchronized wmf-config/CommonSettings.php: Lower $wgMaxUserDBWriteDuration to 3 (duration: 00m 47s) | [production] | 
            
  | 01:27 | <aaron@tin> | Synchronized php-1.28.0-wmf.18/extensions/GlobalUsage: 1843a856c7c648a310a1983d53139e5b79ea6585 (duration: 00m 48s) | [production] | 
            
  | 01:24 | <bblack> | cache_upload: reverting codfw to file storage | [production] |