| 2014-02-20
      
      § | 
    
  | 19:25 | <reedy> | Started scap: testwiki to 1.23wmf15 and build l10n cache (take 4) | [production] | 
            
  | 19:24 | <reedy> | scap aborted: testwiki to 1.23wmf15 and build l10n cache (take 3) (duration: 03m 29s) | [production] | 
            
  | 19:21 | <reedy> | Started scap: testwiki to 1.23wmf15 and build l10n cache (take 3) | [production] | 
            
  | 19:14 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: Stuff back to their deployed versions | [production] | 
            
  | 19:10 | <reedy> | Started scap: testwiki to 1.23wmf15 and build l10n cache (take 2) | [production] | 
            
  | 19:10 | <reedy> | scap aborted: testwiki to 1.23wmf15 and build l10n cache (duration: 131m 36s) | [production] | 
            
  | 19:08 | <Reedy> | For those still following along, scap is still running | [production] | 
            
  | 18:42 | <cmjohnson1> | flipping the switch to off in tampa | [production] | 
            
  | 18:39 | <awight> | crm updated from 07c1943088355bd262786d7134763549c8070ceb to eb3536eb32cbc7400e4e5884d56fbf104e38fc2b | [production] | 
            
  | 17:42 | <bd808> | mw1151 & mw1152 throwing PHP Fatal error that looks to be APC or other cache corruption | [production] | 
            
  | 16:58 | <reedy> | Started scap: testwiki to 1.23wmf15 and build l10n cache | [production] | 
            
  | 16:56 | <reedy> | updated /a/common to {{Gerrit|I19a8c3298}}: Remove UserThrottle extension and log group | [production] | 
            
  | 03:26 | <LocalisationUpdate> | ResourceLoader cache refresh completed at 2014-02-20 03:26:31+00:00 | [production] | 
            
  | 02:41 | <LocalisationUpdate> | completed (1.23wmf13) at 2014-02-20 02:41:15+00:00 | [production] | 
            
  | 02:22 | <LocalisationUpdate> | completed (1.23wmf14) at 2014-02-20 02:22:04+00:00 | [production] | 
            
  | 02:18 | <awight> | update crm from 20684a7e5dd4270e615ca52e6a17b25e880e47c4 to 07c1943088355bd262786d7134763549c8070ceb | [production] | 
            
  | 01:05 | <bd808> | mw-update-l10n test on tin completed | [production] | 
            
  | 01:01 | <bd808> | Testing changes to mw-update-l10n on tin | [production] | 
            
  | 00:51 | <bd808> | mw-update-l10n test on tin completed successfully | [production] | 
            
  | 00:47 | <bd808> | Testing changes to mw-update-l10n on tin | [production] | 
            
  | 00:16 | <demon> | synchronized wmf-config/InitialiseSettings.php  'Wikiversities and itwikiquote get Cirrus as beta' | [production] | 
            
  | 00:10 | <mholmquist> | synchronized wmf-config/CommonSettings.php  'Set MMV network performance sampling level to 0.1% on production wikis' | [production] | 
            
  | 00:09 | <mholmquist> | synchronized wmf-config/CommonSettings-labs.php  'Set MMV network performance sampling level to 100% on labs' | [production] | 
            
  | 00:07 | <mholmquist> | updated /a/common to {{Gerrit|I1319008cb}}: Start sampling detailed network performance for Multimedia Viewer | [production] | 
            
  
    | 2014-02-19
      
      § | 
    
  | 23:47 | <paravoid> | restarting gmetad on nickel | [production] | 
            
  | 23:10 | <mwalker> | updateding Fundraising SmashPig from 8a5e194a1c4827fc4fc75a2d6b298d3c30112d90 to 2fdf982b20f1cbeaf9f57af64ef21b5b69a36f6e for XFF logging | [production] | 
            
  | 21:41 | <bd808> | Finished scap: no-diff scap to test script changes (duration: 04m 52s) | [production] | 
            
  | 21:37 | <bd808> | during scap: snapshot3: ImportError: No module named argparse | [production] | 
            
  | 21:37 | <bd808> | Started scap: no-diff scap to test script changes | [production] | 
            
  | 21:32 | <aaron> | synchronized php-1.23wmf14/includes/filebackend/SwiftFileBackend.php  '10ba3d7caa1e4bb4d521384bebbf42976cea4a22' | [production] | 
            
  | 21:31 | <bd808> | Finished scap: no-diff scap to test script changes (duration: 52m 06s) | [production] | 
            
  | 21:10 | <bd808> | During scap: snapshot2: rsync error: timeout in data send/receive (code 30) at io.c(137) [sender=3.0.9] | [production] | 
            
  | 21:07 | <gwicke> | deployed Parsoid deploy c73ea9d3 and code 76e9b66 | [production] | 
            
  | 21:03 | <LeslieCarr> | fixing firewall bastion to payments deny clause to actually deny | [production] | 
            
  | 20:59 | <ottomata> | initiating controlled shutdown of analytics1022 kafka broker in order to reload configs for replica.lag.max.messages | [production] | 
            
  | 20:39 | <bd808> | Started scap: no-diff scap to test script changes | [production] | 
            
  | 20:19 | <ottomata> | controlled shutdown of analytics1021 Kafka broker in order to reload configs for replica.lag.max.messages | [production] | 
            
  | 17:15 | <^d> | creating elasticsearch indexes for all wikiversities, may see some intermittent icinga spam as shards rebalance | [production] | 
            
  | 17:09 | <demon> | synchronized wmf-config/InitialiseSettings.php  'wikiversities and itwikiquote getting cirrus' | [production] | 
            
  | 17:08 | <demon> | synchronized php-1.23wmf14/extensions/Elastica  'Elastica to master' | [production] | 
            
  | 17:07 | <demon> | synchronized php-1.23wmf14/extensions/CirrusSearch  'Cirrus to master' | [production] | 
            
  | 17:07 | <demon> | synchronized php-1.23wmf13/extensions/Elastica  'Elastica to master' | [production] | 
            
  | 17:06 | <demon> | synchronized php-1.23wmf13/extensions/CirrusSearch  'Cirrus to master' | [production] | 
            
  | 06:53 | <springle> | s5 slaves online reindexing wikidatawiki wb_terms | [production] | 
            
  | 06:01 | <springle> | flowdb schema changes gerrit 111671 | [production] | 
            
  | 04:58 | <demon> | synchronized wmf-config/CirrusSearch-common.php  'Turn on checkDelay for cirrus links update secondary jobs' | [production] | 
            
  | 03:23 | <andrewbogott> | testing the log by logging a test | [production] | 
            
  | 02:55 | <LocalisationUpdate> | ResourceLoader cache refresh completed at 2014-02-19 02:55:36+00:00 | [production] | 
            
  | 02:27 | <LocalisationUpdate> | completed (1.23wmf13) at 2014-02-19 02:27:49+00:00 | [production] | 
            
  | 02:03 | <LocalisationUpdate> | completed (1.23wmf14) at 2014-02-19 02:03:41+00:00 | [production] |