| 
      
        2013-12-18
      
      §
     | 
  
    
  | 15:29 | 
  <paravoid> | 
  powercycling cp1065, kmem_alloc deadlock | 
  [production] | 
            
  | 15:01 | 
  <hashar> | 
  Gerrit back :-) | 
  [production] | 
            
  | 15:00 | 
  <hashar> | 
  Gerrit dead :( | 
  [production] | 
            
  | 13:51 | 
  <paravoid> | 
  setting up new transit peering in eqiad | 
  [production] | 
            
  | 10:00 | 
  <ori-l> | 
  disabled puppet on tungsten to test mwprof changes; re-enabling now | 
  [production] | 
            
  | 07:37 | 
  <springle> | 
  online schema changes on masters with sql_log_bin=0, indexes only, gerrit 85508 | 
  [production] | 
            
  | 05:32 | 
  <ori-l> | 
  DNS update: scholarships.wikimedia.org CNAME misc-web-lb.eqiad | 
  [production] | 
            
  | 03:09 | 
  <LocalisationUpdate> | 
  ResourceLoader cache refresh completed at Wed Dec 18 03:09:14 UTC 2013 | 
  [production] | 
            
  | 02:47 | 
  <LocalisationUpdate> | 
  completed (1.23wmf6) at Wed Dec 18 02:47:41 UTC 2013 | 
  [production] | 
            
  | 02:25 | 
  <LocalisationUpdate> | 
  completed (1.23wmf7) at Wed Dec 18 02:25:17 UTC 2013 | 
  [production] | 
            
  | 01:52 | 
  <reedy> | 
  synchronized wmf-config/  'touch' | 
  [production] | 
            
  | 01:46 | 
  <reedy> | 
  synchronized wmf-config/  'Ensuring consistency...' | 
  [production] | 
            
  | 00:20 | 
  <mflaschen> | 
  synchronized wmf-config  'Deploy Draft namespace to English Wikipedia.  Remove old Labs overrides that were used for testing this.' | 
  [production] | 
            
  | 00:12 | 
  <springle> | 
  ongoing schema changes on slaves, indexing only, logging gerrit 85508 | 
  [production] | 
            
  | 00:05 | 
  <Krinkle> | 
  Reloading zuul to deploy I6df1f4acccd7a9 | 
  [production] | 
            
  
    | 
      
        2013-12-17
      
      §
     | 
  
    
  | 23:30 | 
  <LeslieCarr> | 
  starting atftpd on carbon | 
  [production] | 
            
  | 22:37 | 
  <ori> | 
  synchronized php-1.23wmf6/extensions/NavigationTiming  'I933a1e3a2: Add 'mediaWikiLoadComplete' measurement' | 
  [production] | 
            
  | 22:30 | 
  <ori> | 
  synchronized php-1.23wmf7/extensions/NavigationTiming  'I933a1e3a2: Add 'mediaWikiLoadComplete' measurement' | 
  [production] | 
            
  | 22:16 | 
  <ori> | 
  synchronized php-1.23wmf6/resources/startup.js  'I3d5bcf10e: startup.js: log current time as global 'mediaWikiLoadStart'' | 
  [production] | 
            
  | 22:15 | 
  <ori-l> | 
  restarted apache on zirconium for scholarships vhost | 
  [production] | 
            
  | 22:03 | 
  <ori> | 
  synchronized php-1.23wmf7/resources/startup.js  'I3d5bcf10e: startup.js: log current time as global 'mediaWikiLoadStart'' | 
  [production] | 
            
  | 21:48 | 
  <csteipp> | 
  synchronized php-1.23wmf7/extensions/TimedMediaHandler  'bug56699' | 
  [production] | 
            
  | 21:47 | 
  <csteipp> | 
  synchronized php-1.23wmf6/extensions/TimedMediaHandler  'bug56699' | 
  [production] | 
            
  | 21:37 | 
  <csteipp> | 
  synchronized php-1.23wmf7/extensions/CentralAuth  'bug57081' | 
  [production] | 
            
  | 21:36 | 
  <csteipp> | 
  synchronized php-1.23wmf6/extensions/CentralAuth  'bug57081' | 
  [production] | 
            
  | 21:31 | 
  <csteipp> | 
  synchronized php-1.23wmf6/includes  'bug58088' | 
  [production] | 
            
  | 21:30 | 
  <csteipp> | 
  synchronized php-1.23wmf7/includes  'bug58088' | 
  [production] | 
            
  | 21:23 | 
  <reedy> | 
  synchronized wmf-config/  'Enable GWToolset on commonswiki' | 
  [production] | 
            
  | 21:16 | 
  <reedy> | 
  updated /a/common to {{Gerrit|I2d1c666e1}}: Production configuration for GWToolset | 
  [production] | 
            
  | 21:14 | 
  <reedy> | 
  finished scap: Rebuild l10n cache for GWToolset, remove AFT, no wikis moving onto GWToolset at this point (commented out) | 
  [production] | 
            
  | 20:45 | 
  <manybubbles> | 
  finished the rolling restart of the elasticsearch cluster.  it could have been done more quickly but there was no hurry. | 
  [production] | 
            
  | 20:45 | 
  <reedy> | 
  started scap: Rebuild l10n cache for GWToolset, remove AFT, no wikis moving onto GWToolset at this point (commented out) | 
  [production] | 
            
  | 20:05 | 
  <Ryan_Lane> | 
  deploying change 102285 to OpenStackManager on virt0 | 
  [production] | 
            
  | 19:58 | 
  <reedy> | 
  synchronized php-1.23wmf7/extensions/GWToolset  'Staging' | 
  [production] | 
            
  | 19:54 | 
  <reedy> | 
  synchronized php-1.23wmf7/includes/filerepo/LocalRepo.php  'Fix fatal I12513b40453573124e838d54a72a2f9a2d3de338' | 
  [production] | 
            
  | 19:32 | 
  <reedy> | 
  synchronized wmf-config/  'Revert Revert Cross-wiki backlink purging for commons file changes' | 
  [production] | 
            
  | 19:27 | 
  <reedy> | 
  synchronized wmf-config/  'Revert Cross-wiki backlink purging for commons file changes' | 
  [production] | 
            
  | 19:26 | 
  <reedy> | 
  updated /a/common to {{Gerrit|Id4124ef28}}: All non wikipedias to 1.23wmf7 | 
  [production] | 
            
  | 19:13 | 
  <reedy> | 
  rebuilt wikiversions.cdb and synchronized wikiversions files: All non wikipedias to 1.23wmf7 | 
  [production] | 
            
  | 19:11 | 
  <reedy> | 
  synchronized wmf-config/ | 
  [production] | 
            
  | 18:44 | 
  <reedy> | 
  synchronized wmf-config/ | 
  [production] | 
            
  | 18:19 | 
  <reedy> | 
  updated /a/common to {{Gerrit|I5ae36ae21}}: EasyTimeline support for private wikis via img_auth | 
  [production] | 
            
  | 18:11 | 
  <mutante> | 
  deleted php-1.22wmf17 from tin per reedy | 
  [production] | 
            
  | 18:01 | 
  <reedy> | 
  synchronized php-1.23wmf7/includes/revisiondelete/RevisionDelete.php  'I0bd4a5fe9687c4261ca0f57e30f723e8bf2589ac' | 
  [production] | 
            
  | 17:59 | 
  <reedy> | 
  synchronized php-1.23wmf6/includes/revisiondelete/RevisionDelete.php  'I0bd4a5fe9687c4261ca0f57e30f723e8bf2589ac' | 
  [production] | 
            
  | 16:58 | 
  <hashar> | 
  Jenkins is back up | 
  [production] | 
            
  | 16:51 | 
  <hashar> | 
  Jenkins is busy reloading, should be back around 5:20pm UTC.  Don't kill it meanwhile it is busy reading a bunch of files :-( | 
  [production] | 
            
  | 16:27 | 
  <hashar> | 
  restarting Jenkins (stuck) | 
  [production] | 
            
  | 16:17 | 
  <hashar> | 
  Jenkins web service threads are all hang busy waiting for a long request to achieve. Caused by myself :/ | 
  [production] | 
            
  | 16:10 | 
  <hashar> | 
  jenkins in a bad mood for some unknown reason :( | 
  [production] |