| 2013-05-09
      
      § | 
    
  | 15:56 | <cmjohnson1> | powering down barium /relocate server to c1 | [production] | 
            
  | 15:26 | <anomie> | synchronized wmf-config/CommonSettings.php  'Make $wgCodeEditorEnableCore configurable per wiki, prep for bug 39653' | [production] | 
            
  | 15:26 | <anomie> | synchronized wmf-config/InitialiseSettings.php  'Make $wgCodeEditorEnableCore configurable per wiki, prep for bug 39653' | [production] | 
            
  | 15:26 | <anomie> | synchronized wmf-config/InitialiseSettings-labs.php  'Make $wgCodeEditorEnableCore configurable per wiki, prep for bug 39653' | [production] | 
            
  | 11:24 | <hashar> | upgrading Zuul to latest master (makes report times friendlier) | [production] | 
            
  | 08:54 | <hashar> | Jenkins apparently working fine.  I have upgraded a few plugins, that seems to have fixed the Gerrit jobs. | [production] | 
            
  | 08:22 | <hashar> | restarting Jenkins again | [production] | 
            
  | 08:20 | <hashar> | upgrading Jenkins plugins 'promoted build' and 'M2 release'. That should fix the Gerrit jobs disappearing (ping qchris) | [production] | 
            
  | 07:39 | <hashar> | restarting jenkins. All its http thread are locked :/ | [production] | 
            
  | 07:25 | <hashar> | Jenkins HTTP interface died :/ | [production] | 
            
  | 03:40 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Thu May  9 03:39:55 UTC 2013 | [production] | 
            
  | 02:16 | <LocalisationUpdate> | completed (1.22wmf3) at Thu May  9 02:15:54 UTC 2013 | [production] | 
            
  
    | 2013-05-08
      
      § | 
    
  | 22:26 | <RobH> | authdns update to migrate racktables from hooper to magnesium.  both servers are already using db1001 as master, which is why hooper is slow to load racktables data | [production] | 
            
  | 22:23 | <mholmquist> | synchronized php-1.22wmf3/extensions/UploadWizard/UploadWizardHooks.php  'Fixing Special:Preferences for wikis with UploadWizard' | [production] | 
            
  | 21:44 | <mholmquist> | Finished syncing Wikimedia installation... : Fix UploadWizard concurrency and minor message updates | [production] | 
            
  | 21:36 | <mholmquist> | Started syncing Wikimedia installation... : Fix UploadWizard concurrency and minor message updates | [production] | 
            
  | 21:06 | <Krinkle> | Running small maintenance script foreachwiki to gather data about odd wgRC2UDPPrefix values (bug 28276) | [production] | 
            
  | 20:20 | <mwalker> | synchronized php-1.22wmf3/extensions/CentralNotice/  'CentralNotice security bugfix 48255' | [production] | 
            
  | 20:15 | <Jeff_Green> | authdns-update to swap in grosley for dead aluminium | [production] | 
            
  | 19:41 | <RobH> | magnesium is future racktables host, disregard errors for now | [production] | 
            
  | 19:40 | <cmjohnson1> | accidently removed power from magnesium | [production] | 
            
  | 19:00 | <Reedy> | Ensuring securepoll database tables exist on all wikis | [production] | 
            
  | 18:35 | <reedy> | synchronized php-1.22wmf3/extensions/CentralAuth | [production] | 
            
  | 18:31 | <Jeff_Green> | updating and rebooting grosley | [production] | 
            
  | 18:19 | <reedy> | synchronized wmf-config/InitialiseSettings.php | [production] | 
            
  | 18:16 | <reedy> | synchronized wmf-config/ | [production] | 
            
  | 18:15 | <reedy> | synchronized database lists files: | [production] | 
            
  | 18:08 | <reedy> | synchronized wmf-config/InitialiseSettings.php  'Enable EducationProgram on mkwiki' | [production] | 
            
  | 18:07 | <Reedy> | Created EducationProgram tables on mkwiki | [production] | 
            
  | 18:04 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else to 1.22wmf3 | [production] | 
            
  | 18:00 | <RobH> | authdnsupdate for racktables2 | [production] | 
            
  | 17:46 | <apergos> | ) | [production] | 
            
  | 17:45 | <apergos> | about to graceful apaches *and api servers* (which somehow are not in the actual apache graceful all script | [production] | 
            
  | 17:42 | <Jeff_Green> | kernel updates and reboot on pappas | [production] | 
            
  | 17:29 | <yurik> | Finished syncing Wikimedia installation... : Zero extension | [production] | 
            
  | 17:17 | <RobH> | restarting all mw jobrunners | [production] | 
            
  | 17:13 | <aaron> | synchronized wmf-config/CommonSettings.php  'Disabled TTMServerMessageUpdateJob jobs' | [production] | 
            
  | 17:11 | <aaron> | synchronized php-1.22wmf3/includes/job/JobQueueGroup.php  'revert hack' | [production] | 
            
  | 17:11 | <aaron> | synchronized php-1.22wmf2/includes/job/JobQueueGroup.php  'revert hack' | [production] | 
            
  | 17:07 | <yurik> | Started syncing Wikimedia installation... : Zero extension | [production] | 
            
  | 15:00 | <mark> | synchronized wmf-config/mc-eqiad.php | [production] | 
            
  | 14:22 | <paravoid> | ceph osd out 39, primary for stuck unclean pg 3.5c6 | [production] | 
            
  | 14:20 | <paravoid> | ceph osd out 128, primary for stuck unclean pg 3.19e1 | [production] | 
            
  | 13:58 | <mark> | synchronized wmf-config/mc-eqiad.php | [production] | 
            
  | 13:58 | <mark> | Removing mc1009 from the eqiad memcached pool | [production] | 
            
  | 06:14 | <apergos> | restarted adminbot (morebot) on wikitech-static | [production] |