| 2013-05-09
      
      § | 
    
  | 21:44 | <hashar> | restarted Jenkins to enable access logs in /var/log/jenkins/access.log {{gerrit|63068}} | [production] | 
            
  | 21:42 | <hashar> | forced Zuul to stop. All events have been dropped :( | [production] | 
            
  | 21:31 | <hashar> | restarting Zuul to make it query Jenkins directly without SSL + apache proxy {{gerrit|63069}}. | [production] | 
            
  | 21:29 | <Jeff_Green> | pdns-update to cut back to aluminium, restarted pdns on nescio b/c it failed | [production] | 
            
  | 21:28 | <^demon> | gerrit repositories will now recursively merge | [production] | 
            
  | 21:16 | <olivneh> | synchronized php-1.22wmf3/extensions/WikimediaMessages  'Updating WikimediaMessages to master' | [production] | 
            
  | 20:47 | <olivneh> | synchronized php-1.22wmf3/extensions/GuidedTour  'Actually updating GuidedTour to master' | [production] | 
            
  | 20:46 | <olivneh> | synchronized php-1.22wmf3/extensions/EventLogging  'Updating GuidedTour to master' | [production] | 
            
  | 20:45 | <olivneh> | synchronized php-1.22wmf3/extensions/EventLogging  'Updating EventLogging to master' | [production] | 
            
  | 20:45 | <olivneh> | synchronized php-1.22wmf3/extensions/ConfirmEdit  'Updating ConfirmEdit to master' | [production] | 
            
  | 20:20 | <hashar> | restarting Jenkins. | [production] | 
            
  | 20:20 | <hashar> | Jenkins died :/ | [production] | 
            
  | 19:16 | <mutante> | fixing file permissions in /a/e3/E3Analysis repo on stat1001 | [production] | 
            
  | 19:09 | <mutante> | restarted Apache and then puppetmaster on stafford, that revived it | [production] | 
            
  | 18:55 | <mutante> | attempting to restart broken puppetmaster on stafford | [production] | 
            
  | 18:27 | <pgehres> | synchronized wmf-config/InitialiseSettings.php  'Enabling CentralAuth debug log' | [production] | 
            
  | 18:00 | <pgehres> | synchronized wmf-config/InitialiseSettings.php  'Enabling wgCentralAuthAutoMigrate on all wikis that use CentralAuth' | [production] | 
            
  | 15:56 | <cmjohnson1> | powering down barium /relocate server to c1 | [production] | 
            
  | 15:26 | <anomie> | synchronized wmf-config/CommonSettings.php  'Make $wgCodeEditorEnableCore configurable per wiki, prep for bug 39653' | [production] | 
            
  | 15:26 | <anomie> | synchronized wmf-config/InitialiseSettings.php  'Make $wgCodeEditorEnableCore configurable per wiki, prep for bug 39653' | [production] | 
            
  | 15:26 | <anomie> | synchronized wmf-config/InitialiseSettings-labs.php  'Make $wgCodeEditorEnableCore configurable per wiki, prep for bug 39653' | [production] | 
            
  | 11:24 | <hashar> | upgrading Zuul to latest master (makes report times friendlier) | [production] | 
            
  | 08:54 | <hashar> | Jenkins apparently working fine.  I have upgraded a few plugins, that seems to have fixed the Gerrit jobs. | [production] | 
            
  | 08:22 | <hashar> | restarting Jenkins again | [production] | 
            
  | 08:20 | <hashar> | upgrading Jenkins plugins 'promoted build' and 'M2 release'. That should fix the Gerrit jobs disappearing (ping qchris) | [production] | 
            
  | 07:39 | <hashar> | restarting jenkins. All its http thread are locked :/ | [production] | 
            
  | 07:25 | <hashar> | Jenkins HTTP interface died :/ | [production] | 
            
  | 03:40 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Thu May  9 03:39:55 UTC 2013 | [production] | 
            
  | 02:16 | <LocalisationUpdate> | completed (1.22wmf3) at Thu May  9 02:15:54 UTC 2013 | [production] | 
            
  
    | 2013-05-08
      
      § | 
    
  | 22:26 | <RobH> | authdns update to migrate racktables from hooper to magnesium.  both servers are already using db1001 as master, which is why hooper is slow to load racktables data | [production] | 
            
  | 22:23 | <mholmquist> | synchronized php-1.22wmf3/extensions/UploadWizard/UploadWizardHooks.php  'Fixing Special:Preferences for wikis with UploadWizard' | [production] | 
            
  | 21:44 | <mholmquist> | Finished syncing Wikimedia installation... : Fix UploadWizard concurrency and minor message updates | [production] | 
            
  | 21:36 | <mholmquist> | Started syncing Wikimedia installation... : Fix UploadWizard concurrency and minor message updates | [production] | 
            
  | 21:06 | <Krinkle> | Running small maintenance script foreachwiki to gather data about odd wgRC2UDPPrefix values (bug 28276) | [production] | 
            
  | 20:20 | <mwalker> | synchronized php-1.22wmf3/extensions/CentralNotice/  'CentralNotice security bugfix 48255' | [production] | 
            
  | 20:15 | <Jeff_Green> | authdns-update to swap in grosley for dead aluminium | [production] | 
            
  | 19:41 | <RobH> | magnesium is future racktables host, disregard errors for now | [production] | 
            
  | 19:40 | <cmjohnson1> | accidently removed power from magnesium | [production] | 
            
  | 19:00 | <Reedy> | Ensuring securepoll database tables exist on all wikis | [production] | 
            
  | 18:35 | <reedy> | synchronized php-1.22wmf3/extensions/CentralAuth | [production] | 
            
  | 18:31 | <Jeff_Green> | updating and rebooting grosley | [production] | 
            
  | 18:19 | <reedy> | synchronized wmf-config/InitialiseSettings.php | [production] | 
            
  | 18:16 | <reedy> | synchronized wmf-config/ | [production] | 
            
  | 18:15 | <reedy> | synchronized database lists files: | [production] | 
            
  | 18:08 | <reedy> | synchronized wmf-config/InitialiseSettings.php  'Enable EducationProgram on mkwiki' | [production] | 
            
  | 18:07 | <Reedy> | Created EducationProgram tables on mkwiki | [production] | 
            
  | 18:04 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else to 1.22wmf3 | [production] | 
            
  | 18:00 | <RobH> | authdnsupdate for racktables2 | [production] | 
            
  | 17:46 | <apergos> | ) | [production] | 
            
  | 17:45 | <apergos> | about to graceful apaches *and api servers* (which somehow are not in the actual apache graceful all script | [production] |