| 2014-07-29
      
      § | 
    
  | 15:06 | <manybubbles> | Synchronized wmf-config: SWAT - deploy cirrussearch all field stage 2 part 2 (duration: 00m 04s) | [production] | 
            
  | 15:06 | <manybubbles> | Synchronized wmf-config/InitialiseSettings.php: SWAT - deploy cirrussearch all field stage 2 part 1 (duration: 00m 04s) | [production] | 
            
  | 13:54 | <hashar> | Synchronized wmf-config/InitialiseSettings.php: added Universiteits Museum Utrecht to the wgCopyUploadsDomains array {{gerrit|150163}} (duration: 00m 04s) | [production] | 
            
  | 13:38 | <ottomata> | restarted gmetad on nickel, seems to have brought ganglia back up | [production] | 
            
  | 11:31 | <_joe_> | upgrading packages on mw1053, for testing hhvm with pcre-jit enabled | [production] | 
            
  | 10:35 | <_joe_> | puppet re-enabled on the appservers | [production] | 
            
  | 10:29 | <_joe_> | temporarily stopping puppet on appservers, releasing a potentially dangerous puppet change | [production] | 
            
  | 09:10 | <_joe_> | stopping jobrunner on mw1053, disabling puppet as well - running tests | [production] | 
            
  | 09:02 | <hashar> | restarted zuul-server and zuul-merger on gallium (new version though that is a noop) | [production] | 
            
  | 09:00 | <hashar_> | Zuul bumping Zuul cloner from patchset 21 to patchset 23. Deploying with tag wmf-deploy-2014-07-29-1 | [production] | 
            
  | 07:51 | <akosiaris> | uploaded PHP 5.3.10-1ubuntu3.13+wmf1 on apt.wikimedia.org. Puppet will upgrade it across the fleet within 20 mins | [production] | 
            
  | 03:48 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Tue Jul 29 03:47:39 UTC 2014 (duration 47m 38s) | [production] | 
            
  | 03:11 | <LocalisationUpdate> | completed (1.24wmf15) at 2014-07-29 03:10:31+00:00 | [production] | 
            
  | 02:36 | <LocalisationUpdate> | completed (1.24wmf14) at 2014-07-29 02:35:18+00:00 | [production] | 
            
  | 00:44 | <aaron> | Synchronized php-1.24wmf15/maintenance/runJobs.php: fcfa3153e53dc70e6cd190a087e7bd577fe380fb (duration: 00m 03s) | [production] | 
            
  | 00:27 | <aaron> | Synchronized php-1.24wmf15/maintenance: f754c239ce93fc5f2db19e93f4fe8a1d1ba7bc27 (duration: 00m 04s) | [production] | 
            
  | 00:27 | <aaron> | Synchronized php-1.24wmf15/includes: f754c239ce93fc5f2db19e93f4fe8a1d1ba7bc27 (duration: 00m 06s) | [production] | 
            
  
    | 2014-07-28
      
      § | 
    
  | 23:58 | <ori> | Finished scap: I42c07b64: Update MobileFrontend (duration: 17m 37s) | [production] | 
            
  | 23:41 | <ori> | Started scap: I42c07b64: Update MobileFrontend | [production] | 
            
  | 23:33 | <ori> | Synchronized php-1.24wmf15/extensions/VisualEditor: Update VisualEditor to I944f8fbfa (duration: 00m 04s) | [production] | 
            
  | 23:25 | <ori> | Synchronized wmf-config/InitialiseSettings.php: I369dbad6e: Allow crats to add/remove petitiondata group on foundationWiki (duration: 00m 04s) | [production] | 
            
  | 23:21 | <AaronS> | Updated /srv/jobrunner to 0bb0ad62dd9240e0f67b2ded4519f125de13dfbc | [production] | 
            
  | 23:12 | <mutante> | temp. disabled puppet on neon and ircecho | [production] | 
            
  | 23:06 | <mutante> | graceful apache on palladium | [production] | 
            
  | 21:13 | <hashar> | Gerrit: allowed JenkinsBot to submit patches on wikimedia/bots (and thus on all child repositories) | [production] | 
            
  | 20:50 | <hashar> | operations/puppet.git manifests should no more have leading tabulations {{gerritI69ddc72f5a072ac7dc4f67622b65f36a70d3c021}} | [production] | 
            
  | 20:08 | <bblack> | intermittent 5xx are most likely varnish restarts off and on rest of today | [production] | 
            
  | 19:51 | <hashar> | Zuul: stopped / started process to clear up obsoletes changes stuck in queue | [production] | 
            
  | 19:47 | <hashar> | Jenkins/Zuul lost connection somehow. Disabled/Reenabled gearman client in Jenkins | [production] | 
            
  | 19:44 | <hashar> | Jenkins: updated qunit jobs to roam on both gallium and lanthanum (were previously tied to run only on gallium) | [production] | 
            
  | 19:43 | <ottomata> | restarted varnishkafka on some esams hosts that seem old misconfigured vk processes | [production] | 
            
  | 19:13 | <ottomata> | restarting varnishkafka on amssq31 | [production] | 
            
  | 19:08 | <ottomata> | restarting varnishkafka on cp3013 | [production] | 
            
  | 17:46 | <aaron> | Synchronized php-1.24wmf14/includes/jobqueue/JobQueueFederated.php: 87e7bfceb795d065d6157ac8ce3381a7814000b5 (duration: 00m 03s) | [production] | 
            
  | 17:38 | <aaron> | Synchronized php-1.24wmf15/includes/jobqueue/JobQueueFederated.php: 12ce1dc1ec46b06d1160e142ddfaf8dcb1c9f131 (duration: 00m 04s) | [production] | 
            
  | 16:30 | <andrewbogott> | updated wikitech to 1.24wmf15; turned on OAuth | [production] | 
            
  | 16:05 | <Nemo_bis> | andrewbogott> Nikerabbit: I'm upgrading it [wikitech wiki], it'll be flaky for a bit | [production] | 
            
  | 16:00 | <manybubbles> | deone with SWAT | [production] | 
            
  | 15:57 | <manybubbles> | Synchronized php-1.24wmf14/extensions/VisualEditor/: SWAT - fix visual editor bug - Changes made after reviewing changes are not sent (when caching is enabled) (duration: 00m 07s) | [production] | 
            
  | 15:47 | <manybubbles> | Synchronized php-1.24wmf15/extensions/VisualEditor/: SWAT - fix visual editor bug - Changes made after reviewing changes are not sent (when caching is enabled) (duration: 00m 08s) | [production] | 
            
  | 15:41 | <hoo> | Removed all right holders from closed and inaccessible ukwikimedia (bug 68737) | [production] | 
            
  | 15:39 | <manybubbles> | Synchronized php-1.24wmf15/includes/specials/SpecialRevisiondelete.php: SWAT - fix fatal on revision delete (duration: 00m 08s) | [production] | 
            
  | 15:33 | <manybubbles> | Synchronized wmf-config/CommonSettings.php: SWAT load Mantle before MobileFrontent (duration: 00m 07s) | [production] | 
            
  | 15:31 | <manybubbles> | Synchronized php-1.24wmf14/extensions/Echo/: SWAT fix bad variable name in echo (duration: 00m 08s) | [production] | 
            
  | 15:23 | <manybubbles> | Synchronized wmf-config/InitialiseSettings.php: SWAT - update some permissions on eswiki (duration: 00m 08s) | [production] | 
            
  | 15:17 | <manybubbles> | Synchronized php-1.24wmf15/extensions/Echo/: SWAT - fix incorrect variable name (duration: 00m 08s) | [production] | 
            
  | 15:15 | <manybubbles> | Synchronized wmf-config/InitialiseSettings.php: SWAT - add import sources to bhwiki (duration: 00m 08s) | [production] | 
            
  | 15:10 | <manybubbles> | Synchronized php-1.24wmf14/extensions/FundraisingTranslateWorkflow/: SWAT update fundraising to fix botched deploy | [production] | 
            
  | 12:28 | <hashar> | Upgrading our Jenkins Job Builder fork ( d833015..666e953 ) | [production] | 
            
  | 03:00 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Mon Jul 28 02:59:35 UTC 2014 (duration 59m 34s) | [production] |