| 2014-07-28
      
      § | 
    
  | 19:47 | <hashar> | Jenkins/Zuul lost connection somehow. Disabled/Reenabled gearman client in Jenkins | [production] | 
            
  | 19:44 | <hashar> | Jenkins: updated qunit jobs to roam on both gallium and lanthanum (were previously tied to run only on gallium) | [production] | 
            
  | 19:43 | <ottomata> | restarted varnishkafka on some esams hosts that seem old misconfigured vk processes | [production] | 
            
  | 19:13 | <ottomata> | restarting varnishkafka on amssq31 | [production] | 
            
  | 19:08 | <ottomata> | restarting varnishkafka on cp3013 | [production] | 
            
  | 17:46 | <aaron> | Synchronized php-1.24wmf14/includes/jobqueue/JobQueueFederated.php: 87e7bfceb795d065d6157ac8ce3381a7814000b5 (duration: 00m 03s) | [production] | 
            
  | 17:38 | <aaron> | Synchronized php-1.24wmf15/includes/jobqueue/JobQueueFederated.php: 12ce1dc1ec46b06d1160e142ddfaf8dcb1c9f131 (duration: 00m 04s) | [production] | 
            
  | 16:30 | <andrewbogott> | updated wikitech to 1.24wmf15; turned on OAuth | [production] | 
            
  | 16:05 | <Nemo_bis> | andrewbogott> Nikerabbit: I'm upgrading it [wikitech wiki], it'll be flaky for a bit | [production] | 
            
  | 16:00 | <manybubbles> | deone with SWAT | [production] | 
            
  | 15:57 | <manybubbles> | Synchronized php-1.24wmf14/extensions/VisualEditor/: SWAT - fix visual editor bug - Changes made after reviewing changes are not sent (when caching is enabled) (duration: 00m 07s) | [production] | 
            
  | 15:47 | <manybubbles> | Synchronized php-1.24wmf15/extensions/VisualEditor/: SWAT - fix visual editor bug - Changes made after reviewing changes are not sent (when caching is enabled) (duration: 00m 08s) | [production] | 
            
  | 15:41 | <hoo> | Removed all right holders from closed and inaccessible ukwikimedia (bug 68737) | [production] | 
            
  | 15:39 | <manybubbles> | Synchronized php-1.24wmf15/includes/specials/SpecialRevisiondelete.php: SWAT - fix fatal on revision delete (duration: 00m 08s) | [production] | 
            
  | 15:33 | <manybubbles> | Synchronized wmf-config/CommonSettings.php: SWAT load Mantle before MobileFrontent (duration: 00m 07s) | [production] | 
            
  | 15:31 | <manybubbles> | Synchronized php-1.24wmf14/extensions/Echo/: SWAT fix bad variable name in echo (duration: 00m 08s) | [production] | 
            
  | 15:23 | <manybubbles> | Synchronized wmf-config/InitialiseSettings.php: SWAT - update some permissions on eswiki (duration: 00m 08s) | [production] | 
            
  | 15:17 | <manybubbles> | Synchronized php-1.24wmf15/extensions/Echo/: SWAT - fix incorrect variable name (duration: 00m 08s) | [production] | 
            
  | 15:15 | <manybubbles> | Synchronized wmf-config/InitialiseSettings.php: SWAT - add import sources to bhwiki (duration: 00m 08s) | [production] | 
            
  | 15:10 | <manybubbles> | Synchronized php-1.24wmf14/extensions/FundraisingTranslateWorkflow/: SWAT update fundraising to fix botched deploy | [production] | 
            
  | 12:28 | <hashar> | Upgrading our Jenkins Job Builder fork ( d833015..666e953 ) | [production] | 
            
  | 03:00 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Mon Jul 28 02:59:35 UTC 2014 (duration 59m 34s) | [production] | 
            
  | 02:26 | <LocalisationUpdate> | completed (1.24wmf15) at 2014-07-28 02:25:34+00:00 | [production] | 
            
  | 02:16 | <LocalisationUpdate> | completed (1.24wmf14) at 2014-07-28 02:15:00+00:00 | [production] | 
            
  
    | 2014-07-25
      
      § | 
    
  | 22:52 | <mutante> | Bugzilla - upgraded to 4.4.5 | [production] | 
            
  | 22:41 | <mutante> | ocg - deleted old log dirs | [production] | 
            
  | 19:28 | <hashar> | Jenkins : disabling gearman plugin and reenabling it (just uncheck/save/check a box in  https://integration.wikimedia.org/ci/configure ) | [production] | 
            
  | 19:25 | <hashar> | zuul@gallium:/etc/zuul/wikimedia$ echo status|nc -q 3 localhost 4730|wc -l   ... Yields: 0 .  Which mean jobs are no more registered for some reason. | [production] | 
            
  | 19:24 | <hashar> | Jenkins stalled again yeahhhhh | [production] | 
            
  | 16:59 | <mutante> | powercycled ms-be1010 - unresponsive to ssh, nothing on mgmt | [production] | 
            
  | 16:28 | <MaxSem> | Updating PageImages data for mainspace on Commons from terbium | [production] | 
            
  | 13:09 | <_joe_> | re-enabling puppet, test run on the test host was fine. | [production] | 
            
  | 13:03 | <_joe_> | stopping puppet on all appservers - will reactivate after testing | [production] | 
            
  | 11:26 | <reedy> | Synchronized docroot and w: (no message) (duration: 00m 13s) | [production] | 
            
  | 10:51 | <hashar> | contint: manually cleared /tmp on the 3 labs jenkins slaves. | [production] | 
            
  | 10:47 | <hashar> | integration-slave1001.eqiad.wmflabs  is out of disk space ( / /dev/vda1) | [production] | 
            
  | 07:29 | <springle> | shutdown tantalum per mwalker request | [production] | 
            
  | 04:19 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Fri Jul 25 04:18:45 UTC 2014 (duration 18m 44s) | [production] | 
            
  | 03:31 | <LocalisationUpdate> | completed (1.24wmf15) at 2014-07-25 03:30:33+00:00 | [production] | 
            
  | 02:48 | <LocalisationUpdate> | completed (1.24wmf14) at 2014-07-25 02:47:17+00:00 | [production] | 
            
  | 01:22 | <ori> | Synchronized wmf-config/CommonSettings.php: Ic29ae11fa: On Labs, disable LuaSandbox's profiling feature to isolate bug 68413 (duration: 00m 04s) | [production] | 
            
  | 00:15 | <mutante> | imported jouncebot from github - https://gerrit.wikimedia.org/r/#/q/project:wikimedia/bots/jouncebot,n,z | [production] |