| 
      
        2014-12-15
      
      §
     | 
  
    
  | 14:00 | 
  <robh> | 
  zinc removed from icinga, system is now shutdown for reclaim per RT8939 | 
  [production] | 
            
  | 13:50 | 
  <robh> | 
  reclaiming zinc to spares, stopped puppet agent | 
  [production] | 
            
  | 13:13 | 
  <akosiaris> | 
  uploaded hfst_3.8.1~r4088-1 to apt.wikimedia.org (trusty) | 
  [production] | 
            
  | 11:51 | 
  <hashar> | 
  Zuul: clearing out some old zuul git references  ( https://phabricator.wikimedia.org/T70481 ). Running in a screen on gallium | 
  [production] | 
            
  | 09:48 | 
  <hashar> | 
  Upgrading composer on CI to v1.0.0-alpha9 {{gerrit|178550}} | 
  [production] | 
            
  | 06:57 | 
  <ori> | 
  Synchronized wmf-config/InitialiseSettings.php: Id9023e66c: Sample "api" debug log group at 1:1000 (duration: 00m 06s) | 
  [production] | 
            
  | 03:36 | 
  <LocalisationUpdate> | 
  ResourceLoader cache refresh completed at Mon Dec 15 03:36:24 UTC 2014 (duration 36m 23s) | 
  [production] | 
            
  | 02:15 | 
  <LocalisationUpdate> | 
  completed (1.25wmf12) at 2014-12-15 02:15:01+00:00 | 
  [production] | 
            
  | 02:15 | 
  <l10nupdate> | 
  Synchronized php-1.25wmf12/cache/l10n: (no message) (duration: 00m 01s) | 
  [production] | 
            
  | 02:09 | 
  <LocalisationUpdate> | 
  completed (1.25wmf11) at 2014-12-15 02:09:21+00:00 | 
  [production] | 
            
  | 02:09 | 
  <l10nupdate> | 
  Synchronized php-1.25wmf11/cache/l10n: (no message) (duration: 00m 02s) | 
  [production] | 
            
  
    | 
      
        2014-12-13
      
      §
     | 
  
    
  | 12:20 | 
  <andrewbogott> | 
  graceful'd apache2 on virt1000; puppet master was acting up. | 
  [production] | 
            
  | 09:51 | 
  <hashar> | 
  Restarting Jenkins to get rid of some deadlocks that occurred yesterday | 
  [production] | 
            
  | 03:29 | 
  <LocalisationUpdate> | 
  ResourceLoader cache refresh completed at Sat Dec 13 03:29:31 UTC 2014 (duration 29m 30s) | 
  [production] | 
            
  | 02:14 | 
  <LocalisationUpdate> | 
  completed (1.25wmf12) at 2014-12-13 02:14:00+00:00 | 
  [production] | 
            
  | 02:14 | 
  <l10nupdate> | 
  Synchronized php-1.25wmf12/cache/l10n: (no message) (duration: 00m 01s) | 
  [production] | 
            
  | 02:09 | 
  <LocalisationUpdate> | 
  completed (1.25wmf11) at 2014-12-13 02:09:28+00:00 | 
  [production] | 
            
  | 02:09 | 
  <l10nupdate> | 
  Synchronized php-1.25wmf11/cache/l10n: (no message) (duration: 00m 01s) | 
  [production] | 
            
  | 02:04 | 
  <krinkle> | 
  Synchronized w/robots.php: 54746fdef3402 (duration: 00m 05s) | 
  [production] | 
            
  | 00:58 | 
  <krinkle> | 
  Synchronized w/robots.php: 611892c62349d09c9758 (duration: 00m 06s) | 
  [production] | 
            
  
    | 
      
        2014-12-12
      
      §
     | 
  
    
  | 22:06 | 
  <ori> | 
  Synchronized wmf-config/StartProfiler.php: (no message) (duration: 00m 06s) | 
  [production] | 
            
  | 19:47 | 
  <ottomata1> | 
  initiating kafka preferred-replica-election  to bring analytics1021 back in to leadership :/  need to figure this out, or replace this node soon. | 
  [production] | 
            
  | 18:15 | 
  <YuviPanda> | 
  ran sudo logrotate -f /etc/logrotate.d/dumpwikidatajson on snapshot1003 forhoo | 
  [production] | 
            
  | 18:13 | 
  <ori> | 
  Synchronized wmf-config/StartProfiler.php: (no message) (duration: 00m 08s) | 
  [production] | 
            
  | 16:42 | 
  <akosiaris> | 
  uploaded apertium-sv-da, apertium-en-ca to apt.wikimedia.org | 
  [production] | 
            
  | 15:13 | 
  <hashar> | 
  Zuul Reverting Zuul back to  wmf-deploy-20141030-4 . I previously reverted it to another change which was wrong. | 
  [production] | 
            
  | 14:59 | 
  <hashar> | 
  Zuul status page is no more. https://phabricator.wikimedia.org/T78400 | 
  [production] | 
            
  | 14:50 | 
  <hashar> | 
  upgrading python-statsd on Zuul server and restarting service. | 
  [production] | 
            
  | 14:37 | 
  <godog> | 
  upload python-statsd 3.0.1-1 to precise-wikimedia | 
  [production] | 
            
  | 14:13 | 
  <godog> | 
  upload python-statsd 3.0.1-1 to trusty-wikimedia | 
  [production] | 
            
  | 11:43 | 
  <YuviPanda> | 
  force puppet run on all labs hosts via salt | 
  [production] | 
            
  | 09:33 | 
  <ori> | 
  restarted mwprof on tungsten | 
  [production] | 
            
  | 09:20 | 
  <ori> | 
  Synchronized wmf-config/StartProfiler.php: I63864cc79: xenon log: collate stack samples and fold into single lines (duration: 00m 06s) | 
  [production] | 
            
  | 06:05 | 
  <springle> | 
  Synchronized wmf-config/db-eqiad.php: depool db1055 (duration: 00m 05s) | 
  [production] | 
            
  | 03:50 | 
  <LocalisationUpdate> | 
  ResourceLoader cache refresh completed at Fri Dec 12 03:50:10 UTC 2014 (duration 50m 9s) | 
  [production] | 
            
  | 03:28 | 
  <yurik> | 
  Synchronized php-1.25wmf12/extensions/ZeroPortal/: updatidng ZeroPortal to master - urgent bugfix - retry (duration: 00m 10s) | 
  [production] | 
            
  | 03:27 | 
  <yurik> | 
  Synchronized php-1.25wmf12/extensions/ZeroPortal/: updatidng ZeroPortal to master - urgent bugfix (duration: 00m 05s) | 
  [production] | 
            
  | 03:25 | 
  <ori> | 
  Synchronized wmf-config: I1d218c2d6: Log xenon-captured traces via wfDebugLog (duration: 00m 06s) | 
  [production] | 
            
  | 02:55 | 
  <andrewbogott> | 
  rebooted mw1041 from mgmt | 
  [production] | 
            
  | 02:20 | 
  <LocalisationUpdate> | 
  completed (1.25wmf12) at 2014-12-12 02:20:37+00:00 | 
  [production] | 
            
  | 02:20 | 
  <l10nupdate> | 
  Synchronized php-1.25wmf12/cache/l10n: (no message) (duration: 00m 03s) | 
  [production] | 
            
  | 02:15 | 
  <LocalisationUpdate> | 
  completed (1.25wmf11) at 2014-12-12 02:15:34+00:00 | 
  [production] | 
            
  | 02:15 | 
  <l10nupdate> | 
  Synchronized php-1.25wmf11/cache/l10n: (no message) (duration: 00m 03s) | 
  [production] | 
            
  | 00:41 | 
  <maxsem> | 
  Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/179360 (duration: 00m 06s) | 
  [production] | 
            
  | 00:35 | 
  <godog> | 
  profiler-to-carbon is logging too much on tungsten, cause unknown yet but don't restart | 
  [production] |