| 
      
        2013-09-24
      
      §
     | 
  
    
  | 15:24 | 
  <mutante> | 
  freshly activated  http://wikimediastories.org/ (and .com,.net) | 
  [production] | 
            
  | 15:21 | 
  <mutante> | 
  sync-apache and graceful-all for wikimediastories and other cleanup | 
  [production] | 
            
  | 15:21 | 
  <yurik> | 
  synchronized php-1.22wmf18/extensions/ZeroRatedMobileAccess/ | 
  [production] | 
            
  | 15:13 | 
  <mutante> | 
  during sync-apache: mw1072 - read-only file system | 
  [production] | 
            
  | 15:07 | 
  <cmjohnson> | 
  db1053 removing/replacing disk 3 | 
  [production] | 
            
  | 15:05 | 
  <yurik> | 
  synchronized php-1.22wmf17/extensions/ZeroRatedMobileAccess/ | 
  [production] | 
            
  | 14:59 | 
  <cmjohnson> | 
  db1001 removing/replacing disk 7  | 
  [production] | 
            
  | 14:48 | 
  <cmjohnson> | 
  db1031 removing/replacing disk 7 | 
  [production] | 
            
  | 14:38 | 
  <cmjohnson> | 
  replacing mw1046 hard drive --requires reinstall | 
  [production] | 
            
  | 14:21 | 
  <LocalisationUpdate> | 
  ResourceLoader cache refresh completed at Tue Sep 24 14:21:10 UTC 2013 | 
  [production] | 
            
  | 14:15 | 
  <LocalisationUpdate> | 
  completed (1.22wmf17) at Tue Sep 24 14:15:56 UTC 2013 | 
  [production] | 
            
  | 14:15 | 
  <LocalisationUpdate> | 
  completed (1.22wmf18) at Tue Sep 24 14:15:16 UTC 2013 | 
  [production] | 
            
  | 14:13 | 
  <demon> | 
  synchronized php-1.22wmf18/extensions/Elastica  'Sync message fixes' | 
  [production] | 
            
  | 12:26 | 
  <mutante> | 
  shutting down sq45 | 
  [production] | 
            
  | 11:04 | 
  <mutante> | 
  powercycled sq45  | 
  [production] | 
            
  | 10:08 | 
  <p858snake|l> | 
  [20:01AEST] <mark> upped to weight 5 | 
  [production] | 
            
  | 10:08 | 
  <p858snake|l> | 
  [19:51AEST] <mark> !log Pooled Varnish Text caches in eqiad with weight 1 (affecting wikidata, wikivoyage, *.wikimedia.org) | 
  [production] | 
            
  | 10:07 | 
  <p858snake|l> | 
  [19:36AEST] <akosiaris> !log rebooted labstore4, controller configuration lost ? | 
  [production] | 
            
  | 10:07 | 
  <p858snake|l> | 
  [19:36AEST] <akosiaris> !log rebooted labstore3 (thought the 14days NFS issue was resolved???) | 
  [production] | 
            
  | 10:04 | 
  <akosiaris> | 
  rebooted labstore4, controller configuration lost ? | 
  [production] | 
            
  | 10:04 | 
  <akosiaris> | 
  rebooted labstore3 (thought the 14days NFS issue was resolved???) | 
  [production] | 
            
  | 08:30 | 
  <hashar> | 
  restarting Jenkins again for one last plugin upgrade | 
  [production] | 
            
  | 08:30 | 
  <springle> | 
  labstore4 disk issues, /a failed mount in console, ssh key bouncing. awaiting input from ryan or coren | 
  [production] | 
            
  | 08:30 | 
  <hashar> | 
  8:20am INFO: Jenkins is fully up and running | 
  [production] | 
            
  | 08:20 | 
  <hashar> | 
  restarting Jenkins | 
  [production] | 
            
  | 08:12 | 
  <hashar> | 
  jenkins: upgrading plugins | 
  [production] | 
            
  | 08:12 | 
  <hashar> | 
  8:12am INFO: Jenkins is fully up and running | 
  [production] | 
            
  | 08:06 | 
  <springle> | 
  labstore4 locked up, power cycled | 
  [production] | 
            
  | 07:43 | 
  <hashar> | 
  upgrading Jenkins on gallium from 1.509.2 to 1.509.3 | 
  [production] | 
            
  | 06:36 | 
  <springle> | 
  upgrading db1002 to precise and mariadb | 
  [production] | 
            
  | 06:13 | 
  <^demon|zzz> | 
  ES: enwikisource & cawiki done indexing. Really need to revisit batch indexing, way too slow. | 
  [production] | 
            
  | 03:42 | 
  <springle> | 
  synchronized wmf-config/db-eqiad.php  'depool db1002 for upgrade, db1036 take up slack' | 
  [production] | 
            
  | 03:17 | 
  <springle> | 
  synchronized wmf-config/db-eqiad.php | 
  [production] | 
            
  | 02:40 | 
  <LocalisationUpdate> | 
  ResourceLoader cache refresh completed at Tue Sep 24 02:40:38 UTC 2013 | 
  [production] | 
            
  | 02:21 | 
  <LocalisationUpdate> | 
  completed (1.22wmf17) at Tue Sep 24 02:21:27 UTC 2013 | 
  [production] | 
            
  | 02:15 | 
  <LocalisationUpdate> | 
  completed (1.22wmf18) at Tue Sep 24 02:15:01 UTC 2013 | 
  [production] | 
            
  
    | 
      
        2013-09-23
      
      §
     | 
  
    
  | 23:36 | 
  <RoanKattouw> | 
  mw1072 is unhappy, sync-file throws warnings about read-only filesystem | 
  [production] | 
            
  | 23:35 | 
  <catrope> | 
  synchronized php-1.22wmf18/extensions/VisualEditor  'VisualEditor cherry-picks' | 
  [production] | 
            
  | 23:35 | 
  <catrope> | 
  synchronized php-1.22wmf17/extensions/VisualEditor  'VisualEditor cherry-picks' | 
  [production] | 
            
  | 23:20 | 
  <catrope> | 
  synchronized wmf-config/InitialiseSettings.php  'Set VisualEditor to opt-in on enwiki' | 
  [production] | 
            
  | 20:57 | 
  <demon> | 
  synchronized wmf-config/InitialiseSettings.php | 
  [production] | 
            
  | 20:48 | 
  <reedy> | 
  synchronized wmf-config/  'touch' | 
  [production] | 
            
  | 20:46 | 
  <reedy> | 
  synchronized php-1.22wmf18/extensions/UploadWizard | 
  [production] | 
            
  | 20:09 | 
  <Reedy> | 
  mw1072 has a r/o filesystem | 
  [production] | 
            
  | 20:05 | 
  <demon> | 
  synchronized wmf-config/InitialiseSettings.php  'Cirrus as primary for itwikt, secondary on cawiki and enwikisource' | 
  [production] | 
            
  | 19:46 | 
  <reedy> | 
  synchronized wmf-config  'touch' | 
  [production] | 
            
  | 19:36 | 
  <reedy> | 
  synchronized database lists files: | 
  [production] | 
            
  | 19:35 | 
  <reedy> | 
  synchronized wmf-config/ | 
  [production] | 
            
  | 18:50 | 
  <Ryan_Lane> | 
  enabling cirrussearch for wikitech | 
  [production] | 
            
  | 18:43 | 
  <Reedy> | 
  Created sites and site_identifier tables on commonswiki | 
  [production] |