| 
      
        2013-08-21
      
      §
     | 
  
    
  | 22:31 | 
  <binasher> | 
  opened up wikiadmin grants on es1 shard to accommodate snapshot1004  | 
  [production] | 
            
  | 20:59 | 
  <mutante> | 
  cp1061,cp1062 have dpkg issue upgrading varnish packages | 
  [production] | 
            
  | 20:36 | 
  <mutante> | 
  installing package upgrades on iron | 
  [production] | 
            
  | 20:35 | 
  <mutante> | 
  re-enabled puppet runs on zirconium which was "admin down" | 
  [production] | 
            
  | 20:20 | 
  <mutante> | 
  dist-upgrading zirconium | 
  [production] | 
            
  | 18:41 | 
  <reedy> | 
  synchronized wmf-config/InitialiseSettings.php  'Disable Collection on testwikidatawiki. It's mucho brokenp' | 
  [production] | 
            
  | 18:38 | 
  <reedy> | 
  synchronized wmf-config/InitialiseSettings.php  'Enable Collection on testwikidatawiki' | 
  [production] | 
            
  | 18:29 | 
  <aaron> | 
  synchronized wmf-config/CommonSettings.php  'Enabled $wgMWOAuthSecureTokenTransfer' | 
  [production] | 
            
  | 16:36 | 
  <reedy> | 
  synchronized wmf-config/InitialiseSettings.php | 
  [production] | 
            
  | 16:27 | 
  <paravoid> | 
  DNS migration complete | 
  [production] | 
            
  | 16:21 | 
  <paravoid> | 
  switching ns2 to eeden | 
  [production] | 
            
  | 16:04 | 
  <reedy> | 
  synchronized docroot and w | 
  [production] | 
            
  | 15:51 | 
  <paravoid> | 
  switching ns0 to rubidium | 
  [production] | 
            
  | 15:46 | 
  <reedy> | 
  synchronized wmf-config/ | 
  [production] | 
            
  | 15:46 | 
  <reedy> | 
  synchronized wmf-config/ | 
  [production] | 
            
  | 15:36 | 
  <faidon> | 
  synchronized wmf-config/filebackend.php  'revert ceph promotion to master' | 
  [production] | 
            
  | 13:58 | 
  <paravoid> | 
  stopping ceph-osd 88 & 95 (ms-be1008), evidence of unexplainable high cpu usage | 
  [production] | 
            
  | 13:38 | 
  <paravoid> | 
  switching ns1 traffic to mexia (new authdns) | 
  [production] | 
            
  | 12:08 | 
  <Reedy> | 
  Running extensions/CentralAuth/maintenance/populateHomeDB.php in a screen on tin | 
  [production] | 
            
  | 11:02 | 
  <faidon> | 
  synchronized wmf-config/filebackend.php  'promote ceph to master' | 
  [production] | 
            
  | 08:01 | 
  <nikerabbit> | 
  synchronized php-1.22wmf13/extensions/UniversalLanguageSelector/  'ULS to master' | 
  [production] | 
            
  | 02:46 | 
  <mutante> | 
  RT - create new system dashboard 'wikimedia default' that lists open ops-requests, quick ticket creation, reminders and bookmarked tickets | 
  [production] | 
            
  | 02:46 | 
  <mutante> | 
  RT - grant global right to see system dashboards to privileged users | 
  [production] | 
            
  | 02:35 | 
  <LocalisationUpdate> | 
  ResourceLoader cache refresh completed at Wed Aug 21 02:35:19 UTC 2013 | 
  [production] | 
            
  | 02:28 | 
  <LocalisationUpdate> | 
  completed (1.22wmf12) at Wed Aug 21 02:28:33 UTC 2013 | 
  [production] | 
            
  | 02:15 | 
  <LocalisationUpdate> | 
  completed (1.22wmf13) at Wed Aug 21 02:15:12 UTC 2013 | 
  [production] | 
            
  | 01:03 | 
  <mwalker> | 
  synchronized php-1.22wmf12/extensions/CentralNotice/special  'Applying CentralNotice fix for bug 53032' | 
  [production] | 
            
  | 01:02 | 
  <mwalker> | 
  synchronized php-1.22wmf13/extensions/CentralNotice/special  'Applying CentralNotice fix for bug 53032' | 
  [production] | 
            
  | 00:58 | 
  <mutante> | 
  allow engineering and ops group members in RT to create new saved searches | 
  [production] | 
            
  | 00:11 | 
  <catrope> | 
  synchronized php-1.22wmf13/extensions/VisualEditor/modules/ve/ui/tools/buttons/ve.ui.UnderlineButtonTool.js  'touch' | 
  [production] | 
            
  | 00:04 | 
  <catrope> | 
  Finished syncing Wikimedia installation... : Updating VisualEditor to master | 
  [production] | 
            
  
    | 
      
        2013-08-20
      
      §
     | 
  
    
  | 23:56 | 
  <catrope> | 
  Started syncing Wikimedia installation... : Updating VisualEditor to master | 
  [production] | 
            
  | 23:47 | 
  <mutante> | 
  powercycling frozen mw1173 | 
  [production] | 
            
  | 23:45 | 
  <mutante> | 
  powercycling frozen mw1089 | 
  [production] | 
            
  | 23:42 | 
  <mutante> | 
  powercycling srv281 | 
  [production] | 
            
  | 23:38 | 
  <catrope> | 
  synchronized php-1.22wmf13/extensions/VisualEditor  'Deploy new VE code to wmf13 first for testing' | 
  [production] | 
            
  | 23:38 | 
  <RoanKattouw> | 
  Repeating previously aborted sync-dir of extensions/VisualEditor now that the dsh node list is fixed | 
  [production] | 
            
  | 23:37 | 
  <mutante> | 
  manually removed mw1046 from dsh groups mediawiki-installation,apaches,apaches-eqiad on tin | 
  [production] | 
            
  | 23:32 | 
  <RoanKattouw> | 
  Forcing puppet on tin to update dsh lists for mw1046 removal  | 
  [production] | 
            
  | 23:00 | 
  <ori-l> | 
  During sync-dir, SSH timeouts from srv281, mw1089, mw1173; rsync errors on mw1046 (RO fs; previously reported) | 
  [production] | 
            
  | 22:59 | 
  <olivneh> | 
  synchronized php-1.22wmf13/extensions/CoreEvents  'Updating CoreEvents to master (2/2)' | 
  [production] | 
            
  | 22:58 | 
  <olivneh> | 
  synchronized php-1.22wmf12/extensions/CoreEvents  'Updating CoreEvents to master (1/2)' | 
  [production] | 
            
  | 21:50 | 
  <maxsem> | 
  Finished syncing Wikimedia installation... : Weekly mobile deployment | 
  [production] | 
            
  | 21:25 | 
  <maxsem> | 
  Started syncing Wikimedia installation... : Weekly mobile deployment | 
  [production] | 
            
  | 20:28 | 
  <demon> | 
  synchronized wmf-config/InitialiseSettings.php  'test2wiki to secure login' | 
  [production] |