| 2013-08-21
      
      § | 
    
  | 22:31 | <binasher> | opened up wikiadmin grants on es1 shard to accommodate snapshot1004 | [production] | 
            
  | 20:59 | <mutante> | cp1061,cp1062 have dpkg issue upgrading varnish packages | [production] | 
            
  | 20:36 | <mutante> | installing package upgrades on iron | [production] | 
            
  | 20:35 | <mutante> | re-enabled puppet runs on zirconium which was "admin down" | [production] | 
            
  | 20:20 | <mutante> | dist-upgrading zirconium | [production] | 
            
  | 18:41 | <reedy> | synchronized wmf-config/InitialiseSettings.php  'Disable Collection on testwikidatawiki. It's mucho brokenp' | [production] | 
            
  | 18:38 | <reedy> | synchronized wmf-config/InitialiseSettings.php  'Enable Collection on testwikidatawiki' | [production] | 
            
  | 18:29 | <aaron> | synchronized wmf-config/CommonSettings.php  'Enabled $wgMWOAuthSecureTokenTransfer' | [production] | 
            
  | 16:36 | <reedy> | synchronized wmf-config/InitialiseSettings.php | [production] | 
            
  | 16:27 | <paravoid> | DNS migration complete | [production] | 
            
  | 16:21 | <paravoid> | switching ns2 to eeden | [production] | 
            
  | 16:04 | <reedy> | synchronized docroot and w | [production] | 
            
  | 15:51 | <paravoid> | switching ns0 to rubidium | [production] | 
            
  | 15:46 | <reedy> | synchronized wmf-config/ | [production] | 
            
  | 15:46 | <reedy> | synchronized wmf-config/ | [production] | 
            
  | 15:36 | <faidon> | synchronized wmf-config/filebackend.php  'revert ceph promotion to master' | [production] | 
            
  | 13:58 | <paravoid> | stopping ceph-osd 88 & 95 (ms-be1008), evidence of unexplainable high cpu usage | [production] | 
            
  | 13:38 | <paravoid> | switching ns1 traffic to mexia (new authdns) | [production] | 
            
  | 12:08 | <Reedy> | Running extensions/CentralAuth/maintenance/populateHomeDB.php in a screen on tin | [production] | 
            
  | 11:02 | <faidon> | synchronized wmf-config/filebackend.php  'promote ceph to master' | [production] | 
            
  | 08:01 | <nikerabbit> | synchronized php-1.22wmf13/extensions/UniversalLanguageSelector/  'ULS to master' | [production] | 
            
  | 02:46 | <mutante> | RT - create new system dashboard 'wikimedia default' that lists open ops-requests, quick ticket creation, reminders and bookmarked tickets | [production] | 
            
  | 02:46 | <mutante> | RT - grant global right to see system dashboards to privileged users | [production] | 
            
  | 02:35 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Wed Aug 21 02:35:19 UTC 2013 | [production] | 
            
  | 02:28 | <LocalisationUpdate> | completed (1.22wmf12) at Wed Aug 21 02:28:33 UTC 2013 | [production] | 
            
  | 02:15 | <LocalisationUpdate> | completed (1.22wmf13) at Wed Aug 21 02:15:12 UTC 2013 | [production] | 
            
  | 01:03 | <mwalker> | synchronized php-1.22wmf12/extensions/CentralNotice/special  'Applying CentralNotice fix for bug 53032' | [production] | 
            
  | 01:02 | <mwalker> | synchronized php-1.22wmf13/extensions/CentralNotice/special  'Applying CentralNotice fix for bug 53032' | [production] | 
            
  | 00:58 | <mutante> | allow engineering and ops group members in RT to create new saved searches | [production] | 
            
  | 00:11 | <catrope> | synchronized php-1.22wmf13/extensions/VisualEditor/modules/ve/ui/tools/buttons/ve.ui.UnderlineButtonTool.js  'touch' | [production] | 
            
  | 00:04 | <catrope> | Finished syncing Wikimedia installation... : Updating VisualEditor to master | [production] | 
            
  
    | 2013-08-20
      
      § | 
    
  | 23:56 | <catrope> | Started syncing Wikimedia installation... : Updating VisualEditor to master | [production] | 
            
  | 23:47 | <mutante> | powercycling frozen mw1173 | [production] | 
            
  | 23:45 | <mutante> | powercycling frozen mw1089 | [production] | 
            
  | 23:42 | <mutante> | powercycling srv281 | [production] | 
            
  | 23:38 | <catrope> | synchronized php-1.22wmf13/extensions/VisualEditor  'Deploy new VE code to wmf13 first for testing' | [production] | 
            
  | 23:38 | <RoanKattouw> | Repeating previously aborted sync-dir of extensions/VisualEditor now that the dsh node list is fixed | [production] | 
            
  | 23:37 | <mutante> | manually removed mw1046 from dsh groups mediawiki-installation,apaches,apaches-eqiad on tin | [production] | 
            
  | 23:32 | <RoanKattouw> | Forcing puppet on tin to update dsh lists for mw1046 removal | [production] | 
            
  | 23:00 | <ori-l> | During sync-dir, SSH timeouts from srv281, mw1089, mw1173; rsync errors on mw1046 (RO fs; previously reported) | [production] | 
            
  | 22:59 | <olivneh> | synchronized php-1.22wmf13/extensions/CoreEvents  'Updating CoreEvents to master (2/2)' | [production] | 
            
  | 22:58 | <olivneh> | synchronized php-1.22wmf12/extensions/CoreEvents  'Updating CoreEvents to master (1/2)' | [production] | 
            
  | 21:50 | <maxsem> | Finished syncing Wikimedia installation... : Weekly mobile deployment | [production] | 
            
  | 21:25 | <maxsem> | Started syncing Wikimedia installation... : Weekly mobile deployment | [production] | 
            
  | 20:28 | <demon> | synchronized wmf-config/InitialiseSettings.php  'test2wiki to secure login' | [production] | 
            
  | 20:24 | <demon> | Finished syncing Wikimedia installation... : | [production] | 
            
  | 20:02 | <binasher> | granted "lock tables" to eventlog user on db1047 | [production] | 
            
  | 20:02 | <demon> | Started syncing Wikimedia installation... : | [production] | 
            
  | 19:40 | <demon> | Started syncing Wikimedia installation... : | [production] | 
            
  | 19:27 | <anomie> | synchronized wmf-config/CommonSettings.php  'Fix OAuth rights assignments' | [production] |