| 2013-08-21
      
      § | 
    
  | 16:21 | <paravoid> | switching ns2 to eeden | [production] | 
            
  | 16:04 | <reedy> | synchronized docroot and w | [production] | 
            
  | 15:51 | <paravoid> | switching ns0 to rubidium | [production] | 
            
  | 15:46 | <reedy> | synchronized wmf-config/ | [production] | 
            
  | 15:46 | <reedy> | synchronized wmf-config/ | [production] | 
            
  | 15:36 | <faidon> | synchronized wmf-config/filebackend.php  'revert ceph promotion to master' | [production] | 
            
  | 13:58 | <paravoid> | stopping ceph-osd 88 & 95 (ms-be1008), evidence of unexplainable high cpu usage | [production] | 
            
  | 13:38 | <paravoid> | switching ns1 traffic to mexia (new authdns) | [production] | 
            
  | 12:08 | <Reedy> | Running extensions/CentralAuth/maintenance/populateHomeDB.php in a screen on tin | [production] | 
            
  | 11:02 | <faidon> | synchronized wmf-config/filebackend.php  'promote ceph to master' | [production] | 
            
  | 08:01 | <nikerabbit> | synchronized php-1.22wmf13/extensions/UniversalLanguageSelector/  'ULS to master' | [production] | 
            
  | 02:46 | <mutante> | RT - create new system dashboard 'wikimedia default' that lists open ops-requests, quick ticket creation, reminders and bookmarked tickets | [production] | 
            
  | 02:46 | <mutante> | RT - grant global right to see system dashboards to privileged users | [production] | 
            
  | 02:35 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Wed Aug 21 02:35:19 UTC 2013 | [production] | 
            
  | 02:28 | <LocalisationUpdate> | completed (1.22wmf12) at Wed Aug 21 02:28:33 UTC 2013 | [production] | 
            
  | 02:15 | <LocalisationUpdate> | completed (1.22wmf13) at Wed Aug 21 02:15:12 UTC 2013 | [production] | 
            
  | 01:03 | <mwalker> | synchronized php-1.22wmf12/extensions/CentralNotice/special  'Applying CentralNotice fix for bug 53032' | [production] | 
            
  | 01:02 | <mwalker> | synchronized php-1.22wmf13/extensions/CentralNotice/special  'Applying CentralNotice fix for bug 53032' | [production] | 
            
  | 00:58 | <mutante> | allow engineering and ops group members in RT to create new saved searches | [production] | 
            
  | 00:11 | <catrope> | synchronized php-1.22wmf13/extensions/VisualEditor/modules/ve/ui/tools/buttons/ve.ui.UnderlineButtonTool.js  'touch' | [production] | 
            
  | 00:04 | <catrope> | Finished syncing Wikimedia installation... : Updating VisualEditor to master | [production] | 
            
  
    | 2013-08-20
      
      § | 
    
  | 23:56 | <catrope> | Started syncing Wikimedia installation... : Updating VisualEditor to master | [production] | 
            
  | 23:47 | <mutante> | powercycling frozen mw1173 | [production] | 
            
  | 23:45 | <mutante> | powercycling frozen mw1089 | [production] | 
            
  | 23:42 | <mutante> | powercycling srv281 | [production] | 
            
  | 23:38 | <catrope> | synchronized php-1.22wmf13/extensions/VisualEditor  'Deploy new VE code to wmf13 first for testing' | [production] | 
            
  | 23:38 | <RoanKattouw> | Repeating previously aborted sync-dir of extensions/VisualEditor now that the dsh node list is fixed | [production] | 
            
  | 23:37 | <mutante> | manually removed mw1046 from dsh groups mediawiki-installation,apaches,apaches-eqiad on tin | [production] | 
            
  | 23:32 | <RoanKattouw> | Forcing puppet on tin to update dsh lists for mw1046 removal | [production] | 
            
  | 23:00 | <ori-l> | During sync-dir, SSH timeouts from srv281, mw1089, mw1173; rsync errors on mw1046 (RO fs; previously reported) | [production] | 
            
  | 22:59 | <olivneh> | synchronized php-1.22wmf13/extensions/CoreEvents  'Updating CoreEvents to master (2/2)' | [production] | 
            
  | 22:58 | <olivneh> | synchronized php-1.22wmf12/extensions/CoreEvents  'Updating CoreEvents to master (1/2)' | [production] | 
            
  | 21:50 | <maxsem> | Finished syncing Wikimedia installation... : Weekly mobile deployment | [production] | 
            
  | 21:25 | <maxsem> | Started syncing Wikimedia installation... : Weekly mobile deployment | [production] | 
            
  | 20:28 | <demon> | synchronized wmf-config/InitialiseSettings.php  'test2wiki to secure login' | [production] | 
            
  | 20:24 | <demon> | Finished syncing Wikimedia installation... : | [production] | 
            
  | 20:02 | <binasher> | granted "lock tables" to eventlog user on db1047 | [production] | 
            
  | 20:02 | <demon> | Started syncing Wikimedia installation... : | [production] | 
            
  | 19:40 | <demon> | Started syncing Wikimedia installation... : | [production] | 
            
  | 19:27 | <anomie> | synchronized wmf-config/CommonSettings.php  'Fix OAuth rights assignments' | [production] | 
            
  | 19:19 | <bsitu> | synchronized wmf-config/InitialiseSettings-labs.php  'Turn HTTPs on by default for beta' | [production] | 
            
  | 19:18 | <bsitu> | synchronized wmf-config/InitialiseSettings.php  'Enable Echo and Thanks on fr, hu, pt, pl and sv wiki' | [production] | 
            
  | 19:17 | <bsitu> | synchronized echowikis.dblist  'Add fr, hu, pt, pl and sv to Echo dblist' | [production] | 
            
  | 19:07 | <bsitu> | Finished syncing Wikimedia installation... : Update Echo to master | [production] | 
            
  | 18:35 | <bsitu> | Started syncing Wikimedia installation... : Update Echo to master | [production] | 
            
  | 18:25 | <gwicke> | updated Parsoid to f359548f04e739 | [production] | 
            
  | 14:27 | <paravoid> | starting multiple parallel swift->ceph copy jobs on terbium | [production] | 
            
  | 10:17 | <apergos> | reboot lvs1002 after dist-upgrade | [production] | 
            
  | 07:38 | <apergos> | reboot lvs1004 after dist-upgrade | [production] | 
            
  | 07:09 | <apergos> | rebooting lvs1001 to fix eth2.1003 ip addr, after misguided attempt to simply ifdown/ifup | [production] |