| 
      
        2012-08-15
      
      §
     | 
  
    
  | 20:57 | 
  <K4-713> | 
  synchronized payments cluster to 79275e152b76f | 
  [production] | 
            
  | 20:26 | 
  <ottomata> | 
  staring puppet on brewster.  PARTMAN IS THE ENEMY.  will continue the fight tomorrow. | 
  [production] | 
            
  | 20:22 | 
  <mutante> | 
  http://en.wikipedia.org/w/ now redirects to Main_page on all languages. It was a 403 before. | 
  [production] | 
            
  | 20:20 | 
  <mutante> | 
  apache-graceful-all | 
  [production] | 
            
  | 20:15 | 
  <mutante> | 
  sync-apache pushing out new redirects from /w/ to Main_Page | 
  [production] | 
            
  | 20:04 | 
  <ottomata> | 
  stopping puppet on brewster to experiment with partman on new analytics dells | 
  [production] | 
            
  | 19:29 | 
  <mark> | 
  Removed OSPF/OSPFv3 metric 60 on cr1-sdtpa:xe-1/1/0 (eqiad link) | 
  [production] | 
            
  | 19:17 | 
  <mark> | 
  Moved VRRP mastership from cr2-pmtpa to cr1-sdtpa by reassigning VRRP priorities, relieving the sdtpa-pmtpa link | 
  [production] | 
            
  | 18:44 | 
  <reedy> | 
  rebuilt wikiversions.cdb and synchronized wikiversions files: Remaining wikis over to 1.20wmf9 | 
  [production] | 
            
  | 17:11 | 
  <maplebed> | 
  ms-be1 dropped offline yesterday; powercycling. | 
  [production] | 
            
  | 16:11 | 
  <mark> | 
  Setup SnapMirror replication from nas1-a:fr_archive to nas1001-a:fr_archive | 
  [production] | 
            
  | 16:10 | 
  <mark> | 
  Created 4 TB volumes 'fr_archive' in aggregate prod1 on both nas1-a nad nas1001-a | 
  [production] | 
            
  | 16:10 | 
  <mark> | 
  Increased aggregate 'prod1' by 4 drives on both nas1-a and nas1001-a | 
  [production] | 
            
  | 16:06 | 
  <LocalisationUpdate> | 
  completed (1.20wmf8) at Wed Aug 15 16:06:26 UTC 2012 | 
  [production] | 
            
  | 15:40 | 
  <LocalisationUpdate> | 
  completed (1.20wmf9) at Wed Aug 15 15:40:21 UTC 2012 | 
  [production] | 
            
  | 14:33 | 
  <cmjohnson1> | 
  authdns-update for es9 and es10 | 
  [production] | 
            
  | 13:46 | 
  <RobH> | 
  authdns-update for wikipedia.co.za | 
  [production] | 
            
  | 12:15 | 
  <mark> | 
  Started rsync of /home to nas1-a:/vol/home_pmtpa on nfs1 | 
  [production] | 
            
  | 12:11 | 
  <mark> | 
  Restricted NFS mounting of /vol/root on all NetApps | 
  [production] | 
            
  | 11:39 | 
  <mark> | 
  Destroyed aggregate 'labs' on nas1-a | 
  [production] | 
            
  | 10:00 | 
  <Tim> | 
  deploying bugfix version of wmerrors | 
  [production] | 
            
  | 09:51 | 
  <Tim> | 
  moved old MW debug logs out of /home/wikipedia/logs to avoid confusion, live logs are now on fluorine | 
  [production] | 
            
  | 05:22 | 
  <Tim> | 
  moving MediaWiki logs to fluorine | 
  [production] | 
            
  | 05:17 | 
  <tstarling> | 
  synchronized wmf-config/InitialiseSettings.php  | 
  [production] | 
            
  | 05:15 | 
  <tstarling> | 
  synchronized wmf-config/CommonSettings.php  | 
  [production] | 
            
  | 04:38 | 
  <Tim> | 
  added rate limit exemption for 27.108.200.111 | 
  [production] | 
            
  | 04:38 | 
  <tstarling> | 
  synchronized wmf-config/InitialiseSettings.php  | 
  [production] | 
            
  | 03:00 | 
  <Tim> | 
  built a udplog package for precise and uploaded it | 
  [production] | 
            
  | 02:35 | 
  <tstarling> | 
  synchronized wmf-config/CommonSettings.php  'enabling Scribunto and CodeEditor on test2wiki' | 
  [production] | 
            
  | 02:28 | 
  <tstarling> | 
  synchronized wmf-config/InitialiseSettings.php  | 
  [production] | 
            
  | 02:15 | 
  <tstarling> | 
  Finished syncing Wikimedia installation... :  | 
  [production] | 
            
  | 02:01 | 
  <LocalisationUpdate> | 
  failed: git pull of extensions failed | 
  [production] | 
            
  | 01:46 | 
  <tstarling> | 
  Started syncing Wikimedia installation... :  | 
  [production] | 
            
  | 01:00 | 
  <tstarling> | 
  Started syncing Wikimedia installation... :  | 
  [production] | 
            
  | 00:47 | 
  <tstarling> | 
  Started syncing Wikimedia installation... :  | 
  [production] | 
            
  | 00:39 | 
  <Tim> | 
  running scap to for extension-list update | 
  [production] | 
            
  
    | 
      
        2012-08-14
      
      §
     | 
  
    
  | 23:53 | 
  <catrope> | 
  synchronized wmf-config/CommonSettings.php  'Move VE Parsoid from cadmium to wtp1' | 
  [production] | 
            
  | 21:25 | 
  <kaldari> | 
  synchronized php-1.20wmf9/extensions/PageTriage/tools/cleanupPageTriage.php  'adding PageTriage maintainence script' | 
  [production] | 
            
  | 21:05 | 
  <kaldari> | 
  Finished syncing Wikimedia installation... :  | 
  [production] | 
            
  | 20:45 | 
  <kaldari> | 
  Started syncing Wikimedia installation... :  | 
  [production] | 
            
  | 20:38 | 
  <mutante> | 
  killed duplicate nagios-wm | 
  [production] | 
            
  | 20:20 | 
  <cmjohnson1> | 
  db44 swapping out disk1 rt 3424 | 
  [production] | 
            
  | 20:08 | 
  <ottomata> | 
  installed precise and puppetized stat1001 | 
  [production] | 
            
  | 19:26 | 
  <cmjohnson1> | 
  search23 shutting down to replace DIMM A2 rt 3423 | 
  [production] | 
            
  | 19:00 | 
  <catrope> | 
  synchronized wmf-config/InitialiseSettings.php  'Deploying https://gerrit.wikimedia.org/r/19399' | 
  [production] | 
            
  | 18:52 | 
  <RoanKattouw> | 
  Running /root/fixownership < /root/badownershipfiles in a screen on ms7 | 
  [production] | 
            
  | 18:14 | 
  <notpeter> | 
  stopping puppet on brewster for partman experimentation | 
  [production] | 
            
  | 17:34 | 
  <catrope> | 
  synchronized php-1.20wmf9/.gitmodules  | 
  [production] | 
            
  | 17:33 | 
  <RoanKattouw> | 
  Yesterday's find on ms7 is done, found 1.8 million files with bad ownership. Will run a batch chown on them soon | 
  [production] | 
            
  | 17:32 | 
  <catrope> | 
  synchronized php-1.20wmf9/extensions/UploadWizard  'Deploy 9d9189458ad8d966d952d295918847f00e304cb5' | 
  [production] |