| 
      
        2010-07-07
      
      §
     | 
  
    
  | 21:38 | 
  <Fred> | 
  RIP sfoservices. (box not booting at all anymore) | 
  [production] | 
            
  | 17:10 | 
  <Fred> | 
  re-imaging srv225 to the apache cluster. | 
  [production] | 
            
  | 16:26 | 
  <mark> | 
  Fixed puppet on srv193 | 
  [production] | 
            
  | 15:56 | 
  <mark> | 
  Fixed horrible gmond mess on searchidx1 | 
  [production] | 
            
  | 15:42 | 
  <mark> | 
  Fixed puppet on sr255 | 
  [production] | 
            
  | 15:30 | 
  <mark> | 
  Mounted /mnt/upload6 on srv255 | 
  [production] | 
            
  | 14:23 | 
  <mark> | 
  Fixed /home backup on nfs1/nfs2 to tridge | 
  [production] | 
            
  | 07:57 | 
  <Rob> | 
  srv193 is refusing to take my updates, removed it from pybal so it doesnt serve out of data information | 
  [production] | 
            
  | 07:53 | 
  <robh> | 
  ran sync-common-all  | 
  [production] | 
            
  | 07:50 | 
  <Rob> | 
  updated dns for wikimania wiki | 
  [production] | 
            
  | 07:38 | 
  <Rob> | 
  adding wikimaniawiki apache support, sycning lots of apaches and docroots. | 
  [production] | 
            
  | 01:49 | 
  <Tim> | 
  downloading the DjVu files via rsync/ssh for http://www.wikimedia.fr/wikim%C3%A9dia-france-signe-un-partenariat-avec-la-bnf | 
  [production] | 
            
  
    | 
      
        2010-07-05
      
      §
     | 
  
    
  | 23:55 | 
  <tstarling> | 
  synchronized php-1.5/wmf-config/CommonSettings.php  'new URL for codereview proxy' | 
  [production] | 
            
  | 23:33 | 
  <Tim> | 
  changed CNAME for codereview-proxy to kaulen | 
  [production] | 
            
  | 23:20 | 
  <Tim> | 
  moving codereview-proxy to kaulen to replace isidore (which is down) | 
  [production] | 
            
  | 22:53 | 
  <Tim> | 
  on srv124: remounted /home to fix test.wikipedia.org | 
  [production] | 
            
  | 18:56 | 
  <jeluf> | 
  synchronized php-1.5/wmf-config/flaggedrevs.php  '24010 -  id.wikipedia requesting FlaggedRevs' | 
  [production] | 
            
  | 17:01 | 
  <mark> | 
  Did BGP soft clear outbound on all AMS-IX sessions; no prefixes were being announced as of two weeks ago | 
  [production] | 
            
  | 16:01 | 
  <mark> | 
  Made puppet ensure apache is running on the app servers; running "sync-common" upon start | 
  [production] | 
            
  | 15:38 | 
  <mark> | 
  Fixed puppet on srv145 | 
  [production] | 
            
  | 11:39 | 
  <mark> | 
  Remounted /home on hume | 
  [production] | 
            
  | 11:30 | 
  <root> | 
  synchronizing Wikimedia installation... Revision: 68850 | 
  [production] | 
            
  | 11:29 | 
  <mark> | 
  synchronized php-1.5/wmf-config/CommonSettings.php  'CommonSettings.php out of sync on a few apaches' | 
  [production] | 
            
  | 09:17 | 
  <jeluf> | 
  synchronized php-1.5/wmf-config/InitialiseSettings.php  '24264 - Create a namespace aliases on zhwiki' | 
  [production] | 
            
  | 06:26 | 
  <ronabop> | 
  Kudos to the team who rebuilt a multi-hundred node system under extreme pressure. | 
  [production] | 
            
  | 06:24 | 
  <Tim> | 
  fixed broken ircd auth configuration, irc.wikimedia.org now working again | 
  [production] | 
            
  | 05:13 | 
  <Tim> | 
  on browne: reinstalled udprec to fix IRC server | 
  [production] | 
            
  | 04:16 | 
  <Tim> | 
  switched nagios monitoring for search to less flappy TCP connection check instead of HTTP | 
  [production] | 
            
  | 03:58 | 
  <domas> | 
  s3 pos: db17-bin.321:0 | 
  [production] | 
            
  | 03:58 | 
  <midom> | 
  synchronized php-1.5/wmf-config/db.php  's3 rw' | 
  [production] | 
            
  | 03:54 | 
  <Tim> | 
  fixed search monitoring in nagios | 
  [production] |