| 
      
        2010-11-09
      
      §
     | 
  
    
  | 17:15 | 
  <robh> | 
  synchronized php-1.5/wmf-config/mc.php  'srv230 unresponsive to ssh, needs to reboot, swapped it out for working spare' | 
  [production] | 
            
  | 15:51 | 
  <RobH> | 
  removed srv* under srv151 from pybal, left entry for srv124 as its test.w.o, even though its set to false | 
  [production] | 
            
  | 15:46 | 
  <RobH> | 
  srv229 puppet was hanging, manually ran apt-get update and reran puppet, now its happy | 
  [production] | 
            
  | 15:22 | 
  <RobH> | 
  srv229 rebooted, wouldnt let me ssh in, coming back up now with puppet run | 
  [production] | 
            
  | 15:21 | 
  <robh> | 
  synchronized php-1.5/wmf-config/mc.php  'srv281 is not behaving, swapped it out' | 
  [production] | 
            
  | 15:17 | 
  <robh> | 
  synchronized php-1.5/wmf-config/mc.php  'replacing a server that i am working on' | 
  [production] | 
            
  | 15:01 | 
  <mark> | 
  Reduced CARP weight of new amssq squids from 30 to 20, until they get SSDs | 
  [production] | 
            
  | 12:16 | 
  <mark> | 
  Started copy of ms2:/a/ex-fedora data to tridge | 
  [production] | 
            
  | 12:15 | 
  <mark> | 
  Included class base for server tridge in Puppet | 
  [production] | 
            
  | 12:09 | 
  <mark> | 
  Cleaned up temporary files on image scalers | 
  [production] | 
            
  | 06:33 | 
  <apergos> | 
  cleared out some old bin log files on db27 to get back some space | 
  [production] | 
            
  | 05:20 | 
  <apergos> | 
  reboot of ms4 successful, we need to monitor performance over the next several days. If people are still seeing 503's for thumbs that's a problem | 
  [production] | 
            
  | 05:09 | 
  <apergos> | 
  er, ms4!! | 
  [production] | 
            
  | 05:09 | 
  <apergos> | 
  rebooting ms5 into alternate boot environ, now with new improved patches (:-P), let's see if it works | 
  [production] | 
            
  | 01:00 | 
  <tfinc> | 
  synchronized php-1.5/wmf-config/CommonSettings.php  'Bumping style version for meta udp2log fix' | 
  [production] | 
            
  
    | 
      
        2010-11-08
      
      §
     | 
  
    
  | 23:50 | 
  <robh> | 
  synchronized php-1.5/wmf-config/InitialiseSettings.php  'updating tenwiki logo' | 
  [production] | 
            
  | 23:18 | 
  <tstarling> | 
  ran sync-common-all  | 
  [production] | 
            
  | 22:32 | 
  <jeluf> | 
  synchronized php-1.5/wmf-config/InitialiseSettings.php  '25801 - New logo for et.wikimedia.org' | 
  [production] | 
            
  | 22:27 | 
  <jeluf> | 
  synchronized php-1.5/wmf-config/InitialiseSettings.php  '25742 Please set the Buryat Wikipedia logo' | 
  [production] | 
            
  | 22:25 | 
  <jeluf> | 
  synchronized php-1.5/wmf-config/InitialiseSettings.php  '25742 Please set the Buryat Wikipedia logo' | 
  [production] | 
            
  | 22:20 | 
  <jeluf> | 
  synchronized php-1.5/wmf-config/InitialiseSettings.php  '25836 - Commons' | 
  [production] | 
            
  | 22:13 | 
  <jeluf> | 
  synchronized php-1.5/wmf-config/InitialiseSettings.php  '25779 - Create new namespace "Institution"/"Museum" for Commons' | 
  [production] | 
            
  | 21:55 | 
  <JeLuF> | 
  Fixed typo: Replaced the second "srv290" in node group "apaches" by srv296. | 
  [production] | 
            
  | 21:10 | 
  <jeluf> | 
  synchronizing Wikimedia installation... Revision: 76208 | 
  [production] | 
            
  | 21:10 | 
  <JeLuF> | 
  srv154 didn't receive any updates in the last few days, it was missing in the mediawiki-installation nodegroup | 
  [production] | 
            
  | 20:19 | 
  <mark> | 
  Moved uplink of asw-b-sdtpa from temporary GigE link csw1-sdtpa:3/48 to 2x 10G (LACP) links csw1-sdtpa:16/2 and 16/3, shutdown 3/48 | 
  [production] | 
            
  | 19:44 | 
  <nimishg> | 
  synchronized php-1.5/extensions/LandingCheck/SpecialLandingCheck.php  'r76156' | 
  [production] | 
            
  | 19:44 | 
  <nimishg> | 
  synchronized php-1.5/extensions/LandingCheck/LandingCheck.php  'r76156' | 
  [production] | 
            
  | 19:22 | 
  <RobH> | 
  updated dns, seems it borked, but pdns is running on nescio so it should clear up | 
  [production] | 
            
  | 17:47 | 
  <robh> | 
  synchronized php-1.5/wmf-config/abusefilter.php  'bug' | 
  [production] | 
            
  | 16:38 | 
  <RobH> | 
  esams squid flapping on text squids is due to disk i/o use, they will be replaced with SSD soon | 
  [production] | 
            
  | 06:04 | 
  <apergos> | 
  brought ms4 back up in primary boot environment, testing concluded for tonight, results being sent to Oracle  | 
  [production] | 
            
  | 05:50 | 
  <apergos> | 
  doing reboot of ms4 into alternate boot environ for testing | 
  [production] | 
            
  | 02:34 | 
  <apergos> | 
  restarted torrus, it was out to lunch again | 
  [production] | 
            
  
    | 
      
        2010-11-07
      
      §
     | 
  
    
  | 17:39 | 
  <RoanKattouw> | 
  Starting makeArbcomList.php in a screen on fenari. Tim says this'll take about a day | 
  [production] | 
            
  | 12:43 | 
  <catrope> | 
  synchronized php-1.5/includes/Profiler.php  'r76243' | 
  [production] | 
            
  | 12:04 | 
  <catrope> | 
  synchronized php-1.5/wmf-config/abusefilter.php  'More perm changes for frwiktionary' | 
  [production] | 
            
  | 11:59 | 
  <catrope> | 
  synchronized php-1.5/wmf-config/abusefilter.php  'More perm changes for frwiktionary' | 
  [production] | 
            
  | 11:56 | 
  <catrope> | 
  synchronized php-1.5/wmf-config/abusefilter.php  'More perm changes for frwiktionary' | 
  [production] | 
            
  | 11:36 | 
  <catrope> | 
  synchronized php-1.5/includes/Profiler.php  | 
  [production] | 
            
  | 11:33 | 
  <catrope> | 
  synchronized php-1.5/wmf-config/InitialiseSettings.php  'bug 25711 - enable AbuseFilter on frwiktionary' | 
  [production] | 
            
  | 11:33 | 
  <catrope> | 
  synchronized php-1.5/wmf-config/abusefilter.php  'bug 25711 - enable AbuseFilter on frwiktionary' | 
  [production] | 
            
  | 11:17 | 
  <catrope> | 
  synchronized php-1.5/includes/Profiler.php  'Temp hack to debug fatals in Profiler.php' | 
  [production] | 
            
  | 10:48 | 
  <catrope> | 
  ran sync-common-all  | 
  [production] | 
            
  | 10:47 | 
  <RoanKattouw> | 
  Enabling FlaggedRevs on sqwiki per bug 25822 and disabling new page patrolling | 
  [production] | 
            
  | 10:16 | 
  <RoanKattouw> | 
  srv230 SSH is broken from fenari, Nagios disagrees. Commenting out srv230 from /etc/dsh/group/mediawiki-installation . After fixing srv230, uncomment it and resync the box | 
  [production] | 
            
  | 10:13 | 
  <catrope> | 
  synchronized php-1.5/wmf-config/InitialiseSettings.php  'bug 25674 - Enable $wgBlockAllowsUIEdit on frwiktionary' | 
  [production] |