| 2010-07-20
      
      § | 
    
  | 14:16 | <catrope> | synchronized php-1.5/wmf-config/InitialiseSettings.php  'Change dbname noboardwiki -> noboard_chapterswikimedia' | [production] | 
            
  | 14:12 | <robh> | ran sync-common-all | [production] | 
            
  | 13:54 | <andrew> | synchronized php-1.5/wmf-config/InitialiseSettings.php  'communityapplications is too long for user_groups table' | [production] | 
            
  | 13:48 | <andrew> | synchronizing Wikimedia installation... Revision: 69504 | [production] | 
            
  | 13:48 | <Andrew> | scapping to deploy CommunityApplications extension | [production] | 
            
  | 13:47 | <andrew> | synchronized php-1.5/wmf-config/InitialiseSettings.php  'New group for viewing community applications' | [production] | 
            
  | 13:42 | <andrew> | synchronized php-1.5/wmf-config/InitialiseSettings.php  'New group for viewing community applications' | [production] | 
            
  | 13:32 | <robh> | ran sync-common-all | [production] | 
            
  | 12:21 | <andrew> | synchronized php-1.5/extensions/CommunityHiring/SpecialCommunityHiring.php  'r69606' | [production] | 
            
  | 12:20 | <andrew> | synchronized php-1.5/extensions/CommunityHiring/CommunityHiring.php | [production] | 
            
  | 12:20 | <andrew> | synchronized php-1.5/wmf-config/CommonSettings.php  'Configuration changes for r69606' | [production] | 
            
  | 11:53 | <Andrew> | Adding CommunityHiring tables to officewiki database | [production] | 
            
  
    | 2010-07-19
      
      § | 
    
  | 20:05 | <robh> | synchronized php-1.5/wmf-config/InitialiseSettings.php  'Bug 24448 - w:cs: upload settings' | [production] | 
            
  | 18:58 | <aZaFred_OSCON> | started slave on db10 (sync had stopped on July 8th) | [production] | 
            
  | 16:59 | <Rob> | upded tfinc email quota cuz i was tired of seeing the alerts to postmaster | [production] | 
            
  | 13:09 | <catrope> | synchronized php-1.5/wmf-config/InitialiseSettings.php  'bug 24440: New namespaces for dawikisource' | [production] | 
            
  | 10:20 | <midom> | synchronized wmf-deployment/cache/trusted-xff.cdb | [production] | 
            
  | 08:32 | <catrope> | synchronized php-1.5/wmf-config/InitialiseSettings.php  'bug 24435: Enable $wgBlockAllowsUTEdit on hiwiki' | [production] | 
            
  | 07:32 | <Tim> | changing max retry count on NTP nagios monitoring to avoid constant flapping | [production] | 
            
  | 06:45 | <Tim> | restarted ntpd on mchenry, wasn't responding on IPv4 | [production] | 
            
  
    | 2010-07-18
      
      § | 
    
  | 21:26 | <domas> | fixed opensearch caching, unbreaked API | [production] | 
            
  | 21:18 | <midom> | synchronized php-1.5/includes/api/ApiMain.php  'fix broken API caching' | [production] | 
            
  | 15:10 | <mark> | synchronizing Wikimedia installation... Revision: 69504 | [production] | 
            
  | 14:59 | <mark> | Making puppet upgrade wikimedia-raid-utils on all servers | [production] | 
            
  | 12:58 | <catrope> | synchronized php-1.5/extensions/WikimediaMessages/WikimediaLicenseTexts.i18n.php  'r69504' | [production] | 
            
  | 12:51 | <catrope> | synchronized php-1.5/wmf-config/InitialiseSettings.php  'New groups for hiwiki (bugs 24416, 24417, 24418, 24419)' | [production] | 
            
  | 08:13 | <midom> | synchronized php-1.5/wmf-config/db.php  'db13 and db15 go live' | [production] | 
            
  
    | 2010-07-17
      
      § | 
    
  | 22:01 | <mark> | Apparently package wikimedia-raid-utils truncated /etc/sudoers on many hosts; having puppet put a proper sudoers file back on the application servers | [production] | 
            
  | 21:42 | <mark> | Fixed gmond mess on search12 | [production] | 
            
  | 21:03 | <RoanKattouw> | Importing another batch of ~10,500 files (~22GB) concurrently with the first one | [production] | 
            
  | 19:38 | <RoanKattouw> | Import is running in a screen on fenari | [production] | 
            
  | 19:38 | <RoanKattouw> | Importing ~10,500 files (aggregate size ~21GB) with importImages.php for [[commons:User:OrdnanceSurveyBot]] | [production] | 
            
  | 18:49 | <catrope> | synchronized php-1.5/languages/messages/MessagesLad.php  'r69487' | [production] | 
            
  | 18:48 | <mark> | Having puppet install wikimedia-raid-utils on all servers | [production] | 
            
  | 18:12 | <mark> | srv278 rebooted for no apparent reason | [production] | 
            
  | 17:21 | <mark> | Fixed ganglia mess on search19 | [production] | 
            
  | 16:19 | <mark> | Having puppet install NRPE on all internal servers by default | [production] | 
            
  | 15:52 | <root> | synchronized php-1.5/wmf-config/lucene.php  'Use fixed LVS service search-pool3 instead of search7 directly' | [production] | 
            
  | 15:31 | <mark> | Removed fscking nscd from spence | [production] | 
            
  | 15:20 | <mark> | Added internal LVS services to pmtpa.wmnet DNS | [production] | 
            
  | 15:11 | <mark> | Doing reboot test of nescio (esams DNS) | [production] | 
            
  | 15:08 | <mark> | Fixed puppet exchange of ssh hostkeys which has been broken for a while | [production] | 
            
  | 14:30 | <mark> | Truncated all tables in the puppet db on db9 | [production] | 
            
  | 12:30 | <mark> | Implemented domain search list in resolv.conf for fenari (use sparingly!) | [production] | 
            
  | 12:24 | <mark> | Put /etc/resolv.conf under Puppet management on all servers. Setting timeout option to 3s, to avoid PyBal depools due to 5s timeout when the primary resolver is down. | [production] | 
            
  | 11:38 | <mark> | Removed rkhunter and chkrootkit on bayes. What is the point with 2 year old software? Just creating more cron spam? :) | [production] | 
            
  | 10:06 | <mark> | Fixed degraded raid on nescio | [production] | 
            
  | 09:53 | <mark> | Removed X.org and gdm from bayes. Why was it installed? | [production] | 
            
  | 09:30 | <domas> | doing maintenance on db13 and db15 (BIOS, OS, MySQL upgrades, resync of data) | [production] |