| 2010-05-28
      
      § | 
    
  | 16:54 | <jeluf> | synchronized php-1.5/wmf-config/InitialiseSettings.php  '23486 - Enable Collection on gdwiki' | [production] | 
            
  | 13:35 | <andrew> | synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php | [production] | 
            
  | 13:30 | <andrew> | synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ParserFunctions.php  'Missing svn add' | [production] | 
            
  | 13:27 | <andrew> | synchronizing Wikimedia installation... Revision: 66620 | [production] | 
            
  | 13:27 | <Andrew> | running scap to deploy strategywiki extension updates | [production] | 
            
  | 13:23 | <andrew> | synchronized php-1.5/includes/specials/SpecialRevisiondelete.php | [production] | 
            
  | 09:00 | <midom> | synchronized php-1.5/wmf-config/db.php  'moving over db16 to commons duty, db14 to dewiki, reenabling db4 and db5' | [production] | 
            
  | 07:53 | <midom> | synchronized php-1.5/wmf-config/db.php  'removing db5 and db4 for copies' | [production] | 
            
  | 07:11 | <tstarling> | synchronized php-1.5/includes/Sanitizer.php  'r66994' | [production] | 
            
  | 06:31 | <domas> | upgrading LOM/BIOS/OS on db14,db16,db22 to 3.0/10.4 | [production] | 
            
  | 02:41 | <Tim> | srv92 locked up for a few minutes with syslog showing SATA errors. May be near death. | [production] | 
            
  | 01:17 | <tstarling> | synchronized php-1.5/wmf-config/CommonSettings.php  're-enabled CategoryTree, unless there is some emergency, you can't just disable extensions like this' | [production] | 
            
  
    | 2010-05-27
      
      § | 
    
  | 21:35 | <catrope> | synchronized php-1.5/wmf-config/InitialiseSettings.php  'Disable CollapsibleNav on outreachwiki' | [production] | 
            
  | 21:22 | <aaron> | synchronized php-1.5/wmf-config/flaggedrevs.php | [production] | 
            
  | 21:09 | <AaronSchulz> | deployed r66980 (Flaggedrevs_alpha) | [production] | 
            
  | 21:07 | <aaron> | synchronizing Wikimedia installation... Revision: 66620 | [production] | 
            
  | 20:24 | <catrope> | synchronized php-1.5/wmf-config/InitialiseSettings.php  'Enable Vector and usability features on outreachwiki and foundationwiki' | [production] | 
            
  | 20:17 | <catrope> | synchronized php-1.5/wmf-config/InitialiseSettings.php  'Enable CollapsibleNav for beta users on all wikis' | [production] | 
            
  | 20:17 | <catrope> | synchronized php-1.5/wmf-config/CommonSettings.php  'Enable SimpleSearch for beta users on all wikis' | [production] | 
            
  | 11:59 | <domas> | opened a new sub-categories related bug at https://bugzilla.wikimedia.org/show_bug.cgi?id=23682 ;-) | [production] | 
            
  | 11:30 | <midom> | synchronized php-1.5/wmf-config/CommonSettings.php  'disable category tree' | [production] | 
            
  | 04:13 | <aaron> | synchronized php-1.5/wmf-config/InitialiseSettings.php  'vector as default skin for flaggedrevs_labswikimeida' | [production] | 
            
  | 02:40 | <Tim> | ran namespaceDupes.php with suffix on jawiki to fix talk namespace duplicates | [production] | 
            
  
    | 2010-05-26
      
      § | 
    
  | 23:59 | <Rob> | updated blog to vector earlier today, didnt log... just pushed another update to the theme, blog is still happah | [production] | 
            
  | 21:43 | <robh> | synchronized php-1.5/wmf-config/InitialiseSettings.php  'updated foundationwiki default skin from monobook to vector' | [production] | 
            
  | 17:02 | <Tim> | on mobile1 nginx: set proxy_next_upstream=off to avoid having the site go down regularly for 10s at a time | [production] | 
            
  | 16:30 | <Tim> | changing mobile1 nginx log format | [production] | 
            
  | 06:39 | <jeluf> | synchronized php-1.5/wmf-config/InitialiseSettings.php  '23667 - Local time for Macedonian Wikipedia' | [production] | 
            
  | 03:39 | <Tim> | set up SSLCertificateChainFile on williams (otrs) per bug 23631, used the cut down version of gd_bundle.crt posted on the bug report | [production] | 
            
  | 02:08 | <andrew> | synchronized php-1.5/includes/LogEventsList.php | [production] | 
            
  | 02:07 | <andrew> | synchronized php-1.5/includes/specials/SpecialRevisiondelete.php | [production] | 
            
  | 02:07 | <andrew> | synchronized php-1.5/languages/messages/MessagesEn.php | [production] | 
            
  | 02:05 | <Andrew> | Deploying r66793 and r66823, first to test.wikipedia.org for final testing, then to all sites. | [production] | 
            
  
    | 2010-05-25
      
      § | 
    
  | 17:02 | <catrope> | synchronized php-1.5/wmf-config/CommonSettings.php  'Re-add style version appendix now that borked usability JS was purged from Varnish' | [production] | 
            
  | 12:51 | <domas> | what happened - server goes down, configuration gets updated, server comes up, builds new cache out of old conf, sync has timestamp older than cache, mw has stale config | [production] | 
            
  | 12:47 | <domas> | cleaned /tmp/mw-cache for some servers - apparently sync-common is not enough to get stuff in sync anymore :) we have now monobook pages in caches, I guess. | [production] | 
            
  | 12:26 | <rainman-sr> | freed up some space and restarted indexing on searchidx1, indexing was stuck because the disk was full | [production] | 
            
  | 11:16 | <mark> | Downgraded puppet on db36 | [production] | 
            
  | 06:56 | <domas> | resynced srv206/srv193 - seems to have lost critical configuration updates while they were down. Again, why servers are not resynced after downtime? | [production] | 
            
  | 01:24 | <Tim> | fixed redirect from http://racktables to https://racktables, was broken due to IP address change | [production] | 
            
  | 01:15 | <tstarling> | synchronized php-1.5/wmf-config/CommonSettings.php  'removed cruft from wgStyleVersion' | [production] | 
            
  | 01:12 | <tstarling> | synchronized php-1.5/skins/common/wikibits.js | [production] | 
            
  | 00:19 | <aaron> | synchronized php-1.5/wmf-config/flaggedrevs.php  'removed Editor group from flaggedrevs_labswikimedia' | [production] | 
            
  | 00:02 | <AaronSchulz> | unhide the username of edits by "The Thing That Should Not Be" after accidental account locking (and unlock didn't work) | [production] | 
            
  
    | 2010-05-24
      
      § | 
    
  | 15:01 | <Rob> | sq39 will stay offline for a bit, have a configuration concern with replacement disk performance. | [production] | 
            
  | 15:01 | <Rob> | pushed updates to srv206 and srv193, they should be back online and ok now. | [production] | 
            
  | 14:54 | <Rob> | sq39 has disk replaced and reinstalled, but its failing squid backend due to configuration.  pulling out of rotation to look at later today. | [production] | 
            
  | 14:51 | <Rob> | working on srv206 and srv193 | [production] | 
            
  | 14:21 | <Rob> | replaced bad disk in sq39, reinstalling. | [production] | 
            
  | 12:16 | <domas> | db32, db36, db38 and db40 are now serving enwiki (+ db12 and db26 ) | [production] |