| 2009-06-26
      
      § | 
    
  | 20:33 | <Rob> | blog.wikimedia.org back online, very restrictive settings and improved security (i hope) | [production] | 
            
  | 19:09 | <Rob> | singer https services back online for survey. ocs, wm09schols | [production] | 
            
  | 18:20 | <Rob> | Singer Restore: survey.wikimedia.org - UP, ocs.wikimania2009.wikimedia.org - UP | [production] | 
            
  | 17:11 | <Rob> | pulled blog.wikimedia.org out of squid via changing its dns to point directly at singer.  just to make it easier to secure and fix on the fly later today | [production] | 
            
  | 15:03 | <Rob> | firing singer back up for reinstall and restoration from the haxor | [production] | 
            
  | 14:42 | <midom> | synchronized php-1.5/includes/parser/ParserCache.php  'removing the pcache hack for now' | [production] | 
            
  | 11:44 | <domas> | halted singer, don't start it without contacting me. | [production] | 
            
  | 07:27 | <brion> | synchronized php-1.5/includes/parser/ParserCache.php  'Putting live hack for MJ article back. Total traffic spike down but article traffic still spiking.' | [production] | 
            
  | 05:53 | <Tim> | rebooting srv182, swap death since ~00:10 UTC | [production] | 
            
  | 05:50 | <Tim> | killed broken mysqld_safe instances on srv157, srv161, srv167, srv173, srv174, srv175, srv181, srv185, srv186 | [production] | 
            
  | 05:17 | <jeluf> | synchronized php-1.5/mc-pmtpa.php  'replace srv113 by srv142' | [production] | 
            
  | 05:14 | <jeluf> | synchronized php-1.5/mc-pmtpa.php  'replace srv182 by srv92' | [production] | 
            
  
    | 2009-06-25
      
      § | 
    
  | 23:36 | <brion> | synchronized php-1.5/includes/parser/ParserCache.php  'live hack to extend caching of Michael Jackson' | [production] | 
            
  | 23:17 | <brion> | synchronized php-1.5/includes/db/Database.php  'tweak the db err msg' | [production] | 
            
  | 22:46 | <brion> | synchronized php-1.5/mc-pmtpa.php  'putting 156 mc back' | [production] | 
            
  | 22:44 | <brion> | synchronized php-1.5/mc-pmtpa.php  'swapping in 145 in place of temp down 156' | [production] | 
            
  | 22:43 | <tomaszf> | rebooting srv156 due to hard down | [production] | 
            
  | 22:42 | <midom> | synchronized php-1.5/db.php  '156 down' | [production] | 
            
  | 18:26 | <RobH_A90> | srv92 back up with replaced fans | [production] | 
            
  | 18:16 | <RobH_A90> | srv92 down due to bad fan, going to replace it with another dead server fan, yay for frankenstien servers | [production] | 
            
  | 18:09 | <RobH_A90> | db17 has had a cold reset, service processors are now responsive, as well as system | [production] | 
            
  | 09:21 | <tstarling> | synchronized php-1.5/languages/Language.php | [production] | 
            
  
    | 2009-06-24
      
      § | 
    
  | 16:27 | <robh> | synchronized php-1.5/InitialiseSettings.php  'bug 19364' | [production] | 
            
  | 15:56 | <Fred> | modified /home/wikipedia/conf/nagios/sync to include new nagios server in the sync/restart process. | [production] | 
            
  | 15:55 | <Fred> | cleaned up /home/wikipedia/conf/nagios/conf.php to remove unused server / conf parameters. | [production] | 
            
  | 15:54 | <Fred> | Cleaned up dsh node_group and resynched Nagios. | [production] | 
            
  | 10:26 | <midom> | synchronized php-1.5/includes/specials/SpecialRecentchanges.php  'livehack upper limit, as mediawiki devs are lazy.. oh wait.' | [production] | 
            
  | 08:29 | <domas> | compressed databases on db3-5 using lzip, freed up the space. :) | [production] | 
            
  | 08:16 | <midom> | synchronized php-1.5/db.php  'db24 back to life' | [production] | 
            
  | 06:28 | <Tim> | updated GeSHi to 1.0.8.4 | [production] | 
            
  | 05:54 | <Tim> | removed all SVN externals from the MW working copy. Updated extensions/SyntaxHighlight_GeSHi to r52346. Scapping. | [production] | 
            
  
    | 2009-06-23
      
      § | 
    
  | 22:00 | <Fred> | make && deploy of new squid configuration with added acl for spence.w.o | [production] | 
            
  | 21:07 | <robh> | synchronized php-1.5/InitialiseSettings.php  '19364 Set project namespace of Portuguese Wikibooks' | [production] | 
            
  | 20:49 | <robh> | synchronized php-1.5/InitialiseSettings.php  'Fixing issues with arwiki namespacing and formatting' | [production] | 
            
  | 19:33 | <Rob> | checked up on the updateTitles running against enwiki on hume.  On 11864380 / 23266362 | [production] | 
            
  | 19:31 | <robh> | synchronized php-1.5/InitialiseSettings.php  '19357 Create new namespace on arwiki' | [production] | 
            
  | 19:22 | <robh> | synchronized php-1.5/InitialiseSettings.php | [production] | 
            
  | 19:14 | <robh> | synchronized php-1.5/InitialiseSettings.php  'Bug 19116 adding namespace aliases for itwikibooks' | [production] | 
            
  | 15:39 | <robh> | synchronized php-1.5/InitialiseSettings.php  '19328 set for fywikibooks - typosssss' | [production] | 
            
  | 15:36 | <robh> | synchronized php-1.5/InitialiseSettings.php  '19328 set for fywikibooks' | [production] | 
            
  | 15:24 | <robh> | synchronized php-1.5/abusefilter.php  'Updated with Andrew' | [production] | 
            
  | 15:15 | <Rob> | wrong reason listed in sync, opps, was for bug 19272 | [production] | 
            
  | 15:15 | <robh> | synchronized php-1.5/InitialiseSettings.php  'extensions/AbuseFilter/abusefilter.tables.sql' | [production] | 
            
  | 15:14 | <robh> | synchronized php-1.5/abusefilter.php  '19274 Enable AbuseFilter on Lithuanian Wikipedia.' | [production] | 
            
  | 08:23 | <midom> | synchronized php-1.5/db.php  'add db28 to s1' | [production] | 
            
  | 08:20 | <midom> | synchronized php-1.5/db.php  'db19 going live' | [production] | 
            
  | 00:09 | <brion> | obsoleted useless "web browser" custom field on bugzilla. Doesn't appear in search, hardcoded list would need to be maintained, generally not useful. | [production] |