| 2009-02-26
      
      § | 
    
  | 19:19 | <Rob> | srv31 commented out of mediawiki-installation node group until its back online | [production] | 
            
  | 19:16 | <Rob> | updated mw-lib on pdf1, restarted pdf generation service | [production] | 
            
  | 19:16 | <Rob> | pdf generation offline for update | [production] | 
            
  | 18:17 | <brion> | srv31 appears to be a bit borked again. sigh. | [production] | 
            
  | 18:05 | <Rob> | Added wgRestrictDisplayTitle to InitialiseSettings to support bug 17307 Remove {{DISPLAYTITLE:}} restrictions for wm2009wiki.  Also removed the wgAllowDisplayTitle as it was an old testwiki flag that is no longer used. | [production] | 
            
  | 17:49 | <Rob> | Updated InitialiseSettings to support bug 17361 Enable DyanmicPageList for Wikimania2009wiki | [production] | 
            
  | 16:44 | <mark> | Brought BGP session to 2828 back up | [production] | 
            
  | 14:59 | <Rob> | updated InitialiseSettings for 17201 Enable the ability for crats on the Arbcom private wiki (English Wikipedia) to also remove crat and sysop | [production] | 
            
  | 14:47 | <Rob> | fixed change in InitialiseSettings for bug 17388 | [production] | 
            
  | 14:42 | <Rob> | rolling back change because I changed the wrong entry. | [production] | 
            
  | 14:29 | <Rob> | Implemented changes to Initilizesettings for bug 17388 Allow bureaucrats to revoke sysop status on UK chapter website | [production] | 
            
  | 11:13 | <mark> | Shut down BGP session to 2828, many reports of partial reachability, seemingly due to a broken link in a multipath | [production] | 
            
  | 06:12 | <Tim> | running gearmanRefreshLinks.php on all wikis with 20 worker threads | [production] | 
            
  
    | 2009-02-25
      
      § | 
    
  | 22:21 | <Andrew> | Deploying AbuseFilter to nowiki. | [production] | 
            
  | 18:58 | <mark> | Installed ms3 on 2 of the 4 bootable disk drives (sda and sdi), on a 80 GB RAID-1 root partition, leaving the rest of those 250 GB drives free, as well as the other 46 drives | [production] | 
            
  | 18:24 | <Rob> | srv215 deployed into service | [production] | 
            
  | 18:16 | <Rob> | installed and setting up srv215 | [production] | 
            
  | 18:08 | <brion> | merging r47809 live -- XSS fix for Collection extension | [production] | 
            
  | 18:01 | <brion> | temporarily halting apache on srv78, which seems to have some borkage with nfs mounts and nis | [production] | 
            
  | 17:55 | <brion> | srv78:  \tls: cannot access /mnt/upload5/wikipedia: No such file or directory | [production] | 
            
  | 17:53 | <Rob> | rename auth2 to sockpuppet in dns, pushed changes | [production] | 
            
  | 16:55 | <Rob> | fixed IP allocation for ms4 and ms3 | [production] | 
            
  | 16:55 | <Rob> | reset ms4 by accident due to IP misassignment on service processors for ms4 and ms3 | [production] | 
            
  | 05:29 | <Tim> | updated ApiQueryLogEvents.php to r47781 | [production] | 
            
  | 04:21 | <Tim> | killed long-running ApiQueryLogEvents queries on db4 (12000 seconds) | [production] | 
            
  | 01:11 | <Andrew> | Deployed AbuseFilter on dewiki per request of DaB | [production] | 
            
  | 01:08 | <Andrew> | adding AbuseFilter tables for dewiki | [production] | 
            
  | 00:03 | <brion> | running mw-serve cache cleaning on pdf1; cron job was borked by missing log file | [production] | 
            
  
    | 2009-02-24
      
      § | 
    
  | 22:46 | <Andrew> | Deployed AbuseFilter on metawiki. | [production] | 
            
  | 22:43 | <Andrew> | Adding abuse filter tables to metawiki. | [production] | 
            
  | 22:30 | <brion> | syncing merge of r47769 to fix regression in page history | [production] | 
            
  | 22:22 | <brion> | fix to disable slow tag search has had a side effect breaking page history. Should be tidied up in a few minutes. | [production] | 
            
  | 22:15 | <brion> | merging r47767 live to disable $wgUseTagFilter | [production] | 
            
  | 20:37 | <Rob> | updating dns for wikimedia.cz | [production] | 
            
  | 20:32 | <brion> | turning Collection/PDF on for all wikisources | [production] | 
            
  | 19:47 | <mark> | Shutting down Apache and unmounting /home on srv32, for puppet testing | [production] | 
            
  | 19:03 | <Rob> | updated wikibooks logo stuff per bug # 17034 | [production] | 
            
  | 17:13 | <brion> | just noting that TeliaSonera has a planned maintenance which may affect KNAMS connectivity for a few minutes around 2009-02-25 05:00 UTC. | [production] | 
            
  | 17:12 | <brion> | just noting that TeliaSonera has a planned maintenance which may affect KNAMS connectivity for a few minutes around 2009-02-25 05:00 UTC. | [production] | 
            
  | 16:51 | <brion> | restarting some dump threads on srv31 | [production] | 
            
  | 01:55 | <river> | ms1 is now replicating to ms4 instead of ms2 | [production] | 
            
  | 00:06 | <Rob> | reverted wikimedia.cz and .cs dns redirects for danny | [production] | 
            
  
    | 2009-02-23
      
      § | 
    
  | 22:56 | <RobH_> | srv136 reinstalled and redeployed as apache | [production] | 
            
  | 22:33 | <mark> | Installed puppet (test install) and thereby automatically gmond as aggregator on srv33 | [production] | 
            
  | 21:16 | <domas> | adler has disk with media errors (ID:5, 6th disk in array): http://p.defau.lt/?3_7_6aIatj3DeNBw_jjtBg - needs cannibalized samuel, disk replacement, and ubuntu install on raid10 | [production] | 
            
  | 19:04 | <Rob> | srv136 back from repairs, reinstalling as apache server | [production] | 
            
  | 18:44 | <Rob> | srv217 not running apache, synced and restarted | [production] | 
            
  | 18:29 | <Rob> | srv33 reinstalled to ubuntu and deployed as apache server | [production] | 
            
  | 18:24 | <Rob> | srv32 reinstalled to ubuntu and deployed as apache server | [production] | 
            
  | 17:55 | <Rob> | reinstalling srv32 to ubuntu | [production] |