| 
      
        2009-02-26
      
      §
     | 
  
    
  | 19:19 | 
  <Rob> | 
  srv31 commented out of mediawiki-installation node group until its back online | 
  [production] | 
            
  | 19:16 | 
  <Rob> | 
  updated mw-lib on pdf1, restarted pdf generation service | 
  [production] | 
            
  | 19:16 | 
  <Rob> | 
  pdf generation offline for update | 
  [production] | 
            
  | 18:17 | 
  <brion> | 
  srv31 appears to be a bit borked again. sigh. | 
  [production] | 
            
  | 18:05 | 
  <Rob> | 
  Added wgRestrictDisplayTitle to InitialiseSettings to support bug 17307 Remove {{DISPLAYTITLE:}} restrictions for wm2009wiki.  Also removed the wgAllowDisplayTitle as it was an old testwiki flag that is no longer used. | 
  [production] | 
            
  | 17:49 | 
  <Rob> | 
  Updated InitialiseSettings to support bug 17361 Enable DyanmicPageList for Wikimania2009wiki | 
  [production] | 
            
  | 16:44 | 
  <mark> | 
  Brought BGP session to 2828 back up | 
  [production] | 
            
  | 14:59 | 
  <Rob> | 
  updated InitialiseSettings for 17201 Enable the ability for crats on the Arbcom private wiki (English Wikipedia) to also remove crat and sysop | 
  [production] | 
            
  | 14:47 | 
  <Rob> | 
  fixed change in InitialiseSettings for bug 17388 | 
  [production] | 
            
  | 14:42 | 
  <Rob> | 
  rolling back change because I changed the wrong entry. | 
  [production] | 
            
  | 14:29 | 
  <Rob> | 
  Implemented changes to Initilizesettings for bug 17388 Allow bureaucrats to revoke sysop status on UK chapter website | 
  [production] | 
            
  | 11:13 | 
  <mark> | 
  Shut down BGP session to 2828, many reports of partial reachability, seemingly due to a broken link in a multipath | 
  [production] | 
            
  | 06:12 | 
  <Tim> | 
  running gearmanRefreshLinks.php on all wikis with 20 worker threads | 
  [production] | 
            
  
    | 
      
        2009-02-25
      
      §
     | 
  
    
  | 22:21 | 
  <Andrew> | 
  Deploying AbuseFilter to nowiki. | 
  [production] | 
            
  | 18:58 | 
  <mark> | 
  Installed ms3 on 2 of the 4 bootable disk drives (sda and sdi), on a 80 GB RAID-1 root partition, leaving the rest of those 250 GB drives free, as well as the other 46 drives | 
  [production] | 
            
  | 18:24 | 
  <Rob> | 
  srv215 deployed into service | 
  [production] | 
            
  | 18:16 | 
  <Rob> | 
  installed and setting up srv215 | 
  [production] | 
            
  | 18:08 | 
  <brion> | 
  merging r47809 live -- XSS fix for Collection extension | 
  [production] | 
            
  | 18:01 | 
  <brion> | 
  temporarily halting apache on srv78, which seems to have some borkage with nfs mounts and nis | 
  [production] | 
            
  | 17:55 | 
  <brion> | 
  srv78:  \tls: cannot access /mnt/upload5/wikipedia: No such file or directory | 
  [production] | 
            
  | 17:53 | 
  <Rob> | 
  rename auth2 to sockpuppet in dns, pushed changes | 
  [production] | 
            
  | 16:55 | 
  <Rob> | 
  fixed IP allocation for ms4 and ms3 | 
  [production] | 
            
  | 16:55 | 
  <Rob> | 
  reset ms4 by accident due to IP misassignment on service processors for ms4 and ms3 | 
  [production] | 
            
  | 05:29 | 
  <Tim> | 
  updated ApiQueryLogEvents.php to r47781 | 
  [production] | 
            
  | 04:21 | 
  <Tim> | 
  killed long-running ApiQueryLogEvents queries on db4 (12000 seconds) | 
  [production] | 
            
  | 01:11 | 
  <Andrew> | 
  Deployed AbuseFilter on dewiki per request of DaB | 
  [production] | 
            
  | 01:08 | 
  <Andrew> | 
  adding AbuseFilter tables for dewiki | 
  [production] | 
            
  | 00:03 | 
  <brion> | 
  running mw-serve cache cleaning on pdf1; cron job was borked by missing log file | 
  [production] | 
            
  
    | 
      
        2009-02-24
      
      §
     | 
  
    
  | 22:46 | 
  <Andrew> | 
  Deployed AbuseFilter on metawiki. | 
  [production] | 
            
  | 22:43 | 
  <Andrew> | 
  Adding abuse filter tables to metawiki. | 
  [production] | 
            
  | 22:30 | 
  <brion> | 
  syncing merge of r47769 to fix regression in page history | 
  [production] | 
            
  | 22:22 | 
  <brion> | 
  fix to disable slow tag search has had a side effect breaking page history. Should be tidied up in a few minutes. | 
  [production] | 
            
  | 22:15 | 
  <brion> | 
  merging r47767 live to disable $wgUseTagFilter | 
  [production] | 
            
  | 20:37 | 
  <Rob> | 
  updating dns for wikimedia.cz | 
  [production] | 
            
  | 20:32 | 
  <brion> | 
  turning Collection/PDF on for all wikisources | 
  [production] | 
            
  | 19:47 | 
  <mark> | 
  Shutting down Apache and unmounting /home on srv32, for puppet testing | 
  [production] | 
            
  | 19:03 | 
  <Rob> | 
  updated wikibooks logo stuff per bug # 17034 | 
  [production] | 
            
  | 17:13 | 
  <brion> | 
  just noting that TeliaSonera has a planned maintenance which may affect KNAMS connectivity for a few minutes around 2009-02-25 05:00 UTC. | 
  [production] | 
            
  | 17:12 | 
  <brion> | 
  just noting that TeliaSonera has a planned maintenance which may affect KNAMS connectivity for a few minutes around 2009-02-25 05:00 UTC. | 
  [production] | 
            
  | 16:51 | 
  <brion> | 
  restarting some dump threads on srv31 | 
  [production] | 
            
  | 01:55 | 
  <river> | 
  ms1 is now replicating to ms4 instead of ms2 | 
  [production] | 
            
  | 00:06 | 
  <Rob> | 
  reverted wikimedia.cz and .cs dns redirects for danny | 
  [production] | 
            
  
    | 
      
        2009-02-23
      
      §
     | 
  
    
  | 22:56 | 
  <RobH_> | 
  srv136 reinstalled and redeployed as apache | 
  [production] | 
            
  | 22:33 | 
  <mark> | 
  Installed puppet (test install) and thereby automatically gmond as aggregator on srv33 | 
  [production] | 
            
  | 21:16 | 
  <domas> | 
  adler has disk with media errors (ID:5, 6th disk in array): http://p.defau.lt/?3_7_6aIatj3DeNBw_jjtBg - needs cannibalized samuel, disk replacement, and ubuntu install on raid10 | 
  [production] | 
            
  | 19:04 | 
  <Rob> | 
  srv136 back from repairs, reinstalling as apache server | 
  [production] | 
            
  | 18:44 | 
  <Rob> | 
  srv217 not running apache, synced and restarted | 
  [production] | 
            
  | 18:29 | 
  <Rob> | 
  srv33 reinstalled to ubuntu and deployed as apache server | 
  [production] | 
            
  | 18:24 | 
  <Rob> | 
  srv32 reinstalled to ubuntu and deployed as apache server | 
  [production] | 
            
  | 17:55 | 
  <Rob> | 
  reinstalling srv32 to ubuntu | 
  [production] |