| 2010-09-14
      
      § | 
    
  | 19:06 | <mark> | Restarted pdns on dobson/ns0 | [production] | 
            
  | 19:05 | <mark> | Fixing wikimedia.org MX records in DNS | [production] | 
            
  | 18:59 | <RobH> | mail server issues are being investigated. | [production] | 
            
  | 18:48 | <Ryan_Lane> | removed public IP address to test-payments | [production] | 
            
  | 18:45 | <awjr> | Restarting test-payments post upgrade | [production] | 
            
  | 18:40 | <Ryan_Lane> | added public IP address to test-payments | [production] | 
            
  | 18:37 | <awjr> | Performing Ubuntu upgrade from 8.04 -> 10.04 on test-payments on tesla | [production] | 
            
  | 18:37 | <RobH> | db34 is running memtest86, relocating the memory moved it, who knows if its due to reseating or relocation?  hence memtest is in progress. | [production] | 
            
  | 18:00 | <RobH> | db34 system board replaced, and also bad cpu replaced, back in rack and installing os. | [production] | 
            
  | 17:55 | <jeluf> | synchronized php-1.5/wmf-config/InitialiseSettings.php  '25128 - Books namespace for fiwiki' | [production] | 
            
  | 17:53 | <Ryan_Lane> | delegated corp.wikimedia.org to ns1.corp.wikimedia.org and ns2.corp.wikimedia.org | [production] | 
            
  | 17:49 | <jeluf> | synchronized php-1.5/wmf-config/InitialiseSettings.php  '25160 - Please create an "autopatrolled" usergroup at metawiki' | [production] | 
            
  | 17:46 | <jeluf> | synchronized php-1.5/wmf-config/InitialiseSettings.php  '25142 - Set $wgGroupPermissions['*']['createpage'] = true; zhwiki (Allowed IP createpage)' | [production] | 
            
  | 17:43 | <jeluf> | synchronized php-1.5/wmf-config/InitialiseSettings.php  '25166 - patroller on Fawiki' | [production] | 
            
  | 17:39 | <jeluf> | synchronized php-1.5/wmf-config/InitialiseSettings.php | [production] | 
            
  | 17:36 | <JeLuF> | closed swwikibooks (bug 25170) | [production] | 
            
  | 17:36 | <jeluf> | ran sync-common-all | [production] | 
            
  | 17:10 | <jeluf> | synchronized php-1.5/wmf-config/InitialiseSettings.php  '25125 - Please update logo for Kurdish Wikipedia' | [production] | 
            
  | 11:43 | <tstarling> | synchronized php-1.5/wmf-config/InitialiseSettings.php  'PagedTiffHandler' | [production] | 
            
  | 11:42 | <tstarling> | synchronized php-1.5/wmf-config/CommonSettings.php  'enabling PagedTiffHandler on all wikis instead of just commons' | [production] | 
            
  | 11:41 | <fixed> | logmsgbot UDP listener | [production] | 
            
  | 09:58 | <rainman-sr> | search10-19 show down in ganglia and nagios with message "Backend Squid HTTP", although they are alive and well and not squids. Could someone look into this? | [production] | 
            
  | 02:37 | <Tim> | on locke: filtered mobile packets out of packet-loss input | [production] | 
            
  
    | 2010-09-13
      
      § | 
    
  | 20:32 | <Ryan_Lane> | rebuilt formey as 10.04 | [production] | 
            
  | 20:30 | <RobH> | disregard that I cannot spell disregard. | [production] | 
            
  | 20:30 | <RobH> | disreagrd formey popping up and down, Ryan_Lane is working on it. | [production] | 
            
  | 18:17 | <RoanKattouw> | logmsgbot on IRC but broken | [production] | 
            
  | 18:17 | <RoanKattouw> | Synced CommonSettings.php to bump style version appendix | [production] | 
            
  | 18:17 | <RoanKattouw> | Synced extensions/UsabilityInitiative/WikiEditor/WikiEditor.combined.min.js for r72920 | [production] | 
            
  | 14:33 | <RoanKattouw> | Started logmsgbot as catrope. Filed RT #217 for cleanup of logmsgbot processes | [production] | 
            
  | 14:28 | <RoanKattouw> | Synced CommonSettings.php for bug 25106 (allow .sla uploads on outreachwiki) | [production] | 
            
  | 14:27 | <RoanKattouw> | logmsgbot MIA | [production] | 
            
  | 11:14 | <hcatlin> | deploying a "perm disable" bugfix for m. | [production] | 
            
  | 11:06 | <hcatlin> | deployed m. language update | [production] | 
            
  | 07:31 | <Tim> | deploying 72894 | [production] | 
            
  
    | 2010-09-10
      
      § | 
    
  | 23:51 | <RobH> | bringing back online srv154, it should start up, run puppet, and put itself back into service | [production] | 
            
  | 23:50 | <RobH> | hey look, the nagios-wm change worked, it logs to both channels, all is well. | [production] | 
            
  | 23:46 | <RobH> | srv154 may flap, its intentional, i assure you | [production] | 
            
  | 23:43 | <RobH> | nagios-wm reports in multiple channels, woot | [production] | 
            
  | 23:21 | <RobH> | tired of messing with nagios bot, its back online | [production] | 
            
  | 23:13 | <RobH> | trying to bounce the nagios bot to make it report in different channels. | [production] | 
            
  | 20:23 | <Ryan_Lane> | manually modified /etc/gmond.conf on srv218 temporarily to use an actually existing directory for ganglia plugin configuration | [production] | 
            
  | 20:22 | <Ryan_Lane> | manually added memcache ganglia plugin to srv218 temporarily for testing before pushing to all memcache servers | [production] | 
            
  | 20:21 | <Ryan_Lane> | added patched python-memcache package to srv218 for memcache ganglia plugin testing | [production] | 
            
  | 20:21 | <Ryan_Lane> | created test-payments.tesla.usability.wikimedia.org for payment processing testing (no public IP) | [production] | 
            
  | 19:06 | <RobH> | mobile2 back online and in lvs pool, mobile pool optimal (as optimal as a 3 server cluster can be) | [production] |