| 2011-08-29
      
      § | 
    
  | 13:29 | <mutante> | srv156 - power up, dist-upgrade/kernel, puppet run, reboot | [production] | 
            
  | 12:57 | <RoanKattouw> | Reverted all of my changes to srv162 and started puppet again. Need to do more to get a core dump, will do that later | [production] | 
            
  | 09:26 | <RoanKattouw> | ... on srv162 | [production] | 
            
  | 09:26 | <RoanKattouw> | Changed the core dump directory to /a/tmp/apachecore because the root partition doesn't have much free space but /a does | [production] | 
            
  | 09:23 | <RoanKattouw> | Set up Apache core dumping on srv162 *correctly* by uncommenting CoreDumpDirectory /tmp/apache-core locally in /etc/apache2/wmf/main.conf | [production] | 
            
  | 09:03 | <RoanKattouw> | Changed ownership of /mnt/upload6/math/8/0/0/800618943025315f869e4e1f09471012.png from root:root to apache:apache, permissions errors were causing PHP warnings | [production] | 
            
  | 07:39 | <RoanKattouw> | Reverted my changes on srv163 and started puppet | [production] | 
            
  | 07:38 | <RoanKattouw> | Stopped puppet on srv162, set Apache's cwd to /a/tmp/apachecore in /etc/apache2/envvars , and set ulimit -c 1000000 in /etc/default/apache2 | [production] | 
            
  | 07:34 | <RoanKattouw> | Moving my core dump for segfault debugging test to srv162 instead of srv163, for disk space reasons | [production] | 
            
  | 07:32 | <RoanKattouw> | Stopped puppet on srv163 to prevent it from reverting my hacks | [production] | 
            
  | 07:26 | <RoanKattouw> | Restarting Apache on srv163 so these changes take effect | [production] | 
            
  | 07:26 | <RoanKattouw> | Enabled core dumps for Apache on srv163 by editing /etc/default/apache2 | [production] | 
            
  | 07:19 | <RoanKattouw> | Changing Apache's cwd on srv163 by editing /etc/apache2/envvars | [production] | 
            
  | 02:18 | <LocalisationUpdate> | completed (1.17) at Mon Aug 29 02:20:17 UTC 2011 | [production] | 
            
  
    | 2011-08-26
      
      § | 
    
  | 21:06 | <mutante> | amssq48 - power back up, clean squid, dist-upgrade | [production] | 
            
  | 16:37 | <robh> | updating text-settings to move sq36 into the squid api cluster.  puppet updated already for the same, and pybal updated to remove sq36 frontend from normal text service | [production] | 
            
  | 02:27 | <LocalisationUpdate> | completed (1.17) at Fri Aug 26 02:29:15 UTC 2011 | [production] | 
            
  | 00:11 | <robh> | change reverted, nothing bad, but undesired result.  hooper back to normal | [production] | 
            
  | 00:09 | <robh> | hooper apache config change for https redirection on etherpad | [production] | 
            
  | 00:09 | <robh> | i meant to paste the rt link | [production] | 
            
  | 00:09 | <robh> | testing something in hooper apache config, should result in nothing noticeable to users, unless i did it wrong. | [production] | 
            
  | 00:08 | <maplebed> | changed puppet client run interval from the default (30m) to 2hrs to reduce load on the master. | [production] | 
            
  
    | 2011-08-25
      
      § | 
    
  | 23:29 | <awjrichards> | synchronizing Wikimedia installation... Revision: 95505: | [production] | 
            
  | 23:20 | <awjrichards> | synchronizing Wikimedia installation... Revision: 95505: | [production] | 
            
  | 23:10 | <awjrichards> | synchronized wmf-config/CommonSettings.php | [production] | 
            
  | 23:09 | <awjrichards> | synchronized wmf-config/InitialiseSettings.php | [production] | 
            
  | 23:08 | <awjrichards> | synchronized php/extensions/LandingCheck/LandingCheck.i18n.php  '[[rev:95542|r95542]]' | [production] | 
            
  | 23:08 | <awjrichards> | synchronized php/extensions/LandingCheck/SpecialLandingCheck.php  '[[rev:95542|r95542]]' | [production] | 
            
  | 22:48 | <awjrichards> | synchronizing Wikimedia installation... Revision: 95505: | [production] | 
            
  | 22:47 | <mark> | Rebooting lvs1002, lvs1003, lvs1005, lvs1006 | [production] | 
            
  | 22:43 | <robh> | gallium deployed for continuous integration testing per RT#1204.  Requires further development input for final system configuration. | [production] | 
            
  | 22:29 | <mark> | Deployed cr2-pmtpa as backup bootp forwarder and PIM router | [production] | 
            
  | 22:24 | <mark> | Deployed cr2-pmtpa as backup VRRP router on all production subnets | [production] | 
            
  | 22:05 | <mark> | Enabled cr2-pmtpa:irb.105 (subnet virt-hosts) family inet; activated cr2-pmtpa as backup VRRP router for that subnet | [production] | 
            
  | 21:54 | <awjrichards> | synchronized php/extensions/CentralNotice/centralnotice.css  '[[rev:95518|r95518]]' | [production] | 
            
  | 21:48 | <mark> | Setup iBGP on cr2-pmtpa to all other AS14907 routers | [production] | 
            
  | 21:46 | <awjrichards> | synchronized php/extensions/CentralNotice/centralnotice.js | [production] | 
            
  | 21:41 | <Jeff_Green> | taking payments4 out of the pool for testing | [production] | 
            
  | 21:13 | <robh> | installing gallium for CI testing. | [production] | 
            
  | 19:52 | <catrope> | synchronized php/includes/api/ApiQuerySiteinfo.php  'Fix PHP warning' | [production] | 
            
  | 19:50 | <catrope> | synchronizing Wikimedia installation... Revision: 95505: | [production] | 
            
  | 19:49 | <RoanKattouw> | Run scap to deploy HTTPS / prot rel fixes | [production] | 
            
  | 19:22 | <awjr> | deployed contrib log auditing fixes on Grosley | [production] | 
            
  | 19:20 | <catrope> | synchronized wmf-config/secure.php  'Factor out URL fixing into fixupUrl, add hook functions for GetCanonicalURL and IRCLineURL' | [production] | 
            
  | 19:13 | <catrope> | synchronized wmf-config/secure.php  'rm debugging' | [production] | 
            
  | 19:12 | <catrope> | synchronized wmf-config/secure.php  'Fix setting of $wgInternalServer for protocol-relative URL wikis (was broken since introduction of prot rel) and set $wgCanonicalServer' | [production] | 
            
  | 19:05 | <catrope> | synchronized wmf-config/secure.php  'Debugging' | [production] |