| 2011-05-25
      
      § | 
    
  | 22:39 | <awjr> | svn up'd on civicrm.wikimedia.org to r189 - deploying further optimizations to contribution searching | [production] | 
            
  | 21:22 | <Ryan_Lane> | adding wikimedia-lb svc address to squids | [production] | 
            
  | 21:22 | <Ryan_Lane> | adding wikimedialbsecure service to lvs2, for testing https access to wikimedia.org sites. | [production] | 
            
  | 21:21 | <Ryan_Lane> | modified pybal configuration to add wikimedia-lb ip to text backend | [production] | 
            
  | 19:52 | <demon> | synchronized php-1.17/extensions/CentralAuth/SpecialWikiSets.php  'r88825' | [production] | 
            
  | 18:32 | <Ryan_Lane> | turning off the selenium grid virtual machines on tesla | [production] | 
            
  | 17:04 | <RobH> | updated bugzilla per rt 764 and 768.  whine feature should work now | [production] | 
            
  | 16:44 | <awjr> | deployed patch to civicrm.wikimedia.org to optimize contribution search query (r185, wikimedia repo) | [production] | 
            
  | 14:57 | <demon> | synchronized php-1.17/wmf-config/InitialiseSettings.php  'Don't allow users to create accounts on closed wikis either' | [production] | 
            
  | 14:54 | <demon> | synchronized php-1.17/wmf-config/InitialiseSettings.php  'Cleanup wikimania200[58] entries: bug 29064' | [production] | 
            
  | 14:53 | <demon> | synchronized closed.dblist  'Add wikimania200[58] to closed.dblist: bug 29064' | [production] | 
            
  | 14:45 | <demon> | synchronized php-1.17/wmf-config/InitialiseSettings.php  'bug 29064: close wikimania2009wiki' | [production] | 
            
  | 14:43 | <demon> | synchronized closed.dblist  'bug 29064: close wikimania2009wiki' | [production] | 
            
  | 12:27 | <Tim> | marked the deletion jobs in OTRS as "invalid" to avoid confusing error messages | [production] | 
            
  | 12:25 | <Tim> | re-enabled anti-deletion security patch in OTRS, disabled "temporarily" by Fred on June 4, 2009 | [production] | 
            
  | 11:58 | <domas> | rebooting db9, looks unhealthy | [production] | 
            
  | 11:06 | <apergos> | so that was me on srv225: stopped apache, waited for puppet restart: it now syncs, then does the restart. oh btw srv225 isn't in the pybal apache conf file, weirdly... anyone know why? | [production] | 
            
  | 09:15 | <apergos> | restarted front and backend squids on sq75 | [production] | 
            
  | 08:47 | <apergos> | restarted front and back end squids on sq81, front end was 20gb and back end was out to lunch | [production] | 
            
  | 08:35 | <apergos> | ekrem's root partition was full; cleared out some older logs from /var/log/apache2, compressed the most recently rotated (it had failed from disk full), reloaded puppet | [production] | 
            
  | 07:20 | <Tim> | set read_only=1 on db10, it's a slave so it should be read only. Nagios says it was broken for 6 months. | [production] | 
            
  | 07:07 | <midom> | synchronized php-1.17/wmf-config/db.php | [production] | 
            
  | 05:09 | <Tim> | restarted nagios IRC bot | [production] | 
            
  | 01:05 | <Tim> | created an account for myself in watchmouse | [production] | 
            
  | 00:48 | <Andrew> | Running populateBv2011EditCount.php in a screen on hume | [production] | 
            
  | 00:32 | <Ryan_Lane> | pruning binlogs on db9 | [production] | 
            
  
    | 2011-05-24
      
      § | 
    
  | 23:12 | <pdhanda> | synchronized php-1.17/wmf-config/InitialiseSettings.php  'Bug 29129. Added a namespace alias for metawiki.' | [production] | 
            
  | 20:36 | <notpeter_> | resyncing drbd nfs1 to nfs2 | [production] | 
            
  | 20:28 | <RobH> | mobile4 awaiting final configurations for service, mobile5 installation done, not yet run puppet updates | [production] | 
            
  | 20:20 | <RobH> | mobile4 is running puppet updates, mobile5 is mid-install | [production] | 
            
  | 19:46 | <RobH> | mobile4 & mobile5 racked and accessible for mgmt.  installing OS on mobile4 now | [production] | 
            
  | 18:00 | <Ryan_Lane> | added new reverse zone for svc ips in esams; added entries for bits.svc.esams.wmnet and upload.svc.esams.wmnet | [production] | 
            
  | 17:31 | <Ryan_Lane> | removing bits-lb and upload-lb public ip enties, and adding bits.svc.pmtpa.wmnet and upload.svc.pmtpa.wmnet entries for https testing | [production] | 
            
  | 16:46 | <mark> | Upgraded cr2-eqiad to JUNOS 10.4R4.5 | [production] | 
            
  | 16:44 | <mark> | PIM configuration was missing on some virtual interfaces on csw1-sdtpa, which broke purging to esams. Put back in place | [production] | 
            
  | 16:09 | <mark> | Reloading re1.cr2-eqiad | [production] | 
            
  | 16:01 | <mark> | Fixed tunnel hack between pmtpa and eqiad | [production] | 
            
  | 15:01 | <RobH> | db20 networking restarted, back online | [production] | 
            
  | 12:05 | <mark> | Upgraded bootrom and software on asw-d1-sdtpa and asw-a5-sdtpa | [production] | 
            
  | 10:53 | <mark> | Copied existing 2.7.0.2 image to secondary flash, and installed 2.7.0.3a image on primary flash on csw5-pmtpa | [production] | 
            
  | 10:42 | <mark> | Copied existing 2.6.0 image to secondary flash, and installed 2.7.0.3a image on primary flash on csw1-sdtpa | [production] | 
            
  | 04:20 | <demon> | synchronized php-1.17/extensions/GoogleNewsSitemap/GoogleNewsSitemap_body.php  'r88707' | [production] | 
            
  | 04:03 | <tstarling> | synchronized php-1.17/wmf-config/InitialiseSettings.php | [production] | 
            
  | 04:03 | <Tim> | enabling GoogleNewsSitemap on enwikinews | [production] |