| 2011-04-14
      
      § | 
    
  | 22:04 | <RoanKattouw> | Clearing message blobs | [production] | 
            
  | 22:03 | <catrope> | ran sync-common-all | [production] | 
            
  | 22:03 | <RoanKattouw> | Deploying ArticleFeedback changes to the cluster with sync-common-all | [production] | 
            
  | 22:02 | <catrope> | synchronized php-1.17/wmf-config/InitialiseSettings.php  'Disable ArticleFeedback on enwiki for upgrade' | [production] | 
            
  | 21:42 | <mark> | Added vlan 103 to ports e 15/5 to 15/8 (virt1-4 eth1) (tagged) on csw1-sdtpa | [production] | 
            
  | 21:42 | <mark> | Put ports e 15/1 to 15/4 (virt1-4 eth0) in vlan 105 (untagged) on csw1-sdtpa | [production] | 
            
  | 21:40 | <mark> | Created VLAN 105 ("virt-hosts") on csw1-sdtpa, assigned virtual interface ve 9, assigned ip 10.4.16.0/24 | [production] | 
            
  | 21:39 | <mark> | Added virtual interface ve 7 ("virt") on csw1-sdtpa, assigned ip 10.4.0.1/24 | [production] | 
            
  | 21:39 | <mark> | Removed virtual interface ve 7 ("virt") on csw5-pmtpa | [production] | 
            
  | 21:24 | <Ryan_Lane> | cleaning up searchidx1 | [production] | 
            
  | 21:11 | <RoanKattouw> | Running extensions/ArticleFeedback/populateAFRevisions.php on testwiki | [production] | 
            
  | 21:11 | <RoanKattouw> | Creating article_feedback_revisions table on testwiki | [production] | 
            
  | 21:08 | <RoanKattouw> | Updating ArticleFeedback code on test | [production] | 
            
  | 16:18 | <RobH> | singer back online, blogs, survey, and secure should all be back in service | [production] | 
            
  | 16:15 | <RobH> | blogs offline due to singer outage, as well as survey, working on resolution | [production] | 
            
  | 16:14 | <RobH> | powercycling singer, unresponsive to ssh and to serial console | [production] | 
            
  | 16:13 | <RobH> | secure issue is being investigated | [production] | 
            
  | 16:13 | <pdhanda> | running cleanupTitles.php for all wikis in all.dblist | [production] | 
            
  | 16:13 | <RobH> | investigating singer cpu spike | [production] | 
            
  | 15:36 | <mark> | Implemented LDAP lookups on mail relay mchenry | [production] | 
            
  | 14:13 | <Reedy> | Delayed message. Ran repopulateCodePaths on mediawikiwiki | [production] | 
            
  | 14:13 | <Reedy> | Delayed message. Ran populateFollowupRevisions on mediawikiwiki | [production] | 
            
  | 14:04 | <catrope> | synchronized php-1.17/extensions/CodeReview/modules/ext.codereview.js  'Disable comment field autofocus' | [production] | 
            
  | 07:54 | <catrope> | synchronized php-1.17/extensions/CodeReview/CodeReview.php  'Fix CodeReview fatals' | [production] | 
            
  | 07:50 | <catrope> | ran sync-common-all | [production] | 
            
  | 07:47 | <RoanKattouw> | Running sync-common-all and clearMessageBlobs.php to deploy UploadWizard changes | [production] | 
            
  | 07:35 | <RoanKattouw> | Updating MediaWiki on fenari to r86034 for UploadWizard deployment. Only  pushing changes to testwiki for now | [production] | 
            
  | 07:14 | <tstarling> | synchronized php-1.17/includes/WebRequest.php  'r86031' | [production] | 
            
  | 07:14 | <tstarling> | synchronized php-1.17/img_auth.php  'r86031' | [production] | 
            
  
    | 2011-04-13
      
      § | 
    
  | 23:18 | <notpeter> | restarting nagios | [production] | 
            
  | 21:16 | <notpeter> | restarting nagios | [production] | 
            
  | 19:54 | <awjr> | disabling drupal/civicrm cron on contacts.wikimedia.org (singer) as part of an interim solution to civimail problems in CiviCRM 3.4beta1 | [production] | 
            
  | 18:31 | <^demon> | ran code review updates for mediawikiwiki | [production] | 
            
  | 12:54 | <mark> | Restarted pdns on linne | [production] | 
            
  | 12:54 | <mark> | Defined /64 neighbor blocks in 1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa in DNS | [production] | 
            
  | 12:31 | <mark> | Defined /31 neighbor blocks in 154.80.208.in-addr.arpa in DNS | [production] | 
            
  | 09:15 | <ariel> | synchronized php-1.17/wmf-config/CommonSettings.php  'increase account creation throttle value for el wiki for editing workshop (soon we have a two week break from these :-P)' | [production] | 
            
  | 01:55 | <Ryan_Lane> | upgrading squid to 2.7.9-5 on sq58 and sq73 | [production] | 
            
  | 01:35 | <Ryan_Lane> | upgrading squid to 2.7.9-4 on sq58 and sq73 | [production] | 
            
  | 00:42 | <Ryan_Lane|food> | also removing sheepdog from nova-*.tesla | [production] | 
            
  | 00:39 | <Ryan_Lane|food> | removing natty versions of libvirt and friends and re-installing the lucid versions | [production] | 
            
  
    | 2011-04-12
      
      § | 
    
  | 20:18 | <RobH> | killed apache vhost for outreachcivi since it appears defunct in favor of contactscivi on singer | [production] | 
            
  | 19:08 | <Ryan_Lane> | upgrading nova-* on nova-*.tesla | [production] | 
            
  | 19:05 | <Ryan_Lane> | updating squid on sq73 to test new squid and patches | [production] | 
            
  | 05:51 | <Ryan_Lane> | purging nova-ajax-console-proxy from nova-compute1/2.tesla and adding to nova-controller.tesla | [production] | 
            
  | 04:57 | <Ryan_Lane> | purging nova-ajax-console-proxy from nova-controller.tesla and adding to nova-compute1/2.tesla | [production] | 
            
  | 02:11 | <tstarling> | synchronized php-1.17/includes/Sanitizer.php  'r85859' | [production] | 
            
  | 01:30 | <root> | synchronizing Wikimedia installation... Revision: 85853: | [production] | 
            
  | 01:28 | <Tim> | svn up/scap to deploy r85848 | [production] | 
            
  | 00:31 | <Ryan_Lane> | installing glance on nova-controller.tesla | [production] |