| 2011-11-14
      
      ยง | 
    
  | 23:51 | <ben> | synchronized wmf-config/db.php  'taking out ms boxes from cluster22, adding es4 so all ES traffic is only on new hosts.' | [production] | 
            
  | 23:50 | <awjr> | synchronizing payments cluster [[rev:103082|r103082]] | [production] | 
            
  | 23:43 | <RobH> | ocg3 drac flashed and repaired, installing os now. | [production] | 
            
  | 23:43 | <maplebed> | master switch complete - cluster22 is now writing to es3 | [production] | 
            
  | 23:34 | <ben> | synchronized wmf-config/db.php  'making es3 the new master for cluster22' | [production] | 
            
  | 23:32 | <maplebed> | about to switch master for the external store cluster22 from ms3 to es3 | [production] | 
            
  | 23:06 | <awjrichards> | synchronized php/extensions/ContributionReporting/ContributionReporting.php  '[[rev:103065|r103065]]' | [production] | 
            
  | 23:01 | <Reedy> | That wasBug 32417 - Enable MoodBar on Incubator | [production] | 
            
  | 23:01 | <reedy> | synchronized wmf-config/InitialiseSettings.php | [production] | 
            
  | 21:29 | <^demon> | killed my svn dump from earlier, ate up all the free space on formey | [production] | 
            
  | 21:03 | <reedy> | synchronized wmf-config/CommonSettings.php  'Enable REL1_18 in ED' | [production] | 
            
  | 21:00 | <Ryan_Lane> | upgraded raskin | [production] | 
            
  | 20:21 | <^demon> | doing full svn dump in screen on formey, probably will take awhile. | [production] | 
            
  | 18:32 | <preilly> | emergency mobile frontend fix for broken lists | [production] | 
            
  | 18:32 | <preilly> | synchronized php-1.18/extensions/MobileFrontend/views/layout/application.html.php  'weekly update to mobile frontend' | [production] | 
            
  | 18:27 | <nikerabbit> | synchronized php-1.18/tests/phpunit/languages/LanguageMlTest.php  '[[rev:102998|r102998]] - i18n deploy' | [production] | 
            
  | 18:27 | <preilly> | synchronized php-1.18/extensions/MobileFrontend/stylesheets/common.css  'weekly update to mobile frontend' | [production] | 
            
  | 18:26 | <nikerabbit> | synchronized php-1.18/extensions/CategoryTree/  '[[rev:102998|r102998]] - i18n deploy' | [production] | 
            
  | 18:25 | <nikerabbit> | synchronized php-1.18/extensions/Babel/  '[[rev:102998|r102998]] - i18n deploy' | [production] | 
            
  | 18:25 | <nikerabbit> | synchronized php-1.18/languages/  '[[rev:102998|r102998]] - i18n deploy' | [production] | 
            
  | 17:18 | <reedy> | synchronized php-1.18/extensions/Contest/specials/SpecialContest.php  '[[rev:102992|r102992]]' | [production] | 
            
  | 17:14 | <reedy> | synchronized php-1.18/extensions/Contest/Contest.i18n.php  '[[rev:102992|r102992]]' | [production] | 
            
  | 16:41 | <mark> | Shutting down sq69 and sq70 for relocation | [production] | 
            
  | 14:56 | <mark> | Migrated sq70 to varnish3; running apt-get dist-upgrade && reboot | [production] | 
            
  | 14:53 | <mark> | Migrated sq69 to varnish3; running apt-get dist-upgrade && reboot | [production] | 
            
  | 14:39 | <mark> | Migrated sq68 to varnish3; running apt-get dist-upgrade && reboot | [production] | 
            
  | 14:29 | <mark> | Migrated sq67 to varnish3; running apt-get dist-upgrade && reboot | [production] | 
            
  | 14:08 | <mark> | Temporarily disabled pmtpa as esams bits backend - preparation for pmtpa varnish upgrade | [production] | 
            
  | 14:04 | <catrope> | synchronized php-1.18/extensions/WikiEditor/modules/jquery.wikiEditor.toolbar.config.js  '[[rev:102982|r102982]]' | [production] | 
            
  | 14:04 | <RoanKattouw> | Strike that, it is responding to SSH because neither the connection nor the setup time out. It's just terribly, terribly slow | [production] | 
            
  | 14:02 | <RoanKattouw> | searchidx2 is dead, not responding to SSH. Ganglia shows elevated load | [production] | 
            
  | 13:57 | <mark> | Set all esams text amssq (SSD) squids to CARP weight 30 | [production] | 
            
  | 13:46 | <mark> | Depooled knsq1 and knsq2 from esams bits duty, preparing for decommissioning | [production] | 
            
  | 13:26 | <mark> | Ran dist-upgrade and reboot on cp3001 and cp3002 | [production] | 
            
  | 13:26 | <mark> | Setup interface aggregates on cp3001 and cp3002 | [production] | 
            
  | 12:04 | <mark> | Deploying bits.eqiad | [production] | 
            
  | 11:58 | <mark> | bits-lb.esams record addition wasn't picked up by pdns on reload during the previous update, quickly moving bits-lb back to bits.esams | [production] | 
            
  | 11:53 | <mark> | Added bits-lb.esams.wikimedia.org DNS record, and updated geodns to use that instead of bits.esams | [production] | 
            
  | 11:52 | <reedy> | synchronized wmf-config/InitialiseSettings.php  'Bug 31459 - Gender in user namespaces for Portuguese Wikipedia' | [production] | 
            
  | 11:39 | <mark> | synchronized wmf-config/CommonSettings.php  'remove old Squids' | [production] | 
            
  | 11:30 | <mark> | Renamed ganglia clusters/dirs on streber | [production] | 
            
  | 11:22 | <mark> | Renamed ganglia RRD dirs on spence | [production] | 
            
  | 10:35 | <catrope> | synchronized php-1.18/extensions/CentralNotice/special/SpecialBannerController.php  '[[rev:102966|r102966]]' | [production] | 
            
  | 10:34 | <catrope> | synchronized php-1.18/extensions/Vector/modules/ext.vector.collapsibleTabs.js  '[[rev:102966|r102966]]' | [production] | 
            
  | 09:58 | <mark> | Fixing ganglia's confusedness about $site | [production] | 
            
  | 08:39 | <catrope> | synchronized php-1.18/extensions/Collection/Collection.body.php  '[[rev:102950|r102950]]' | [production] | 
            
  | 08:26 | <RoanKattouw> | I meant [[rev:102944|r102944]] | [production] | 
            
  | 07:52 | <catrope> | synchronized php-1.18/extensions/Gadgets/Gadgets_body.php  '[[rev:102994|r102994]]' | [production] | 
            
  | 05:09 | <awjr> | synchroinzing payments cluster to [[rev:102939|r102939]] | [production] | 
            
  | 03:31 | <awjr> | synchronizing payments cluster to [[rev:102935|r102935]] | [production] |