| 
      
        2011-11-14
      
      §
     | 
  
    
  | 23:51 | 
  <ben> | 
  synchronized wmf-config/db.php  'taking out ms boxes from cluster22, adding es4 so all ES traffic is only on new hosts.' | 
  [production] | 
            
  | 23:50 | 
  <awjr> | 
  synchronizing payments cluster [[rev:103082|r103082]] | 
  [production] | 
            
  | 23:43 | 
  <RobH> | 
  ocg3 drac flashed and repaired, installing os now. | 
  [production] | 
            
  | 23:43 | 
  <maplebed> | 
  master switch complete - cluster22 is now writing to es3 | 
  [production] | 
            
  | 23:34 | 
  <ben> | 
  synchronized wmf-config/db.php  'making es3 the new master for cluster22' | 
  [production] | 
            
  | 23:32 | 
  <maplebed> | 
  about to switch master for the external store cluster22 from ms3 to es3 | 
  [production] | 
            
  | 23:06 | 
  <awjrichards> | 
  synchronized php/extensions/ContributionReporting/ContributionReporting.php  '[[rev:103065|r103065]]' | 
  [production] | 
            
  | 23:01 | 
  <Reedy> | 
  That wasBug 32417 - Enable MoodBar on Incubator | 
  [production] | 
            
  | 23:01 | 
  <reedy> | 
  synchronized wmf-config/InitialiseSettings.php  | 
  [production] | 
            
  | 21:29 | 
  <^demon> | 
  killed my svn dump from earlier, ate up all the free space on formey | 
  [production] | 
            
  | 21:03 | 
  <reedy> | 
  synchronized wmf-config/CommonSettings.php  'Enable REL1_18 in ED' | 
  [production] | 
            
  | 21:00 | 
  <Ryan_Lane> | 
  upgraded raskin | 
  [production] | 
            
  | 20:21 | 
  <^demon> | 
  doing full svn dump in screen on formey, probably will take awhile. | 
  [production] | 
            
  | 18:32 | 
  <preilly> | 
  emergency mobile frontend fix for broken lists  | 
  [production] | 
            
  | 18:32 | 
  <preilly> | 
  synchronized php-1.18/extensions/MobileFrontend/views/layout/application.html.php  'weekly update to mobile frontend' | 
  [production] | 
            
  | 18:27 | 
  <nikerabbit> | 
  synchronized php-1.18/tests/phpunit/languages/LanguageMlTest.php  '[[rev:102998|r102998]] - i18n deploy' | 
  [production] | 
            
  | 18:27 | 
  <preilly> | 
  synchronized php-1.18/extensions/MobileFrontend/stylesheets/common.css  'weekly update to mobile frontend' | 
  [production] | 
            
  | 18:26 | 
  <nikerabbit> | 
  synchronized php-1.18/extensions/CategoryTree/  '[[rev:102998|r102998]] - i18n deploy' | 
  [production] | 
            
  | 18:25 | 
  <nikerabbit> | 
  synchronized php-1.18/extensions/Babel/  '[[rev:102998|r102998]] - i18n deploy' | 
  [production] | 
            
  | 18:25 | 
  <nikerabbit> | 
  synchronized php-1.18/languages/  '[[rev:102998|r102998]] - i18n deploy' | 
  [production] | 
            
  | 17:18 | 
  <reedy> | 
  synchronized php-1.18/extensions/Contest/specials/SpecialContest.php  '[[rev:102992|r102992]]' | 
  [production] | 
            
  | 17:14 | 
  <reedy> | 
  synchronized php-1.18/extensions/Contest/Contest.i18n.php  '[[rev:102992|r102992]]' | 
  [production] | 
            
  | 16:41 | 
  <mark> | 
  Shutting down sq69 and sq70 for relocation | 
  [production] | 
            
  | 14:56 | 
  <mark> | 
  Migrated sq70 to varnish3; running apt-get dist-upgrade && reboot | 
  [production] | 
            
  | 14:53 | 
  <mark> | 
  Migrated sq69 to varnish3; running apt-get dist-upgrade && reboot | 
  [production] | 
            
  | 14:39 | 
  <mark> | 
  Migrated sq68 to varnish3; running apt-get dist-upgrade && reboot | 
  [production] | 
            
  | 14:29 | 
  <mark> | 
  Migrated sq67 to varnish3; running apt-get dist-upgrade && reboot | 
  [production] | 
            
  | 14:08 | 
  <mark> | 
  Temporarily disabled pmtpa as esams bits backend - preparation for pmtpa varnish upgrade | 
  [production] | 
            
  | 14:04 | 
  <catrope> | 
  synchronized php-1.18/extensions/WikiEditor/modules/jquery.wikiEditor.toolbar.config.js  '[[rev:102982|r102982]]' | 
  [production] | 
            
  | 14:04 | 
  <RoanKattouw> | 
  Strike that, it is responding to SSH because neither the connection nor the setup time out. It's just terribly, terribly slow | 
  [production] | 
            
  | 14:02 | 
  <RoanKattouw> | 
  searchidx2 is dead, not responding to SSH. Ganglia shows elevated load | 
  [production] | 
            
  | 13:57 | 
  <mark> | 
  Set all esams text amssq (SSD) squids to CARP weight 30 | 
  [production] | 
            
  | 13:46 | 
  <mark> | 
  Depooled knsq1 and knsq2 from esams bits duty, preparing for decommissioning | 
  [production] | 
            
  | 13:26 | 
  <mark> | 
  Ran dist-upgrade and reboot on cp3001 and cp3002 | 
  [production] | 
            
  | 13:26 | 
  <mark> | 
  Setup interface aggregates on cp3001 and cp3002 | 
  [production] | 
            
  | 12:04 | 
  <mark> | 
  Deploying bits.eqiad | 
  [production] | 
            
  | 11:58 | 
  <mark> | 
  bits-lb.esams record addition wasn't picked up by pdns on reload during the previous update, quickly moving bits-lb back to bits.esams | 
  [production] | 
            
  | 11:53 | 
  <mark> | 
  Added bits-lb.esams.wikimedia.org DNS record, and updated geodns to use that instead of bits.esams | 
  [production] | 
            
  | 11:52 | 
  <reedy> | 
  synchronized wmf-config/InitialiseSettings.php  'Bug 31459 - Gender in user namespaces for Portuguese Wikipedia' | 
  [production] | 
            
  | 11:39 | 
  <mark> | 
  synchronized wmf-config/CommonSettings.php  'remove old Squids' | 
  [production] | 
            
  | 11:30 | 
  <mark> | 
  Renamed ganglia clusters/dirs on streber | 
  [production] | 
            
  | 11:22 | 
  <mark> | 
  Renamed ganglia RRD dirs on spence | 
  [production] | 
            
  | 10:35 | 
  <catrope> | 
  synchronized php-1.18/extensions/CentralNotice/special/SpecialBannerController.php  '[[rev:102966|r102966]]' | 
  [production] | 
            
  | 10:34 | 
  <catrope> | 
  synchronized php-1.18/extensions/Vector/modules/ext.vector.collapsibleTabs.js  '[[rev:102966|r102966]]' | 
  [production] |