| 
      
        2012-09-18
      
      §
     | 
  
    
  | 15:41 | 
  <cmjohnson1> | 
  srv249 powercycling to check drac configuration | 
  [production] | 
            
  | 14:08 | 
  <hashar> | 
  synchronized wmf-config/InitialiseSettings.php  '([[bugzilla:40122|bug 40122]]) disable GlobalBlocking on fishbowl, private wikis' | 
  [production] | 
            
  | 13:53 | 
  <hashar> | 
  synchronized php-1.20wmf11/includes/Sanitizer.php  '{{bug|40306}} - Only convert align to float for table' | 
  [production] | 
            
  | 13:53 | 
  <hashar> | 
  synchronized php-1.20wmf11/tests/phpunit/includes/SanitizerTest.php  '{{bug|40306}} - Only convert align to float for table' | 
  [production] | 
            
  | 13:48 | 
  <apergos1> | 
  ms-fe1 back in swift frontend pool, upgrade to precise complete. | 
  [production] | 
            
  | 13:41 | 
  <hashar> | 
  {{gerrit|23798}} manually deleted /apache/common-local/docroot/wikipedia.org/{million,test}.txt files from all mediawiki-installation hosts | 
  [production] | 
            
  | 12:53 | 
  <hashar> | 
  synchronized php-1.20wmf12/includes/Sanitizer.php  '{{bug|40306}} - Only convert align to float for table' | 
  [production] | 
            
  | 12:53 | 
  <hashar> | 
  synchronized php-1.20wmf12/tests/phpunit/includes/SanitizerTest.php  '{{bug|40306}} - Only convert align to float for table' | 
  [production] | 
            
  | 12:22 | 
  <apergos1> | 
  reboot ms-fe1, upgrading to precise | 
  [production] | 
            
  | 05:59 | 
  <Aaron|home> | 
  Last of the math/ file copy scripts has finished | 
  [production] | 
            
  | 02:42 | 
  <LocalisationUpdate> | 
  completed (1.20wmf12) at Tue Sep 18 02:42:15 UTC 2012 | 
  [production] | 
            
  | 02:26 | 
  <LocalisationUpdate> | 
  completed (1.20wmf11) at Tue Sep 18 02:26:08 UTC 2012 | 
  [production] | 
            
  | 00:20 | 
  <Ryan_Lane> | 
  adding an autofs direct map to LDAP, pointing at the gluster storage for testing home directory move | 
  [production] | 
            
  
    | 
      
        2012-09-17
      
      §
     | 
  
    
  | 23:43 | 
  <paravoid> | 
  adding a modified ixgbe.ko to precise-installer's initrd.gz; adding ixgbe-dkms to apt | 
  [production] | 
            
  | 23:30 | 
  <notpeter> | 
  starting puppet on brewster | 
  [production] | 
            
  | 23:17 | 
  <notpeter> | 
  stopping puppet on brewster | 
  [production] | 
            
  | 21:45 | 
  <apergos> | 
  (from 5 hours ago) last of the swift proxy services restarted, it had reached 8.6gb of swap use.  good for a few more days now | 
  [production] | 
            
  | 20:52 | 
  <Jeff_Green> | 
  stopped puppet on aluminium for config testing | 
  [production] | 
            
  | 20:45 | 
  <preilly> | 
  synchronized wmf-config/CommonSettings.php  'enable EPUB' | 
  [production] | 
            
  | 18:55 | 
  <notpeter> | 
  removing srv190 and mw55-59 from apaches pool for upgrade to precise | 
  [production] | 
            
  | 18:48 | 
  <reedy> | 
  rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.20wmf12 | 
  [production] | 
            
  | 18:47 | 
  <reedy> | 
  Finished syncing Wikimedia installation... : Rebuild localisation cache | 
  [production] | 
            
  | 18:20 | 
  <reedy> | 
  Started syncing Wikimedia installation... : Rebuild localisation cache | 
  [production] | 
            
  | 18:13 | 
  <aaron> | 
  synchronized php-1.20wmf12/extensions/timeline/Timeline.php  | 
  [production] | 
            
  | 18:10 | 
  <reedy> | 
  synchronized wmf-config/InitialiseSettings.php  'Enabled html5 on all wikis' | 
  [production] | 
            
  | 18:07 | 
  <binasher> | 
  granted 'research' read access to user rollup/filter dbs on dbs 42,1047 | 
  [production] | 
            
  | 18:06 | 
  <reedy> | 
  Finished syncing Wikimedia installation... : test2wiki to 1.20wmf12 and rebuild localisation cachae | 
  [production] | 
            
  | 16:36 | 
  <reedy> | 
  Started syncing Wikimedia installation... : test2wiki to 1.20wmf12 and rebuild localisation cachae | 
  [production] | 
            
  | 16:21 | 
  <reedy> | 
  synchronized php-1.20wmf11/extensions/ArticleFeedbackv5/  'Bring in logging related fixes etc' | 
  [production] | 
            
  | 16:17 | 
  <reedy> | 
  synchronized php-1.20wmf12  'Initial sync' | 
  [production] | 
            
  | 15:31 | 
  <Reedy> | 
  Deleted 1.20wmf10 l10n cache | 
  [production] | 
            
  | 15:28 | 
  <Reedy> | 
  Pointing php to php-1.20wmf11 | 
  [production] | 
            
  | 08:21 | 
  <apergos1> | 
  swap was using 7gb on ms-fe1... restarted swift-proxy over there.  wlll prolly restart it on ms-fe2 through 4 over the next day or two | 
  [production] | 
            
  | 02:26 | 
  <LocalisationUpdate> | 
  completed (1.20wmf11) at Mon Sep 17 02:26:30 UTC 2012 | 
  [production] | 
            
  
    | 
      
        2012-09-15
      
      §
     | 
  
    
  | 19:04 | 
  <mark> | 
  Power cycled srv212, 211, 226, 273, 237 | 
  [production] | 
            
  | 19:04 | 
  <tstarling> | 
  synchronized wmf-config/mc.php  'swapping out other slow servers' | 
  [production] | 
            
  | 18:53 | 
  <tstarling> | 
  synchronized wmf-config/mc.php  'swap out srv237 for srv249' | 
  [production] | 
            
  | 18:46 | 
  <tstarling> | 
  synchronized php-1.20wmf11/includes/objectcache/MemcachedClient.php  'memcached profiling' | 
  [production] | 
            
  | 18:37 | 
  <mark> | 
  Reduced MaxRequestsPerClient from 300 to 4 in apache.conf | 
  [production] | 
            
  | 18:29 | 
  <LeslieCarr> | 
  logs stopped on flourine between 18:11 and 18:27 UTC | 
  [production] | 
            
  | 14:03 | 
  <LeslieCarr> | 
  powercycling srv259 (unresponsive) | 
  [production] | 
            
  | 13:57 | 
  <LeslieCarr> | 
  rebooting srv270 (unresponsive) | 
  [production] | 
            
  | 13:36 | 
  <LeslieCarr> | 
  rebooting srv288 | 
  [production] | 
            
  | 11:54 | 
  <andrew> | 
  synchronized wmf-config/throttle.php  'Fix undeclared variable error' | 
  [production] | 
            
  | 09:56 | 
  <reedy> | 
  synchronized wmf-config/throttle.php  'Bug 40263 - Request to lift IP address for mass creation - Philippines SFD 2012' | 
  [production] | 
            
  | 08:52 | 
  <thedj> | 
  someone should deploy Planets update, or add blogs to the old config because otherwise WLM has passed before the blogs are added to the stream. | 
  [production] |