| 2012-09-12
      
      § | 
    
  | 18:57 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else over to 1.20wmf11 | [production] | 
            
  | 18:47 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki back to 1.20wmf11 | [production] | 
            
  | 18:09 | <notpeter> | adding mw59 to apaches pool | [production] | 
            
  | 18:03 | <reedy> | synchronized php-1.20wmf10 | [production] | 
            
  | 17:58 | <reedy> | synchronized php-1.20wmf11 | [production] | 
            
  | 17:53 | <notpeter> | re-adding srv194-199 to apaches pool | [production] | 
            
  | 17:38 | <notpeter> | removing mw1-16 from apaches pool to reimage them as precise jobrunner boxes | [production] | 
            
  | 14:46 | <paravoid> | restarted swift on ms-fe1-4, proxy-server was leaking memory and had been swapping | [production] | 
            
  | 02:52 | <LocalisationUpdate> | completed (1.20wmf10) at Wed Sep 12 02:52:12 UTC 2012 | [production] | 
            
  | 02:27 | <LocalisationUpdate> | completed (1.20wmf11) at Wed Sep 12 02:27:16 UTC 2012 | [production] | 
            
  
    | 2012-09-11
      
      § | 
    
  | 22:29 | <notpeter> | removing srv195-199 from apaches pool for upgarde to precise | [production] | 
            
  | 20:51 | <RobH> | authdns-update per rt 1326 | [production] | 
            
  | 18:31 | <apergos> | second reboot had whines of device not present/not ready for /dev/sdn1 and /dev/sdh1 (note the list is not the same as the previous list), waited a little and it booted up with all disks mounted | [production] | 
            
  | 18:26 | <apergos> | first reboot of msbe10 showed complaints of not ready on devices sdi, k, n but they did eventually mount when the box came up fully.  rebooting a second time to see what happens | [production] | 
            
  | 18:13 | <apergos> | rebooting ms-be10 after recreation and remount of swift filesystems (except those on ssds), see if they stick | [production] | 
            
  | 17:55 | <apergos> | new degradedarray event on ms-be6, booooo | [production] | 
            
  | 17:54 | <apergos> | rebooting ms-bs10 to see if we can get some reasonable device names, in prep for trying the same trick on it as on ms-be6 and 7 | [production] | 
            
  | 17:43 | <Reedy> | Running sync-common on mw8 | [production] | 
            
  | 17:43 | <Reedy> | mw8 seems to have out of date mw files | [production] | 
            
  | 17:42 | <apergos> | rebooting ms-be7 after recreating all filesystems (except those on the ssds) and remounting them, to see if the disks stay visible | [production] | 
            
  | 17:38 | <reedy> | synchronized php-1.20wmf11/extensions/ConfirmEdit/FancyCaptcha.class.php | [production] | 
            
  | 17:31 | <apergos> | puppet disabled on ms-be7 via /etc/default/puppet, swift processes shut down while we test | [production] | 
            
  | 17:10 | <apergos> | one more reboot of ms-be6 after raid repaired, see if it and all mounts stay ok | [production] | 
            
  | 15:57 | <apergos> | rebuilding the degraded raid array on ms-be6 | [production] | 
            
  | 15:15 | <apergos> | stopping swift processes on ms-be6 til we decide what's next; all disks came up and were mounted (but we still do have degraded raid array, not so awesome) | [production] | 
            
  | 15:12 | <apergos> | redid other ms-be6 filesystems so labels would be correct with disk layout without ssds.  reboot again to see if it comes up properly | [production] | 
            
  | 14:44 | <apergos> | manually recreated filesystems and mounted missing drives on ms-be6 (without ssds cabled), seems to have worked without errors.  rebooting to see what happens. | [production] | 
            
  | 14:18 | <apergos> | stopped swift processes on ms-be6 and disabled puppet (/etc/default/puppet) so it won't restart them.  still doing testing of hw over there so we need the box up. | [production] | 
            
  | 04:46 | <reedy> | synchronized php-1.20wmf11/extensions/ConfirmEdit/  '9643ccd3964827651e8a23a19a92ba3d240e60ee' | [production] | 
            
  | 02:42 | <LocalisationUpdate> | completed (1.20wmf10) at Tue Sep 11 02:41:59 UTC 2012 | [production] | 
            
  | 02:22 | <LocalisationUpdate> | completed (1.20wmf11) at Tue Sep 11 02:22:02 UTC 2012 | [production] | 
            
  | 01:30 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki back to 1.20wmf10 | [production] | 
            
  | 01:02 | <paravoid> | adjusting spence's firewall to include all eqiad subnets | [production] | 
            
  
    | 2012-09-10
      
      § | 
    
  | 22:53 | <Ryan_Lane> | adding all current labsconsole users to mediawiki shell group | [production] | 
            
  | 22:47 | <maxsem> | synchronized php-1.20wmf11/extensions/MobileFrontend/  'MF  update' | [production] | 
            
  | 22:46 | <maxsem> | synchronized php-1.20wmf10/extensions/MobileFrontend/  'MF  update' | [production] | 
            
  | 22:40 | <paravoid> | swift: weight 66->100 for ms-be9, 11, 12 | [production] | 
            
  | 21:40 | <Ryan_Lane> | added salt and dependent packages to lucid-wikimedia and precise-wikimedia repos | [production] | 
            
  | 21:39 | <maxsem> | synchronized php-1.20wmf11/extensions/MobileFrontend/api/ApiMobileView.php  'Bug 39489' | [production] | 
            
  | 19:42 | <mutante> | push change for glwiki logo to cluster (gerrit 23175) | [production] | 
            
  | 19:42 | <dzahn> | synchronized ./wmf-config/InitialiseSettings.php | [production] | 
            
  | 18:32 | <RobH> | authdns-update for wicipediacymraeg.org | [production] | 
            
  | 18:01 | <demon> | rebuilt wikiversions.cdb and synchronized wikiversions files: | [production] | 
            
  | 17:08 | <Jeff_Green> | adjusted locke:/etc/udp2log/squid to fix pattern match for fundraiser banners | [production] | 
            
  | 02:44 | <LocalisationUpdate> | completed (1.20wmf10) at Mon Sep 10 02:44:38 UTC 2012 | [production] | 
            
  | 02:26 | <LocalisationUpdate> | completed (1.20wmf11) at Mon Sep 10 02:26:50 UTC 2012 | [production] |