| 2013-05-02
      
      § | 
    
  | 16:59 | <Krinkle> | Zuul is somehow having trouble kicking off Jenkins jobs (less than 1 event processed per minute). Jenkins shows that 10/10 executors are idle. Investigating... | [production] | 
            
  | 16:52 | <nikerabbit> | synchronized php-1.22wmf3/extensions/UniversalLanguageSelector/  'ULS to master' | [production] | 
            
  | 15:12 | <hashar> | Jenkins made Jenkins to instantly restart (was {{bug|47120}}) ) by deleting the downstream-buildview  plugin. | [production] | 
            
  | 15:11 | <hashar> | restarted Zuul | [production] | 
            
  | 14:11 | <hashar> | Upgraded Jenkins from 1.480.3 to LTS 1.509.1.  Restarted it. | [production] | 
            
  | 14:09 | <hashar> | shutting down Zuul | [production] | 
            
  | 14:08 | <hashar> | upgrading Jenkins (unscheduled maintenance). | [production] | 
            
  | 13:01 | <maxsem> | synchronized wmf-config/mobile.php  'https://gerrit.wikimedia.org/r/#/c/61941/' | [production] | 
            
  | 12:58 | <mark> | Migrated amslvs3 and amslvs4 PyBal BGP peerings from csw2-esams to cr2-knams | [production] | 
            
  | 09:54 | <maxsem> | synchronized php-1.22wmf2/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php  'https://gerrit.wikimedia.org/r/#/c/61926/' | [production] | 
            
  | 09:53 | <maxsem> | synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php  'https://gerrit.wikimedia.org/r/#/c/61926/' | [production] | 
            
  | 09:50 | <maxsem> | synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php  'https://gerrit.wikimedia.org/r/#/c/61926/' | [production] | 
            
  | 09:39 | <maxsem> | synchronized php-1.22wmf3/extensions/GeoData/GeoData.body.php  'https://gerrit.wikimedia.org/r/#/c/61962/' | [production] | 
            
  | 06:03 | <apergos> | powercycle snapshot1001, upgrade to precise | [production] | 
            
  | 05:45 | <apergos> | powercycle snapshot1004, upgrade to precise | [production] | 
            
  | 05:27 | <apergos> | snapshot1003 powercucle, upgrading to precise | [production] | 
            
  | 05:27 | <Tim> | on professor: manually started carbon-cache.py | [production] | 
            
  | 05:14 | <Tim> | professor is down, no response on serial console, rebooting | [production] | 
            
  | 05:11 | <tstarling> | synchronized wmf-config/jobqueue-pmtpa.php | [production] | 
            
  | 05:07 | <tstarling> | synchronized wmf-config/jobqueue-eqiad.php | [production] | 
            
  | 04:46 | <Tim> | on rdb1001-1002, set requirepass and masterauth in soft state | [production] | 
            
  | 04:44 | <Tim> | on mc1-16 and mc1001-1016, setting requirepass and masterauth to the new password in soft state | [production] | 
            
  | 04:39 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Thu May  2 04:39:48 UTC 2013 | [production] | 
            
  | 04:03 | <tstarling> | synchronized wmf-config/CommonSettings.php | [production] | 
            
  | 03:59 | <tstarling> | synchronized wmf-config/CommonSettings.php | [production] | 
            
  | 03:56 | <tstarling> | synchronized wmf-config/jobqueue-eqiad.php | [production] | 
            
  | 03:46 | <tstarling> | synchronized php-1.22wmf2/extensions/GettingStarted | [production] | 
            
  | 03:39 | <tstarling> | synchronized php-1.22wmf3/extensions/GettingStarted | [production] | 
            
  | 02:45 | <kaldari> | Finished syncing Wikimedia installation... : | [production] | 
            
  | 02:32 | <LocalisationUpdate> | completed (1.22wmf2) at Thu May  2 02:32:10 UTC 2013 | [production] | 
            
  | 02:21 | <LocalisationUpdate> | completed (1.22wmf3) at Thu May  2 02:21:30 UTC 2013 | [production] | 
            
  | 02:20 | <Tim> | on wikibugs-l: disabled bounce processing and re-enabled mail delivery to wikibugs-irc (was disabled due to excessive bounces) | [production] | 
            
  | 02:20 | <kaldari> | Started syncing Wikimedia installation... : | [production] | 
            
  | 01:20 | <mwalker> | synchronized php-1.22wmf3/extensions/CentralNotice/modules/ext.centralNotice.bannerController/bannerController.js  'Poking bits to try and get the new banner controller deployed for CentralNotice' | [production] | 
            
  | 01:17 | <mwalker> | synchronized php-1.22wmf2/extensions/CentralNotice/modules/ext.centralNotice.bannerController/bannerController.js  'Poking bits to try and get the new banner controller deployed for CentralNotice' | [production] | 
            
  | 00:15 | <mutante> | @stafford:~# puppetstoredconfigclean.rb db10.pmtpa.wmnet | [production] | 
            
  
    | 2013-05-01
      
      § | 
    
  | 23:28 | <tstarling> | synchronized php-1.22wmf3/extensions/GettingStarted | [production] | 
            
  | 23:23 | <tstarling> | synchronized php-1.22wmf2/extensions/GettingStarted | [production] | 
            
  | 23:18 | <tstarling> | synchronized php-1.22wmf2/extensions/GettingStarted | [production] | 
            
  | 22:42 | <mutante> | powercycling frozen mw1041 | [production] | 
            
  | 22:13 | <mwalker> | Finished syncing Wikimedia installation... : Updating CentralNotice from a8797938bc for historical allocations and some refactoring | [production] | 
            
  | 22:06 | <dzahn> | synchronized ./multiversion/MWMultiVersion.php | [production] | 
            
  | 22:02 | <dzahn> | synchronized docroot | [production] | 
            
  | 21:43 | <mwalker> | Started syncing Wikimedia installation... : Updating CentralNotice from a8797938bc for historical allocations and some refactoring | [production] | 
            
  | 21:32 | <dzahn> | gracefulled all apaches | [production] | 
            
  | 21:25 | <ottomata> | syncing new redirects.conf to apachces to redirect wikipediazero.org | [production] | 
            
  | 21:13 | <Jeff_Green> | stopping mysql on db78 for cloning | [production] | 
            
  | 20:49 | <ottomata> | adding new wikipediazero.org zone alias, running authdns-update on dobson | [production] | 
            
  | 19:44 | <reedy> | synchronized php-1.22wmf3/cache/l10n/ | [production] | 
            
  | 19:15 | <mutante> | DNS update - remove project2, anthony and yongle :p | [production] |