| 
      
        2013-03-14
      
      §
     | 
  
    
  | 20:30 | 
  <ori-l> | 
  Added article edit event to EventLogging, which upped events/sec from ~3.5 to ~30. Expect increased load on vanadium and db1047; bits caches unaffected. | 
  [production] | 
            
  | 20:20 | 
  <olivneh> | 
  synchronized php-1.21wmf11/extensions/EventLogging  'Log ['HTTP_HOST'] as webHost' | 
  [production] | 
            
  | 20:05 | 
  <olivneh> | 
  synchronized php-1.21wmf11/includes/WikiPage.php  'Adds CategoryAfterPageAdded / CategoryAfterPageRemoved hooks' | 
  [production] | 
            
  | 19:32 | 
  <aaron> | 
  synchronized php-1.21wmf11/includes/job/jobs  'deployed 6f76ede163cb114724bce7a1c1b8938e1e30606f ' | 
  [production] | 
            
  | 18:49 | 
  <aaron> | 
  synchronized wmf-config/InitialiseSettings.php  'Enable wgEnableAsyncUploads only on testwikis for now' | 
  [production] | 
            
  | 18:44 | 
  <mlitn> | 
  synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/  'Update ArticleFeedbackv5 to master' | 
  [production] | 
            
  | 18:41 | 
  <asher> | 
  synchronized wmf-config/db-eqiad.php  'returning db1009 to full weight' | 
  [production] | 
            
  | 18:31 | 
  <asher> | 
  synchronized wmf-config/db-eqiad.php  'returning db1009 at a low weight for warmup' | 
  [production] | 
            
  | 18:12 | 
  <aaron> | 
  synchronized wmf-config/InitialiseSettings.php  'Enabled $wgEnableAsyncUploads on all wikis' | 
  [production] | 
            
  | 18:08 | 
  <binasher> | 
  cleaned up myisam cruft (unused t���ables) on s2 | 
  [production] | 
            
  | 17:49 | 
  <aaron> | 
  synchronized wmf-config  'Enabled redis job aggregator fully' | 
  [production] | 
            
  | 17:47 | 
  <cmjohnson1> | 
  labsdb1001 powering down | 
  [production] | 
            
  | 17:42 | 
  <aaron> | 
  synchronized wmf-config  'Set redis job aggregator on for testwiki.' | 
  [production] | 
            
  | 17:27 | 
  <reedy> | 
  synchronized php-1.21wmf11/extensions/Math/ | 
  [production] | 
            
  | 17:23 | 
  <binasher> | 
  shutting down mysql on db1009, upgrading to mariadb | 
  [production] | 
            
  | 16:59 | 
  <cmjohnson1> | 
  powering down labsdb1002 and labsdb1003 | 
  [production] | 
            
  | 16:59 | 
  <asher> | 
  synchronized wmf-config/db-eqiad.php  'pulling db1009 for mariadb migration' | 
  [production] | 
            
  | 16:57 | 
  <cmjohnson1> | 
  addin mw1041 back to dsh groups | 
  [production] | 
            
  | 16:26 | 
  <mlitn> | 
  synchronized wmf-config  'Prepare AFTv5 config for deployment new features' | 
  [production] | 
            
  | 13:15 | 
  <mark> | 
  Set OSPF cost to 10 on csw1-esams:ve7 to facilitate ip multipath | 
  [production] | 
            
  | 13:15 | 
  <mark> | 
  Fixed OSPF on cr2-knams:xe-0/0/0 - csw1-esams:e8/2 (earlier) | 
  [production] | 
            
  | 12:16 | 
  <hashar> | 
  gallium setting up /srv/ as a copy of integration/docroot.git | 
  [production] | 
            
  | 09:03 | 
  <apergos> | 
  restarted varnishncsa.vanadium on niobium, it was using 9gb memory or so | 
  [production] | 
            
  | 02:25 | 
  <LocalisationUpdate> | 
  completed (1.21wmf11) at Thu Mar 14 02:25:19 UTC 2013 | 
  [production] | 
            
  | 02:20 | 
  <hashar> | 
  jenkins: made Zuul to block changes on operations/debs/ircecho whenever pep8/pyflakes fails. | 
  [production] | 
            
  | 01:35 | 
  <mutante> | 
  running puppet on celsus to investigate puppet freshness reports from icinga - it started parsoid, but also celsus is a Wikimedia DECOMMISSIONED server (base::decommissioned). | 
  [production] | 
            
  | 01:10 | 
  <reedy> | 
  synchronized php-1.21wmf11/cache/l10n/ | 
  [production] | 
            
  | 00:59 | 
  <reedy> | 
  synchronized php-1.21wmf11/extensions/Scribunto | 
  [production] | 
            
  | 00:53 | 
  <reedy> | 
  synchronized wmf-config/ | 
  [production] | 
            
  
    | 
      
        2013-03-13
      
      §
     | 
  
    
  | 23:43 | 
  <mutante> | 
  installing package upgrades on zirconium | 
  [production] | 
            
  | 23:41 | 
  <demon> | 
  Finished syncing Wikimedia installation... : | 
  [production] | 
            
  | 23:38 | 
  <Ryan_Lane> | 
  deploying change 53464 to OpenStackManager on wikitech | 
  [production] | 
            
  | 23:23 | 
  <RobH> | 
  no one sync apaches, we wre doing crazy magic on them | 
  [production] | 
            
  | 23:17 | 
  <demon> | 
  Started syncing Wikimedia installation... : | 
  [production] | 
            
  | 23:05 | 
  <py> | 
  synchronized wmf-config/lucene-production.php  'moving all search traffic back to eqiad. upgrades over.' | 
  [production] | 
            
  | 23:00 | 
  <py> | 
  synchronized wmf-config/lucene-production.php  'moving all search traffic back to eqiad. upgrades over.' | 
  [production] | 
            
  | 22:53 | 
  <reedy> | 
  synchronized wmf-config/ | 
  [production] | 
            
  | 22:44 | 
  <demon> | 
  synchronized php-1.21wmf11/extensions/Scribunto/  'Updating Scribunto to master' | 
  [production] | 
            
  | 22:30 | 
  <robh> | 
  gracefulled all apaches | 
  [production] | 
            
  | 21:46 | 
  <awight> | 
  payments cluster updated to 61d3f1f | 
  [production] | 
            
  | 21:43 | 
  <pp-pdf1> | 
  upgraded mwlib to 0.15.3 | 
  [production] | 
            
  | 21:38 | 
  <Tim> | 
  on professor: restarted collector | 
  [production] | 
            
  | 21:21 | 
  <RobH> | 
  repooling all ssl servers (except broken ssl3004) in all sites.  new unified certificate is now in use. | 
  [production] | 
            
  | 21:15 | 
  <RobH> | 
  depooling ssl servers 3 and 4 in each dc site | 
  [production] | 
            
  | 21:14 | 
  <RobH> | 
  repooling the first two ssl servers per site | 
  [production] | 
            
  | 21:08 | 
  <py> | 
  synchronized wmf-config/lucene-production.php  'temp moving all earch traffic to pmtpa for upgrades in eqiad' | 
  [production] | 
            
  | 21:05 | 
  <RobH> | 
  setting first 2 ssl servers per dc site to false in pybal | 
  [production] | 
            
  | 20:21 | 
  <Ryan_Lane> | 
  repooling ssl1001 | 
  [production] | 
            
  | 20:20 | 
  <reedy> | 
  synchronized wmf-config/InitialiseSettings.php  'Re-enable collection' | 
  [production] | 
            
  | 20:16 | 
  <Ryan_Lane> | 
  depooling ssl1001 | 
  [production] |