| 
      
        2016-04-05
      
      §
     | 
  
    
  | 19:57 | 
  <dduvall@tin> | 
  Started scap: testwiki to php-1.27.0-wmf.20 and rebuild l10n cache | 
  [production] | 
            
  | 19:31 | 
  <marxarelli> | 
  Applying security patches to wmf/1.27.0-wmf.20 checkout on tin | 
  [production] | 
            
  | 19:23 | 
  <marxarelli> | 
  Cloning mediawiki for checkout of new 1.27.0-wmf.20 branch on tin | 
  [production] | 
            
  | 18:11 | 
  <ejegg> | 
  updated payments wiki from cc298682f7ee9f43f5be863c6078e1c2ba70e3e6 to d50c1b15f3d09c4724d7ad28e3668368ac8112be | 
  [production] | 
            
  | 18:09 | 
  <marxarelli> | 
  Creating new wmf/1.27.0-wmf.20 branch on tin | 
  [production] | 
            
  | 17:00 | 
  <ejegg> | 
  updated payments wiki from a9659965d8b55b11518680a1170242f311c7f1d2  to cc298682f7ee9f43f5be863c6078e1c2ba70e3e6 | 
  [production] | 
            
  | 15:43 | 
  <godog> | 
  ms-be1019 to weight 3500 - T116842 | 
  [production] | 
            
  | 14:40 | 
  <oblivian@tin> | 
  Synchronized wmf-config/ProductionServices.php: make mediawiki talk to codfw restbase only (duration: 00m 47s) | 
  [production] | 
            
  | 14:10 | 
  <_joe_> | 
  external traffic for restbase, citoid, cxserver fully switched to codfw | 
  [production] | 
            
  | 14:05 | 
  <ema> | 
  repooling cp1043 (maps) after varnish upgrade to 4.1.2-1wm2. Bug: T131830 | 
  [production] | 
            
  | 13:58 | 
  <ema> | 
  depooling cp1043 (maps) for varnish upgrade to 4.1.2-1wm2. Bug: T131830 | 
  [production] | 
            
  | 13:52 | 
  <ema> | 
  repooling cp1044 (maps) after varnish upgrade to 4.1.2-1wm2. Bug: T131830 | 
  [production] | 
            
  | 13:49 | 
  <urandom> | 
  Throttling outbound stream throughput to 15Mbps on restbase2004-a.codfw.wmnet and restbase2003.codfw.wmnet : T95253 | 
  [production] | 
            
  | 13:40 | 
  <ema> | 
  depooling cp1044 (maps) for varnish upgrade to 4.1.2-1wm2. Bug: T131830 | 
  [production] | 
            
  | 13:29 | 
  <cmjohnson1> | 
  re-installing snapshot1007 with trusy | 
  [production] | 
            
  | 13:14 | 
  <elukey> | 
  restarted hhvm on mw1248 (hhvm was segfaulting) | 
  [production] | 
            
  | 10:16 | 
  <_joe_> | 
  all hhvm extensions uploaded for debian jessie as well | 
  [production] | 
            
  | 07:14 | 
  <_joe_> | 
  uploading the hhvm 3.12.1 backport package for jessie to reprepro | 
  [production] | 
            
  | 02:36 | 
  <l10nupdate@tin> | 
  ResourceLoader cache refresh completed at Tue Apr  5 02:36:21 UTC 2016 (duration 9m 19s) | 
  [production] | 
            
  | 02:27 | 
  <mwdeploy@tin> | 
  sync-l10n completed (1.27.0-wmf.19) (duration: 11m 55s) | 
  [production] | 
            
  | 01:29 | 
  <eileen> | 
  Updating CiviCRM from bfe563f2451725e734d6ddbddb3be716463a3512 to 4cc17b635eb84204cced107d5de78533cc5ce06c | 
  [production] | 
            
  | 00:14 | 
  <Krinkle> | 
  krinkle@terbium Running deleteEqualMessages.php over previously cleaned wikis with --lang-code (T45917) | 
  [production] | 
            
  | 00:12 | 
  <ori> | 
  renamed frontend.navtiming.loading -> frontend.navtiming.loadEventStart and frontend.navtiming.sending -> frontend.navtiming.fetchStart on graphite2001 and graphite1001 ahead of merging https://gerrit.wikimedia.org/r/#/c/281082 | 
  [production] | 
            
  
    | 
      
        2016-04-04
      
      §
     | 
  
    
  | 23:30 | 
  <maxsem@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/281565/ (duration: 00m 29s) | 
  [production] | 
            
  | 23:25 | 
  <maxsem@tin> | 
  Synchronized wmf-config/: https://gerrit.wikimedia.org/r/#/c/281070/ (duration: 00m 32s) | 
  [production] | 
            
  | 23:18 | 
  <maxsem@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/280445/ (duration: 00m 27s) | 
  [production] | 
            
  | 23:17 | 
  <maxsem@tin> | 
  Synchronized static/images/project-logos/astwiki.png: https://gerrit.wikimedia.org/r/#/c/280445/ (duration: 00m 27s) | 
  [production] | 
            
  | 23:13 | 
  <maxsem@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/281314/ (duration: 00m 28s) | 
  [production] | 
            
  | 23:08 | 
  <awight> | 
  update fundraising tool from 1bc23cb4bfaf2a9d4d215aad79dd67d891b5d973 to d8c5cc0b399411efa3b0634fa891a236c3dbaab2 | 
  [production] | 
            
  | 20:12 | 
  <subbu> | 
  finished deploying parsoid sha 579ec3e6 | 
  [production] | 
            
  | 20:07 | 
  <subbu> | 
  synced code; restarted parsoid on wtp1002 as a canary | 
  [production] | 
            
  | 20:04 | 
  <subbu> | 
  starting parsoid deploy | 
  [production] | 
            
  | 19:48 | 
  <ejegg> | 
  rolled payments-wiki back to a9659965d8b55b11518680a1170242f311c7f1d2 | 
  [production] | 
            
  | 19:40 | 
  <apergos> | 
  disabled puppet on snapshot1001,2,4  while new hosts come on line, til probably Apr 5-6 | 
  [production] | 
            
  | 19:38 | 
  <akosiaris> | 
  depool maps-test2004  | 
  [production] | 
            
  | 18:59 | 
  <ejegg> | 
  updated payments-wiki from a9659965d8b55b11518680a1170242f311c7f1d2 to eac5f979099056ef142500570555771dfb1992c1 | 
  [production] | 
            
  | 18:36 | 
  <tgr> | 
  ran MassMessages/sendMessages.php for T128056 | 
  [production] | 
            
  | 18:18 | 
  <cmjohnson1> | 
  stat1002 swapping failed disk slot 11 | 
  [production] | 
            
  | 18:04 | 
  <cmjohnson1> | 
  db1052 swapping failed disk slot 8 | 
  [production] | 
            
  | 17:37 | 
  <cmjohnson1> | 
  shutting down iridium to reapply thermal paste | 
  [production] | 
            
  | 17:30 | 
  <greg-g> | 
  Phabricator going down in about 10 minutes to hopefully address the overheating issue: T131742 | 
  [production] | 
            
  | 16:36 | 
  <urandom> | 
  Restarting bootstrap of restbase2004.codfw.wmnet : T95253 | 
  [production] | 
            
  | 15:53 | 
  <greg-g> | 
  15:45 <    elukey>  !log aqs1001 re-added to the aqs pool (nodejs NOT upgraded due to issues with Cassandra) | 
  [production] | 
            
  | 15:42 | 
  <Krenair> | 
  ran wikitech-static updates | 
  [production] | 
            
  | 15:18 | 
  <thcipriani@tin> | 
  Synchronized php-1.27.0-wmf.19/extensions/MobileApp/config/config.json: SWAT: Roll out RESTBase usage to Android production app: 50% [[gerrit:280957]] (duration: 00m 46s) | 
  [production] | 
            
  | 14:55 | 
  <urandom> | 
  Restarting restbase2004-a.codfw.wmnet (cancelling bootstrap of 2004-b) | 
  [production] | 
            
  | 14:46 | 
  <elukey> | 
  de-pooled aqs1001.eqiad from the confd pool for nodejs upgrade | 
  [production] | 
            
  | 13:40 | 
  <ema> | 
  nginx rolling restart for openssl upgrade on cache hosts | 
  [production] | 
            
  | 10:42 | 
  <elukey> | 
  re-pooled aqs1001.eqiad (no node upgrade, need more info about restbase) | 
  [production] | 
            
  | 10:23 | 
  <godog> | 
  reduce reserved blocks for /srv on restbase2004 | 
  [production] |