| 2016-04-05
      
      § | 
    
  | 21:37 | <cscott> | restarted parsoid on wtp2001.codfw.wmnet as a (better) canary | [production] | 
            
  | 21:34 | <cscott> | restarted parsoid on wtp1001.eqiad.wmnet as a canary | [production] | 
            
  | 21:34 | <cscott> | updated Parsoid to version a5be1cdc | [production] | 
            
  | 21:34 | <bd808> | https://tools.wmflabs.org/sal missing entries since 2016-04-04T16:36 | [production] | 
            
  | 20:36 | <dduvall@tin> | Purged l10n cache for 1.27.0-wmf.18 | [production] | 
            
  | 20:32 | <dduvall@tin> | rebuilt wikiversions.php and synchronized wikiversions files: Group0 to 1.27.0-wmf.20 | [production] | 
            
  | 20:29 | <dduvall@tin> | Finished scap: testwiki to php-1.27.0-wmf.20 and rebuild l10n cache (duration: 32m 19s) | [production] | 
            
  | 19:57 | <dduvall@tin> | Started scap: testwiki to php-1.27.0-wmf.20 and rebuild l10n cache | [production] | 
            
  | 19:31 | <marxarelli> | Applying security patches to wmf/1.27.0-wmf.20 checkout on tin | [production] | 
            
  | 19:23 | <marxarelli> | Cloning mediawiki for checkout of new 1.27.0-wmf.20 branch on tin | [production] | 
            
  | 18:11 | <ejegg> | updated payments wiki from cc298682f7ee9f43f5be863c6078e1c2ba70e3e6 to d50c1b15f3d09c4724d7ad28e3668368ac8112be | [production] | 
            
  | 18:09 | <marxarelli> | Creating new wmf/1.27.0-wmf.20 branch on tin | [production] | 
            
  | 17:00 | <ejegg> | updated payments wiki from a9659965d8b55b11518680a1170242f311c7f1d2  to cc298682f7ee9f43f5be863c6078e1c2ba70e3e6 | [production] | 
            
  | 15:43 | <godog> | ms-be1019 to weight 3500 - T116842 | [production] | 
            
  | 14:40 | <oblivian@tin> | Synchronized wmf-config/ProductionServices.php: make mediawiki talk to codfw restbase only (duration: 00m 47s) | [production] | 
            
  | 14:10 | <_joe_> | external traffic for restbase, citoid, cxserver fully switched to codfw | [production] | 
            
  | 14:05 | <ema> | repooling cp1043 (maps) after varnish upgrade to 4.1.2-1wm2. Bug: T131830 | [production] | 
            
  | 13:58 | <ema> | depooling cp1043 (maps) for varnish upgrade to 4.1.2-1wm2. Bug: T131830 | [production] | 
            
  | 13:52 | <ema> | repooling cp1044 (maps) after varnish upgrade to 4.1.2-1wm2. Bug: T131830 | [production] | 
            
  | 13:49 | <urandom> | Throttling outbound stream throughput to 15Mbps on restbase2004-a.codfw.wmnet and restbase2003.codfw.wmnet : T95253 | [production] | 
            
  | 13:40 | <ema> | depooling cp1044 (maps) for varnish upgrade to 4.1.2-1wm2. Bug: T131830 | [production] | 
            
  | 13:29 | <cmjohnson1> | re-installing snapshot1007 with trusy | [production] | 
            
  | 13:14 | <elukey> | restarted hhvm on mw1248 (hhvm was segfaulting) | [production] | 
            
  | 10:16 | <_joe_> | all hhvm extensions uploaded for debian jessie as well | [production] | 
            
  | 07:14 | <_joe_> | uploading the hhvm 3.12.1 backport package for jessie to reprepro | [production] | 
            
  | 02:36 | <l10nupdate@tin> | ResourceLoader cache refresh completed at Tue Apr  5 02:36:21 UTC 2016 (duration 9m 19s) | [production] | 
            
  | 02:27 | <mwdeploy@tin> | sync-l10n completed (1.27.0-wmf.19) (duration: 11m 55s) | [production] | 
            
  | 01:29 | <eileen> | Updating CiviCRM from bfe563f2451725e734d6ddbddb3be716463a3512 to 4cc17b635eb84204cced107d5de78533cc5ce06c | [production] | 
            
  | 00:14 | <Krinkle> | krinkle@terbium Running deleteEqualMessages.php over previously cleaned wikis with --lang-code (T45917) | [production] | 
            
  | 00:12 | <ori> | renamed frontend.navtiming.loading -> frontend.navtiming.loadEventStart and frontend.navtiming.sending -> frontend.navtiming.fetchStart on graphite2001 and graphite1001 ahead of merging https://gerrit.wikimedia.org/r/#/c/281082 | [production] | 
            
  
    | 2016-04-04
      
      § | 
    
  | 23:30 | <maxsem@tin> | Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/281565/ (duration: 00m 29s) | [production] | 
            
  | 23:25 | <maxsem@tin> | Synchronized wmf-config/: https://gerrit.wikimedia.org/r/#/c/281070/ (duration: 00m 32s) | [production] | 
            
  | 23:18 | <maxsem@tin> | Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/280445/ (duration: 00m 27s) | [production] | 
            
  | 23:17 | <maxsem@tin> | Synchronized static/images/project-logos/astwiki.png: https://gerrit.wikimedia.org/r/#/c/280445/ (duration: 00m 27s) | [production] | 
            
  | 23:13 | <maxsem@tin> | Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/281314/ (duration: 00m 28s) | [production] | 
            
  | 23:08 | <awight> | update fundraising tool from 1bc23cb4bfaf2a9d4d215aad79dd67d891b5d973 to d8c5cc0b399411efa3b0634fa891a236c3dbaab2 | [production] | 
            
  | 20:12 | <subbu> | finished deploying parsoid sha 579ec3e6 | [production] | 
            
  | 20:07 | <subbu> | synced code; restarted parsoid on wtp1002 as a canary | [production] | 
            
  | 20:04 | <subbu> | starting parsoid deploy | [production] | 
            
  | 19:48 | <ejegg> | rolled payments-wiki back to a9659965d8b55b11518680a1170242f311c7f1d2 | [production] | 
            
  | 19:40 | <apergos> | disabled puppet on snapshot1001,2,4  while new hosts come on line, til probably Apr 5-6 | [production] | 
            
  | 19:38 | <akosiaris> | depool maps-test2004 | [production] | 
            
  | 18:59 | <ejegg> | updated payments-wiki from a9659965d8b55b11518680a1170242f311c7f1d2 to eac5f979099056ef142500570555771dfb1992c1 | [production] | 
            
  | 18:36 | <tgr> | ran MassMessages/sendMessages.php for T128056 | [production] | 
            
  | 18:18 | <cmjohnson1> | stat1002 swapping failed disk slot 11 | [production] | 
            
  | 18:04 | <cmjohnson1> | db1052 swapping failed disk slot 8 | [production] | 
            
  | 17:37 | <cmjohnson1> | shutting down iridium to reapply thermal paste | [production] | 
            
  | 17:30 | <greg-g> | Phabricator going down in about 10 minutes to hopefully address the overheating issue: T131742 | [production] | 
            
  | 16:36 | <urandom> | Restarting bootstrap of restbase2004.codfw.wmnet : T95253 | [production] | 
            
  | 15:53 | <greg-g> | 15:45 <    elukey>  !log aqs1001 re-added to the aqs pool (nodejs NOT upgraded due to issues with Cassandra) | [production] |