| 
      
        2017-03-28
      
      §
     | 
  
    
  | 14:41 | 
  <elukey@puppetmaster1001> | 
  conftool action : set/pooled=yes; selector: name=mw2256.codfw.wmnet | 
  [production] | 
            
  | 14:40 | 
  <marostegui> | 
  Convert dewiki UNIQUE keys into PK on db1091 (commonswiki) - T17441 | 
  [production] | 
            
  | 14:38 | 
  <ppchelko@tin> | 
  Finished deploy [changeprop/deploy@bfbaa17]: Increase log level for processinng failures (duration: 01m 07s) | 
  [production] | 
            
  | 14:38 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-eqiad.php: Depool db1091 - T17441 (duration: 00m 43s) | 
  [production] | 
            
  | 14:38 | 
  <elukey> | 
  ran restart-hhvm on mw1242, hhvm threads stuck (dump debug in /tmp/hhvm.9008.bt.) - HHVM 3.12 | 
  [production] | 
            
  | 14:37 | 
  <ppchelko@tin> | 
  Started deploy [changeprop/deploy@bfbaa17]: Increase log level for processinng failures | 
  [production] | 
            
  | 13:54 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-eqiad.php: Repool db1092 - T17441 (duration: 00m 43s) | 
  [production] | 
            
  | 13:44 | 
  <elukey> | 
  started hhvm on mw1261 (still depooled) - no hhvm process running | 
  [production] | 
            
  | 13:29 | 
  <RoanKattouw> | 
  Ran initUserPreference.php -s ores-enabled -t rcenhancedfilters   and -s ores-enabled -t oresHighlight  on plwiki and ptwiki | 
  [production] | 
            
  | 13:22 | 
  <catrope@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: Enable RCFilters beta feature on plwiki and ptwiki T158336 (duration: 00m 43s) | 
  [production] | 
            
  | 12:58 | 
  <moritzm> | 
  depooled mw1261 | 
  [production] | 
            
  | 10:39 | 
  <ema> | 
  upgrading twisted to 16.2.0 on lvs3001 and lvs3002 (esams primaries) T160433 | 
  [production] | 
            
  | 10:36 | 
  <ema> | 
  upgrading twisted to 16.2.0 on lvs3003 and lvs3004 (esams secondaries) T160433 | 
  [production] | 
            
  | 10:27 | 
  <marostegui> | 
  Convert dewiki UNIQUE keys into PK on db1092 - https://phabricator.wikimedia.org/T17441 | 
  [production] | 
            
  | 10:14 | 
  <elukey> | 
  Switching hue.w.o's backend (cache misc) from anaytics1027 to thorium - T159527 | 
  [production] | 
            
  | 10:10 | 
  <moritzm> | 
  upgraded mw1262 to HHVM 3.18 | 
  [production] | 
            
  | 08:48 | 
  <marostegui> | 
  Convert wikidatawiki UNIQUE keys into PK on db1092 - T17441 | 
  [production] | 
            
  | 08:48 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-eqiad.php: Depool db1092 - T17441 (duration: 00m 44s) | 
  [production] | 
            
  | 08:29 | 
  <akosiaris> | 
  enable IGMP snooping on all VLANs on asw2-d-eqiad. T133387 | 
  [production] | 
            
  | 07:19 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-eqiad.php: Repool db1089 - T17441 (duration: 00m 43s) | 
  [production] | 
            
  | 07:18 | 
  <moritzm> | 
  installing eject security updates on trusty hosts | 
  [production] | 
            
  | 06:11 | 
  <marostegui> | 
  Keep converting unique keys into PK on db1089 - T17441 | 
  [production] | 
            
  | 06:01 | 
  <marostegui> | 
  Deploy schema change on s2.enwiktionary.templatelinks - on codfw master, this will generate lag on codfw slaves (which have been silenced) - T154097 | 
  [production] | 
            
  | 05:52 | 
  <marostegui> | 
  Run pt-table-checksum on es2 - T161510 | 
  [production] | 
            
  | 02:39 | 
  <l10nupdate@tin> | 
  ResourceLoader cache refresh completed at Tue Mar 28 02:39:53 UTC 2017 (duration 5m 28s) | 
  [production] | 
            
  | 02:34 | 
  <l10nupdate@tin> | 
  scap sync-l10n completed (1.29.0-wmf.17) (duration: 12m 37s) | 
  [production] | 
            
  | 00:36 | 
  <reedy@tin> | 
  Synchronized private: Remove mwblocker.log (duration: 00m 44s) | 
  [production] | 
            
  | 00:34 | 
  <reedy@tin> | 
  Synchronized wmf-config/CommonSettings.php: Remove $wgProxyList (duration: 00m 43s) | 
  [production] | 
            
  
    | 
      
        2017-03-27
      
      §
     | 
  
    
  | 23:06 | 
  <ebernhardson@tin> | 
  Synchronized php-1.29.0-wmf.17/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT T160006, turning off cirrussearch AB test for sistersearch (duration: 00m 44s) | 
  [production] | 
            
  | 22:00 | 
  <bawolff> | 
  deployed patch T151735 | 
  [production] | 
            
  | 21:27 | 
  <andrewbogott> | 
  disabling puppet on labvirt* and labcontrol* to stagger roll out of https://gerrit.wikimedia.org/r/#/c/344689/ | 
  [production] | 
            
  | 20:29 | 
  <arlolra> | 
  Updated Parsoid to 6eaad376 (T160599, T161178, T133267) | 
  [production] | 
            
  | 20:21 | 
  <arlolra@tin> | 
  Finished deploy [parsoid/deploy@371ba4f]: Updating Parsoid to 6eaad376 (duration: 07m 06s) | 
  [production] | 
            
  | 20:14 | 
  <arlolra@tin> | 
  Started deploy [parsoid/deploy@371ba4f]: Updating Parsoid to 6eaad376 | 
  [production] | 
            
  | 19:57 | 
  <mutante> | 
  ruthenium/varnish misc - remove parsoid-tests.wikimedia.org server_name / backend - replaced by parsoid-rt-test and parsoid-vd-tests | 
  [production] | 
            
  | 19:56 | 
  <legoktm@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: Linter: whitelist parsoid canaries too - https://gerrit.wikimedia.org/r/#/c/344998/ - T160573 (duration: 00m 44s) | 
  [production] | 
            
  | 18:08 | 
  <mobrovac@tin> | 
  Finished deploy [mobileapps/deploy@92f693c]: Remove the proxy from the config, deploying to scb2004 (duration: 00m 43s) | 
  [production] | 
            
  | 18:07 | 
  <mobrovac@tin> | 
  Started deploy [mobileapps/deploy@92f693c]: Remove the proxy from the config, deploying to scb2004 | 
  [production] | 
            
  | 18:05 | 
  <mobrovac@tin> | 
  Finished deploy [mobileapps/deploy@92f693c]: Remove the proxy from the config (duration: 03m 29s) | 
  [production] | 
            
  | 18:02 | 
  <mobrovac@tin> | 
  Started deploy [mobileapps/deploy@92f693c]: Remove the proxy from the config | 
  [production] | 
            
  | 17:19 | 
  <mutante> | 
  tin/mira: welcome new mediawiki deployer 'musikanimal' (T161181) | 
  [production] | 
            
  | 17:03 | 
  <gehel@tin> | 
  Finished deploy [wdqs/wdqs@d07586c]: (no justification provided) (duration: 01m 26s) | 
  [production] | 
            
  | 17:02 | 
  <gehel@tin> | 
  Started deploy [wdqs/wdqs@d07586c]: (no justification provided) | 
  [production] | 
            
  | 16:40 | 
  <mobrovac> | 
  restbase deploying f53bec41 | 
  [production] | 
            
  | 16:34 | 
  <_joe_> | 
  cleaned the bc cache on mw1261, restarted hhvm and repooled | 
  [production] | 
            
  | 15:46 | 
  <mobrovac@tin> | 
  Finished deploy [mobileapps/deploy@aed916b]: Add discovery.wmnet to no_proxy_list (duration: 04m 05s) | 
  [production] | 
            
  | 15:42 | 
  <mobrovac@tin> | 
  Started deploy [mobileapps/deploy@aed916b]: Add discovery.wmnet to no_proxy_list | 
  [production] | 
            
  | 15:38 | 
  <mobrovac@tin> | 
  Finished deploy [cxserver/deploy@40e86ad]: Add discovery.wmnet to no_proxy_list (duration: 02m 39s) | 
  [production] | 
            
  | 15:35 | 
  <mobrovac@tin> | 
  Started deploy [cxserver/deploy@40e86ad]: Add discovery.wmnet to no_proxy_list | 
  [production] | 
            
  | 14:33 | 
  <dcausse> | 
  rebuilding ttmserver index in elastic@codfw from wasat | 
  [production] |