| 
      
        2017-06-15
      
      §
     | 
  
    
  | 23:37 | 
  <mutante> | 
  added stretch support for jenkins (https://gerrit.wikimedia.org/r/#/c/359227/, https://gerrit.wikimedia.org/r/#/c/359356/) | 'reprepro copy stretch-wikimedia jessie-wikimedia jenkins' to make .deb available on stretch | releases1001 now running jenkins , icinga recovered | (hashar) (T164030) | 
  [production] | 
            
  | 23:30 | 
  <mutante> | 
  APT - reprepro copy strech-wikimedia jessie-wikimedia jenkins (copy existing jenkins package to stretch, it can be used on both) | 
  [production] | 
            
  | 23:18 | 
  <ebernhardson@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: T166408: Remove dead config variable MinervaPrintStyles (duration: 00m 41s) | 
  [production] | 
            
  | 23:15 | 
  <ebernhardson@tin> | 
  Finished scap: wmf-config Scap: T162276: Enable crossproject search (duration: 03m 37s) | 
  [production] | 
            
  | 23:11 | 
  <ebernhardson@tin> | 
  Started scap: wmf-config Scap: T162276: Enable crossproject search | 
  [production] | 
            
  | 23:10 | 
  <ebernhardson@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: Scap: T162276: Enable crossproject search (duration: 00m 51s) | 
  [production] | 
            
  | 22:59 | 
  <mutante> | 
  mw2251 - repooled | 
  [production] | 
            
  | 22:56 | 
  <mutante> | 
  mw2251 - scap pull | 
  [production] | 
            
  | 22:53 | 
  <ebernhardson> | 
  restarting elasticsearch on relforge to pickup new ltr-query plugin | 
  [production] | 
            
  | 22:30 | 
  <ejegg> | 
  updated DjangoBannerStats from 9e6b11712851919e4318930e19bdf9c941d8ae38 to 5963e7c116f083ada28476c1c41ff1b46232d772 | 
  [production] | 
            
  | 22:02 | 
  <volans> | 
  restarted pdfrender on scb1001 T159922 | 
  [production] | 
            
  | 21:45 | 
  <mutante> | 
  powercycling mw2251 (frozen console) | 
  [production] | 
            
  | 21:39 | 
  <volans@puppetmaster1001> | 
  conftool action : set/pooled=inactive; selector: name=mw2251.codfw.wmnet | 
  [production] | 
            
  | 21:37 | 
  <volans> | 
  re-enabled puppet and force run to re-enable ircecho on einstenium | 
  [production] | 
            
  | 21:29 | 
  <demon@tin> | 
  Finished scap: Removing Cards extension (duration: 21m 49s) | 
  [production] | 
            
  | 21:08 | 
  <demon@tin> | 
  Started scap: Removing Cards extension | 
  [production] | 
            
  | 20:57 | 
  <mutante> | 
  upgrading RT (request tracker) | 
  [production] | 
            
  | 19:35 | 
  <demon@tin> | 
  rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.5 | 
  [production] | 
            
  | 19:22 | 
  <ladsgroup@tin> | 
  Finished deploy [ores/deploy@ab88a74]: Deploying gerrit:359224/1 for missing config variables (duration: 24m 15s) | 
  [production] | 
            
  | 19:17 | 
  <XioNoX> | 
  Re-enabled link between cr2-codfw and cr1-eqdfw - T167261 | 
  [production] | 
            
  | 18:58 | 
  <ladsgroup@tin> | 
  Started deploy [ores/deploy@ab88a74]: Deploying gerrit:359224/1 for missing config variables | 
  [production] | 
            
  | 18:44 | 
  <paravoid> | 
  restarting all puppetmasters | 
  [production] | 
            
  | 18:40 | 
  <paravoid> | 
  temporarily stopping icinga-wm | 
  [production] | 
            
  | 18:27 | 
  <demon@tin> | 
  Synchronized wmf-config/CirrusSearch-common.php: Remove quirks and enable token_count_router thingie (duration: 00m 44s) | 
  [production] | 
            
  | 18:16 | 
  <demon@tin> | 
  Synchronized php-1.30.0-wmf.5/includes/libs/objectcache/MultiWriteBagOStuff.php: T167465 (duration: 00m 44s) | 
  [production] | 
            
  | 18:14 | 
  <demon@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: T167617 (duration: 00m 44s) | 
  [production] | 
            
  | 18:12 | 
  <demon@tin> | 
  Synchronized wmf-config/FeaturedFeedsWMF.php: T167617 (duration: 00m 44s) | 
  [production] | 
            
  | 17:50 | 
  <mutante> | 
  install2002 - re-enabled puppet, reverted live hack, back to normal (issue seems to be NIC or other) | 
  [production] | 
            
  | 17:28 | 
  <mutante> | 
  install2002 - temp disabling puppet and applying hot fix to debug install issue for papaul | 
  [production] | 
            
  | 17:27 | 
  <bblack> | 
  disabling puppet on cp*wmnet to avoid puppet races on https://gerrit.wikimedia.org/r/#/c/341729 merge | 
  [production] | 
            
  | 14:39 | 
  <gehel> | 
  killing stuck replication on maps1001 | 
  [production] | 
            
  | 14:38 | 
  <krinkle@tin> | 
  Synchronized wmf-config/CommonSettings.php: no-op Ifc7b1ea80 - Remove EtcdConfig from beta (duration: 00m 45s) | 
  [production] | 
            
  | 13:24 | 
  <gehel> | 
  elasticsearch upgrade to 5.3.2 on relforge cluster completed, cluster still recovering - T163708 | 
  [production] | 
            
  | 13:23 | 
  <aude@tin> | 
  Synchronized wmf-config/Wikibase.php: Add constraints statements section on Wikidata T167126 (duration: 00m 43s) | 
  [production] | 
            
  | 13:19 | 
  <dcausse> | 
  [cirrus] reindexing all zh wikis (eqiad & codfw) | 
  [production] | 
            
  | 13:14 | 
  <aude@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: Enable BM25 for Chinese wikis (duration: 00m 44s) | 
  [production] | 
            
  | 13:13 | 
  <aude@tin> | 
  Synchronized tests/cirrusTest.php: (no justification provided) (duration: 00m 45s) | 
  [production] | 
            
  | 13:02 | 
  <gehel> | 
  starting elasticsearch upgrade to 5.3.2 on relforge cluster - T163708 | 
  [production] | 
            
  | 12:14 | 
  <gehel> | 
  restart elasticsearch on relforge1001 to validate latest config changes | 
  [production] | 
            
  | 10:16 | 
  <moritzm> | 
  rollout remaining systemd updates from jessie point release | 
  [production] | 
            
  | 09:14 | 
  <jynus> | 
  shutting down and deleting data at pc1004 for cloning from db1096 | 
  [production] | 
            
  | 09:10 | 
  <hashar> | 
  Jenkins back up and happy. | 
  [production] | 
            
  | 09:05 | 
  <moritzm> | 
  reenable puppet on notebook1002, was disabled for the merge of the zookeeper role refactor two days ago, can be re-enabled now | 
  [production] | 
            
  | 09:04 | 
  <hashar> | 
  Restarting Jenkins. It seems I managed to deadlock it | 
  [production] | 
            
  | 08:52 | 
  <ariel@tin> | 
  Finished deploy [dumps/dumps@1734c6d]: history dump rebalance script, fixup for extension script dumps, root logger for misc dumps (duration: 00m 02s) | 
  [production] | 
            
  | 08:52 | 
  <ariel@tin> | 
  Started deploy [dumps/dumps@1734c6d]: history dump rebalance script, fixup for extension script dumps, root logger for misc dumps | 
  [production] | 
            
  | 08:40 | 
  <gehel> | 
  restart relforge1001 to validate latest config changes | 
  [production] | 
            
  | 08:16 | 
  <akosiaris@tin> | 
  Finished deploy [citoid/deploy@ba0db9c]: Remove the bad PMCID test from spec (duration: 07m 44s) | 
  [production] |