| 
      
        2016-10-19
      
      §
     | 
  
    
  | 13:21 | 
  <mobrovac> | 
  change-prop stopping instances on scb100[12] so that scb1003 picks up more load | 
  [production] | 
            
  | 13:21 | 
  <dereckson@mira> | 
  Synchronized wmf-config/InitialiseSettings.php: Create a new namespace "Príloha" for skwikt (T148563) (duration: 00m 50s) | 
  [production] | 
            
  | 13:14 | 
  <mobrovac> | 
  change-prop stopping instance on scb1004 so that scb1004 picks up more load | 
  [production] | 
            
  | 13:14 | 
  <dereckson@mira> | 
  Synchronized wmf-config/InitialiseSettings.php: Create a 'templateeditor' user group at en.wiktionary plus adittional configuration (T148007) (duration: 02m 33s) | 
  [production] | 
            
  | 13:11 | 
  <ema> | 
  stopping varnishlog service on v4 cp hosts and removing log file | 
  [production] | 
            
  | 12:52 | 
  <bblack> | 
  upgrading nginx on codfw text+upload caches - T144523 | 
  [production] | 
            
  | 12:24 | 
  <marostegui> | 
  Stopping db2055 to clone another host - T146261 | 
  [production] | 
            
  | 12:17 | 
  <akosiaris> | 
  update cr{1,2}-eqiad configuration to add tegmen+einsteinium | 
  [production] | 
            
  | 12:01 | 
  <akosiaris@puppetmaster1001> | 
  conftool action : set/pooled=yes; selector: scb2004.codfw.wmnet (tags: ['dc=codfw', 'cluster=scb', 'service=ores']) | 
  [production] | 
            
  | 12:00 | 
  <akosiaris@puppetmaster1001> | 
  conftool action : set/pooled=yes; selector: scb2003.codfw.wmnet (tags: ['dc=codfw', 'cluster=scb', 'service=ores']) | 
  [production] | 
            
  | 12:00 | 
  <akosiaris@puppetmaster1001> | 
  conftool action : set/pooled=yes; selector: scb1004.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=ores']) | 
  [production] | 
            
  | 12:00 | 
  <akosiaris@puppetmaster1001> | 
  conftool action : set/pooled=yes; selector: scb1003.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=ores']) | 
  [production] | 
            
  | 11:55 | 
  <marostegui> | 
  Deploying schema change db2055 - S1 enwiki.change_tag - T147166 | 
  [production] | 
            
  | 11:35 | 
  <bblack> | 
  upgrading nginx on cp2002 (codfw upload canary) - T144523 | 
  [production] | 
            
  | 11:30 | 
  <bblack> | 
  upgrading nginx on cp2001 (codfw text canary) - T144523 | 
  [production] | 
            
  | 10:27 | 
  <akosiaris@puppetmaster1001> | 
  conftool action : set/pooled=no; selector: scb1004.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=mobileapps']) | 
  [production] | 
            
  | 10:27 | 
  <akosiaris@puppetmaster1001> | 
  conftool action : set/pooled=no; selector: scb1003.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=mobileapps']) | 
  [production] | 
            
  | 10:05 | 
  <_joe_> | 
  converting owner of files for l10nupdate usermod on tin | 
  [production] | 
            
  | 10:02 | 
  <_joe_> | 
  ran usermod -u 10002 l10nupdate on tin | 
  [production] | 
            
  | 09:50 | 
  <marostegui@mira> | 
  Synchronized wmf-config/db-eqiad.php: Repool db1064 after finishing the ALTER table - T147305 (duration: 01m 08s) | 
  [production] | 
            
  | 09:22 | 
  <moritzm> | 
  installing rsyslog bugfix updates | 
  [production] | 
            
  | 09:20 | 
  <marostegui> | 
  Deploying schema change on db1069 S4 instance commonswiki revision table - T147305 | 
  [production] | 
            
  | 08:15 | 
  <marostegui> | 
  Stopping db2062.codfw.wmnet to use it to clone another server - T146261 | 
  [production] | 
            
  | 08:14 | 
  <moritzm> | 
  installing tor security update on radium | 
  [production] | 
            
  | 08:12 | 
  <moritzm> | 
  installing quagga security updates | 
  [production] | 
            
  | 07:31 | 
  <_joe_> | 
  disabled profiling on mw1189, hhvm keeps crashing | 
  [production] | 
            
  | 06:50 | 
  <_joe_> | 
  installing jemalloc with memory profiling enabled on mw1189 | 
  [production] | 
            
  
    | 
      
        2016-10-18
      
      §
     | 
  
    
  | 23:04 | 
  <Dereckson> | 
  This full scap pulled three changes of the EU SWAT: [[gerrit:316069] TimedMediaHandler, [[gerrit:316585]] MobileFrontend, [[gerrit:315901]] ULS | 
  [production] | 
            
  | 23:03 | 
  <demon@mira> | 
  Finished scap: bringing full cluster back into sync (duration: 25m 13s) | 
  [production] | 
            
  | 22:38 | 
  <demon@mira> | 
  Started scap: bringing full cluster back into sync | 
  [production] | 
            
  | 22:28 | 
  <demon@mira> | 
  Synchronized README: Bringing co-masters back in sync (duration: 13m 10s) | 
  [production] | 
            
  | 21:37 | 
  <mutante> | 
  added Dpatrick to WMF LDAP group | 
  [production] | 
            
  | 18:32 | 
  <dereckson@mira> | 
  Synchronized wmf-config/LabsServices.php: Elastic@deployment-prep: Remove deployment-elastic08 from the cluster (no-op in prod, labs only) (duration: 00m 47s) | 
  [production] | 
            
  | 18:30 | 
  <dereckson@mira> | 
  Synchronized wmf-config/CirrusSearch-labs.php: Elastic@deployment-prep: force the number of replicas to 1 max (no-op in prod, labs only) (duration: 01m 18s) | 
  [production] | 
            
  | 17:55 | 
  <dcausse> | 
  warming up elastic@codfw from wasat.codfw.wmnet | 
  [production] | 
            
  | 17:34 | 
  <jynus> | 
  stopping mysql, cloning db1064->db1053; upgrading | 
  [production] | 
            
  | 17:01 | 
  <bblack> | 
  upgrading nginx on cache_maps - T144523 | 
  [production] | 
            
  | 16:57 | 
  <ejegg> | 
  updated payments-wiki from b4ad60e739b9dbb97f08a3623db961a74682422a to 27b464fd4383647fc2e7f0a613f290d6edccd22f | 
  [production] | 
            
  | 15:47 | 
  <godog> | 
  eqiad-prod: ms-be1022 to weight 3000 T136631 | 
  [production] | 
            
  | 15:16 | 
  <andrewbogott> | 
  upgrading puppetmaster on labtestcontrol2001 to trusty/3.8.5 | 
  [production] | 
            
  | 15:06 | 
  <bblack> | 
  upgrading nginx on all remaining cache_misc (eqiad, esams) - T144523 | 
  [production] | 
            
  | 14:54 | 
  <bblack> | 
  upgrading nginx on all cache_misc @ codfw - T144523 | 
  [production] | 
            
  | 14:54 | 
  <chasemp> | 
  rsync tools from labstore1001 to labstore1004 | 
  [production] | 
            
  | 14:43 | 
  <bblack> | 
  upgrading nginx on all cache_misc @ ulsfo - T144523 | 
  [production] | 
            
  | 14:40 | 
  <marostegui> | 
  Shutting down es2015 for hardware maintenance - T147769 | 
  [production] | 
            
  | 14:21 | 
  <bblack> | 
  upgrading nginx on cp4001 (cache_misc ulsfo) as prod canary | 
  [production] | 
            
  | 14:18 | 
  <bblack> | 
  uploading nginx-1.11.4+wmf3 to carbon jessie-wikimedia - T144523 | 
  [production] | 
            
  | 13:58 | 
  <jynus> | 
  restarting and upgrading db2049 and es2019 to test new config | 
  [production] | 
            
  | 13:53 | 
  <jynus> | 
  applying new init.d script on all mariadb 10 servers | 
  [production] | 
            
  | 12:52 | 
  <elukey> | 
  mw1169 back in service after reimage (MW Jobrunner) | 
  [production] |