| 
      
        2016-08-04
      
      §
     | 
  
    
  | 19:51 | 
  <awight> | 
  paymentswiki config fix | 
  [production] | 
            
  | 19:47 | 
  <otto@palladium> | 
  conftool action : set/pooled=yes; selector: kafka2002.codfw.wmnet | 
  [production] | 
            
  | 19:46 | 
  <akosiaris> | 
  restart puppetmaster on palladium to activate loadfactor change. puppet related icinga spam will ensue* | 
  [production] | 
            
  | 19:46 | 
  <awight> | 
  enabling paymentswiki queue mirroring | 
  [production] | 
            
  | 19:46 | 
  <akosiaris> | 
  restart puppetmaster on palladium to activate loadfactor change. puppet related icinga spam will ensure | 
  [production] | 
            
  | 19:45 | 
  <otto@palladium> | 
  conftool action : set/pooled=no; selector: kafka2002.codfw.wmnet | 
  [production] | 
            
  | 19:43 | 
  <otto@palladium> | 
  conftool action : set/pooled=yes; selector: kafka2001.codfw.wmnet | 
  [production] | 
            
  | 19:42 | 
  <urandom> | 
  T140825,T140869: Restarting Cassandra, restbase1010-c.eqiad.wmnet | 
  [production] | 
            
  | 19:40 | 
  <otto@palladium> | 
  conftool action : set/pooled=no; selector: kafka2001.codfw.wmnet | 
  [production] | 
            
  | 19:40 | 
  <urandom> | 
  T140825,T140869: Restarting Cassandra, restbase1010-b.eqiad.wmnet | 
  [production] | 
            
  | 19:39 | 
  <ottomata> | 
  deploying eventlogging-service-eventbus to codfw hosts, depooling and pooling | 
  [production] | 
            
  | 19:37 | 
  <urandom> | 
  T140825,T140869: Restarting Cassandra, restbase1010-a.eqiad.wmnet | 
  [production] | 
            
  | 19:06 | 
  <thcipriani@tin> | 
  rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.28.0-wmf.13 | 
  [production] | 
            
  | 18:49 | 
  <thcipriani@tin> | 
  Synchronized README: test logstash host (duration: 00m 51s) | 
  [production] | 
            
  | 18:36 | 
  <cwd> | 
  updated payments from 3a724bfb1a3e20e17b5886dae0ba7572020abd6b to b737b60c87da82543ab812ece4611c68af01307f | 
  [production] | 
            
  | 18:12 | 
  <jynus@tin> | 
  Synchronized wmf-config/db-eqiad.php: Pool db1074 as backup rc node in case db1036 lags (duration: 00m 25s) | 
  [production] | 
            
  | 18:12 | 
  <awight> | 
  update paymentswiki config to 71cc55194f4465600ce4da0ea9f7dfaefdda5479 | 
  [production] | 
            
  | 17:36 | 
  <elukey> | 
  added the analytics-deploy key to the Keyholder for the Analytics Refinery scap3 migration (also updated https://wikitech.wikimedia.org/wiki/Keyholder) | 
  [production] | 
            
  | 17:18 | 
  <ostriches> | 
  gerrit: Restarting really quick, trying alternative mysql library. | 
  [production] | 
            
  | 17:17 | 
  <urandom> | 
  T140825,T140869: Restarting Cassandra, restbase1007-c.eqiad.wmnet | 
  [production] | 
            
  | 17:14 | 
  <urandom> | 
  T140825,T140869: Restarting Cassandra, restbase1007-b.eqiad.wmnet | 
  [production] | 
            
  | 17:08 | 
  <urandom> | 
  T140825,T140869: Restarting Cassandra, restbase1007-a.eqiad.wmnet | 
  [production] | 
            
  | 17:07 | 
  <elukey> | 
  restarting keyholder-proxy on tin to let the new analytics key to be picked up | 
  [production] | 
            
  | 15:08 | 
  <thcipriani@tin> | 
  Synchronized portals: SWAT: [[gerrit:302921|Bumping portals to master.]] (duration: 00m 37s) | 
  [production] | 
            
  | 15:07 | 
  <thcipriani@tin> | 
  Synchronized portals/prod/wikipedia.org/assets: SWAT: [[gerrit:302921|Bumping portals to master.]] (duration: 00m 36s) | 
  [production] | 
            
  | 14:11 | 
  <godog> | 
  manually config s1 dbs to scrape on prometheus2001 as a test | 
  [production] | 
            
  | 13:51 | 
  <akosiaris> | 
  delete+accept ganeti1004 salt minion key | 
  [production] | 
            
  | 13:43 | 
  <akosiaris@palladium> | 
  conftool action : set/pooled=yes; selector: wtp1012.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid']) | 
  [production] | 
            
  | 13:43 | 
  <akosiaris@palladium> | 
  conftool action : set/pooled=yes; selector: wtp1011.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid']) | 
  [production] | 
            
  | 13:43 | 
  <akosiaris@palladium> | 
  conftool action : set/pooled=yes; selector: wtp1010.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid']) | 
  [production] | 
            
  | 13:43 | 
  <akosiaris@palladium> | 
  conftool action : set/pooled=yes; selector: wtp1009.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid']) | 
  [production] | 
            
  | 13:43 | 
  <akosiaris@palladium> | 
  conftool action : set/pooled=yes; selector: wtp1008.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid']) | 
  [production] | 
            
  | 10:59 | 
  <jynus> | 
  applying prometheus grants to pc* hosts | 
  [production] | 
            
  | 10:33 | 
  <elukey> | 
  upgrading httpd on mw126[56] to 2.4.10-10+deb8u4+wmf3 (T73487) | 
  [production] | 
            
  | 07:31 | 
  <akosiaris@palladium> | 
  conftool action : set/pooled=no; selector: wtp1012.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid']) | 
  [production] | 
            
  | 07:31 | 
  <akosiaris@palladium> | 
  conftool action : set/pooled=no; selector: wtp1011.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid']) | 
  [production] | 
            
  | 07:31 | 
  <akosiaris@palladium> | 
  conftool action : set/pooled=no; selector: wtp1010.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid']) | 
  [production] | 
            
  | 07:31 | 
  <akosiaris@palladium> | 
  conftool action : set/pooled=no; selector: wtp1009.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid']) | 
  [production] | 
            
  | 07:31 | 
  <akosiaris@palladium> | 
  conftool action : set/pooled=no; selector: wtp1008.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid']) | 
  [production] | 
            
  | 07:30 | 
  <akosiaris@palladium> | 
  conftool action : set/pooled=no; selector: wtp1009.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid']) | 
  [production] | 
            
  | 07:30 | 
  <akosiaris@palladium> | 
  conftool action : set/pooled=no; selector: wtp1008.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid']) | 
  [production] | 
            
  | 02:43 | 
  <l10nupdate@tin> | 
  ResourceLoader cache refresh completed at Thu Aug  4 02:43:16 UTC 2016 (duration 6m 23s) | 
  [production] | 
            
  | 02:36 | 
  <mwdeploy@tin> | 
  scap sync-l10n completed (1.28.0-wmf.13) (duration: 05m 52s) | 
  [production] | 
            
  | 02:24 | 
  <mwdeploy@tin> | 
  scap sync-l10n completed (1.28.0-wmf.12) (duration: 08m 39s) | 
  [production] | 
            
  | 00:19 | 
  <awight> | 
  revert paymentswiki config to 70b2ff90d9c8f19716f2e9c07a8dc8cfa17991ca | 
  [production] | 
            
  | 00:17 | 
  <mutante> | 
  gerrit is restarting for config change 301822 (set default project owners). gerrit apache is restarting for 301829 (redirect /r) 301895 (logs for 10 days) and 301824 (renaming logs) | 
  [production] | 
            
  | 00:16 | 
  <awight> | 
  update payments-wiki config to 793389ac8fa34cfc6a4ba1df67f2f9fac1ca02fe | 
  [production] |