| 
      
        2016-10-03
      
      §
     | 
  
    
  | 16:19 | 
  <ejegg> | 
  updated payments-wiki settings to stop sending completed donations to ActiveMQ | 
  [production] | 
            
  | 15:15 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-eqiad.php: Increase weight for db1084 after its maintenance to its original value: 500 - T147113 (duration: 00m 48s) | 
  [production] | 
            
  | 14:58 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-eqiad.php: Increase weight for db1084 after its maintenance - T147113 (duration: 00m 48s) | 
  [production] | 
            
  | 14:53 | 
  <chasemp> | 
  adding volans (RCoccioli) to phab security, confirmed staff account association and membership in ops acl already, confirmed w/ riccardo he is missing, and there is a long standing agreement all members of ops should be in #security | 
  [production] | 
            
  | 14:37 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-eqiad.php: Repool db1084 after its maintenance - T147113 (duration: 00m 48s) | 
  [production] | 
            
  | 14:32 | 
  <zeljkof> | 
  ending EU SWAT | 
  [production] | 
            
  | 14:28 | 
  <zfilipin@tin> | 
  Synchronized robots.txt: SWAT: [[gerrit:313763|Fix an invalid empty line in the global robots.txt (T146908)]] (duration: 00m 47s) | 
  [production] | 
            
  | 14:24 | 
  <zfilipin@tin> | 
  Synchronized static/images/project-logos/olowiki-2x.png: SWAT: [[gerrit:313658|Add 1.5 and 2x logos for olowiki (T146745)]] (duration: 00m 48s) | 
  [production] | 
            
  | 14:23 | 
  <zfilipin@tin> | 
  Synchronized static/images/project-logos/olowiki-1.5x.png: SWAT: [[gerrit:313658|Add 1.5 and 2x logos for olowiki (T146745)]] (duration: 00m 48s) | 
  [production] | 
            
  | 14:20 | 
  <hashar> | 
  T146271  mwscript purgeList.php --wiki=testwikidatawiki --namespace=121 --verbose | 
  [production] | 
            
  | 14:19 | 
  <hashar> | 
  Purged wikidata wiki property talk page, they now allow subpages (T146271). Ran: mwscript purgeList.php --wiki=wikidatawiki --namespace=121 --verbose | 
  [production] | 
            
  | 14:15 | 
  <zfilipin@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:312500|Enable subpages in 121 namespace in wikidata (T146271)]] (duration: 00m 49s) | 
  [production] | 
            
  | 14:02 | 
  <zeljkof> | 
  extending EU SWAT | 
  [production] | 
            
  | 13:55 | 
  <zfilipin@tin> | 
  Synchronized static/images/project-logos/olowiki.png: SWAT: [[gerrit:312977|Upload 1x logo for olowiki (T146745)]] (duration: 00m 48s) | 
  [production] | 
            
  | 13:51 | 
  <zfilipin@tin> | 
  Synchronized static/images/project-logos/hewiki.png: SWAT: [[gerrit:310798|Fix hewiki logos (T145017)]] (duration: 00m 47s) | 
  [production] | 
            
  | 13:38 | 
  <godog> | 
  reenable puppet on scb1* ores/celery spamming is over | 
  [production] | 
            
  | 13:31 | 
  <zfilipin@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:312734|Change protection level autoreview in arwiki (T146575)]] (duration: 00m 48s) | 
  [production] | 
            
  | 13:23 | 
  <dcausse> | 
  elasticsearch@eqiad: reducing replica count from 5 to 2 for jawiki_titlesuggest and eswiki_titlesuggest | 
  [production] | 
            
  | 13:21 | 
  <zfilipin@tin> | 
  Synchronized wmf-config/throttle.php: SWAT: [[gerrit:313784|[throttle] Rule for Winona State University (T146600)]] [[gerrit:312852|[throttle] Ada Lovelace Day Edit-a-thon (T146654)]] (duration: 00m 49s) | 
  [production] | 
            
  | 13:19 | 
  <dcausse> | 
  elasticsearch@eqiad: reducing replica count from 5 to 2 for ruwiki_titlesuggest | 
  [production] | 
            
  | 13:15 | 
  <dcausse> | 
  elasticsearch@eqiad: reducing replica count from 5 to 2 for zhwiki_titlesuggest | 
  [production] | 
            
  | 13:13 | 
  <gehel> | 
  reimage of maps-test2001 - T147194 | 
  [production] | 
            
  | 13:11 | 
  <dcausse> | 
  elasticsearch@eqiad: reducing replica count from 5 to 2 for frwiki_titlesuggest | 
  [production] | 
            
  | 13:09 | 
  <gehel> | 
  shutting down services on maps-test* servers prior to reimage -T147194 | 
  [production] | 
            
  | 13:02 | 
  <zeljkof> | 
  starting EU SWAT | 
  [production] | 
            
  | 13:00 | 
  <dcausse> | 
  elasticsearch@eqiad: reducing replica count from 5 to 3 for enwiki_titlesuggest | 
  [production] | 
            
  | 12:52 | 
  <dcausse> | 
  elasticsearch@eqiad: reducing replica count from 5 to 3 for dewiki_titlesuggest | 
  [production] | 
            
  | 12:23 | 
  <marostegui> | 
  Deploying alter table in S4 - T147113 | 
  [production] | 
            
  | 12:14 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-eqiad.php: Depool db1084 for maintenance - T147113 (duration: 00m 48s) | 
  [production] | 
            
  | 11:51 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-eqiad.php: Increase db1081 weight to its original value after finishing maintenance - T147113 (duration: 00m 48s) | 
  [production] | 
            
  | 11:21 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-eqiad.php: Increase db1081 weight after finishing its maintenance - T147113 (duration: 00m 48s) | 
  [production] | 
            
  | 10:33 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-eqiad.php: Increase db1081 weight after finishing its maintenance - T147113 (duration: 00m 48s) | 
  [production] | 
            
  | 10:21 | 
  <akosiaris> | 
  restarting slapd on dubnium.wikimedia.org T143302 | 
  [production] | 
            
  | 10:16 | 
  <akosiaris> | 
  restarting slapd on seaborgium.wikimedia.org T143302 | 
  [production] | 
            
  | 10:13 | 
  <akosiaris> | 
  restarting slapd on serpens.wikimedia.org T143302 | 
  [production] | 
            
  | 10:11 | 
  <akosiaris> | 
  restarting slapd on pollux.wikimedia.org T143302 | 
  [production] | 
            
  | 09:54 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-eqiad.php: Repool db1081 after finishing its maintenance - T147113 (duration: 00m 49s) | 
  [production] | 
            
  | 09:48 | 
  <elukey> | 
  lowered down builds log retention from 90 to 60 days for the puppet compiler (https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/)  | 
  [production] | 
            
  | 09:32 | 
  <akosiaris> | 
  T147173 clean exim queues on mx1001 from backscatter spam. Seems to be originating from mx.{east,west}.cox.net, blocked them for now | 
  [production] | 
            
  | 09:28 | 
  <marostegui> | 
  dbstore2001 going to be reimaged as jessie | 
  [production] | 
            
  | 09:27 | 
  <gehel> | 
  rolling restart of elasticsearch codfw cluster for kernel upgrade - T146123 | 
  [production] | 
            
  | 09:14 | 
  <akosiaris> | 
  T147173 clean exim queues on mx1001 from backscatter spam | 
  [production] | 
            
  | 09:08 | 
  <akosiaris> | 
  clean exim queues on mx1001 from backscatter spam | 
  [production] | 
            
  | 08:46 | 
  <elukey> | 
  rebooted compiler02.puppet3-diffs.eqiad.wmflabs (not reachable by Jenkins, pingable from bastions but no ssh available) | 
  [production] | 
            
  | 08:04 | 
  <_joe_> | 
  powercycling mw1207 | 
  [production] | 
            
  | 07:56 | 
  <marostegui@tin> | 
  Synchronized wmf-config/db-eqiad.php: Depool db1081 for maintenance - T147113 (duration: 00m 50s) | 
  [production] | 
            
  | 07:24 | 
  <volans> | 
  emptying /var/log/debug on dubnium because of disk full (the same data is on syslog) T147173 | 
  [production] | 
            
  | 06:30 | 
  <marostegui> | 
  altering S3,S4,S5,S6,S7 user_groups tables in sanitarium to avoid tokudb bug - T146121 | 
  [production] | 
            
  | 02:28 | 
  <mwdeploy@tin> | 
  scap sync-l10n completed (1.28.0-wmf.20) (duration: 13m 16s) | 
  [production] |