|
2016-08-04
ยง
|
| 21:26 |
<urandom> |
T140825,T140869: Rolling restart of Cassandra instances, eqiad Rack `b' |
[production] |
| 21:17 |
<ejegg> |
updated SmashPig from e6aa6fe6fdcaab8e961a8b0668cc742d4c443c46 to b7f5e449aa62cc5518dea580de96b8ed7a2489d0 |
[production] |
| 21:14 |
<urandom> |
T140825,T140869: Rolling restart of codfw Cassandra instances complete |
[production] |
| 20:40 |
<akosiaris> |
restart apache on palladium. Managed to get it into a deadlock. OFC puppet spam will flood the channel |
[production] |
| 20:13 |
<urandom> |
T140825,T140869: Performing rolling restart of codfw Cassandra instances |
[production] |
| 20:10 |
<urandom> |
T140825,T140869: Cassandra instance restarts complete: restbase1011.eqiad.wmnet |
[production] |
| 20:03 |
<urandom> |
T140825,T140869: Performing Cassandra instance rolling restart of restbase1011.eqiad.wmnet |
[production] |
| 19:59 |
<reedy@tin> |
Synchronized wmf-config/extension-list: More to extension.json in extension-list (duration: 00m 52s) |
[production] |
| 19:58 |
<ottomata> |
testing some eventbus kafka failure scenarios in codfw with test.event. (short icinga downtime has been scheduled) |
[production] |
| 19:51 |
<awight> |
paymentswiki config fix |
[production] |
| 19:47 |
<otto@palladium> |
conftool action : set/pooled=yes; selector: kafka2002.codfw.wmnet |
[production] |
| 19:46 |
<akosiaris> |
restart puppetmaster on palladium to activate loadfactor change. puppet related icinga spam will ensue* |
[production] |
| 19:46 |
<awight> |
enabling paymentswiki queue mirroring |
[production] |
| 19:46 |
<akosiaris> |
restart puppetmaster on palladium to activate loadfactor change. puppet related icinga spam will ensure |
[production] |
| 19:45 |
<otto@palladium> |
conftool action : set/pooled=no; selector: kafka2002.codfw.wmnet |
[production] |
| 19:43 |
<otto@palladium> |
conftool action : set/pooled=yes; selector: kafka2001.codfw.wmnet |
[production] |
| 19:42 |
<urandom> |
T140825,T140869: Restarting Cassandra, restbase1010-c.eqiad.wmnet |
[production] |
| 19:40 |
<otto@palladium> |
conftool action : set/pooled=no; selector: kafka2001.codfw.wmnet |
[production] |
| 19:40 |
<urandom> |
T140825,T140869: Restarting Cassandra, restbase1010-b.eqiad.wmnet |
[production] |
| 19:39 |
<ottomata> |
deploying eventlogging-service-eventbus to codfw hosts, depooling and pooling |
[production] |
| 19:37 |
<urandom> |
T140825,T140869: Restarting Cassandra, restbase1010-a.eqiad.wmnet |
[production] |
| 19:06 |
<thcipriani@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.28.0-wmf.13 |
[production] |
| 18:49 |
<thcipriani@tin> |
Synchronized README: test logstash host (duration: 00m 51s) |
[production] |
| 18:36 |
<cwd> |
updated payments from 3a724bfb1a3e20e17b5886dae0ba7572020abd6b to b737b60c87da82543ab812ece4611c68af01307f |
[production] |
| 18:12 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Pool db1074 as backup rc node in case db1036 lags (duration: 00m 25s) |
[production] |
| 18:12 |
<awight> |
update paymentswiki config to 71cc55194f4465600ce4da0ea9f7dfaefdda5479 |
[production] |
| 17:36 |
<elukey> |
added the analytics-deploy key to the Keyholder for the Analytics Refinery scap3 migration (also updated https://wikitech.wikimedia.org/wiki/Keyholder) |
[production] |
| 17:18 |
<ostriches> |
gerrit: Restarting really quick, trying alternative mysql library. |
[production] |
| 17:17 |
<urandom> |
T140825,T140869: Restarting Cassandra, restbase1007-c.eqiad.wmnet |
[production] |
| 17:14 |
<urandom> |
T140825,T140869: Restarting Cassandra, restbase1007-b.eqiad.wmnet |
[production] |
| 17:08 |
<urandom> |
T140825,T140869: Restarting Cassandra, restbase1007-a.eqiad.wmnet |
[production] |
| 17:07 |
<elukey> |
restarting keyholder-proxy on tin to let the new analytics key to be picked up |
[production] |
| 15:08 |
<thcipriani@tin> |
Synchronized portals: SWAT: [[gerrit:302921|Bumping portals to master.]] (duration: 00m 37s) |
[production] |
| 15:07 |
<thcipriani@tin> |
Synchronized portals/prod/wikipedia.org/assets: SWAT: [[gerrit:302921|Bumping portals to master.]] (duration: 00m 36s) |
[production] |
| 14:11 |
<godog> |
manually config s1 dbs to scrape on prometheus2001 as a test |
[production] |
| 13:51 |
<akosiaris> |
delete+accept ganeti1004 salt minion key |
[production] |
| 13:43 |
<akosiaris@palladium> |
conftool action : set/pooled=yes; selector: wtp1012.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid']) |
[production] |
| 13:43 |
<akosiaris@palladium> |
conftool action : set/pooled=yes; selector: wtp1011.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid']) |
[production] |
| 13:43 |
<akosiaris@palladium> |
conftool action : set/pooled=yes; selector: wtp1010.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid']) |
[production] |
| 13:43 |
<akosiaris@palladium> |
conftool action : set/pooled=yes; selector: wtp1009.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid']) |
[production] |
| 13:43 |
<akosiaris@palladium> |
conftool action : set/pooled=yes; selector: wtp1008.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid']) |
[production] |
| 10:59 |
<jynus> |
applying prometheus grants to pc* hosts |
[production] |
| 10:33 |
<elukey> |
upgrading httpd on mw126[56] to 2.4.10-10+deb8u4+wmf3 (T73487) |
[production] |
| 07:31 |
<akosiaris@palladium> |
conftool action : set/pooled=no; selector: wtp1012.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid']) |
[production] |
| 07:31 |
<akosiaris@palladium> |
conftool action : set/pooled=no; selector: wtp1011.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid']) |
[production] |
| 07:31 |
<akosiaris@palladium> |
conftool action : set/pooled=no; selector: wtp1010.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid']) |
[production] |
| 07:31 |
<akosiaris@palladium> |
conftool action : set/pooled=no; selector: wtp1009.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid']) |
[production] |
| 07:31 |
<akosiaris@palladium> |
conftool action : set/pooled=no; selector: wtp1008.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid']) |
[production] |
| 07:30 |
<akosiaris@palladium> |
conftool action : set/pooled=no; selector: wtp1009.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid']) |
[production] |
| 07:30 |
<akosiaris@palladium> |
conftool action : set/pooled=no; selector: wtp1008.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid']) |
[production] |