|
2019-10-23
ยง
|
| 18:49 |
<milimetric@deploy1001> |
Started deploy [analytics/refinery@3aaabf6]: Minor: fix two scripts |
[production] |
| 18:29 |
<mforns@deploy1001> |
Finished deploy [analytics/refinery@1110d59]: deploying refinery up to 1110d59c3983bcff4986bce1baf885f05ee06ba5 (duration: 06m 40s) |
[production] |
| 18:22 |
<mforns@deploy1001> |
Started deploy [analytics/refinery@1110d59]: deploying refinery up to 1110d59c3983bcff4986bce1baf885f05ee06ba5 |
[production] |
| 17:31 |
<akosiaris> |
restart varnish-be on cp1089 as a response to HTTP availability alerts. High mailbox lag |
[production] |
| 17:25 |
<akosiaris> |
restart varnish-be on cp1081 as a response to HTTP availability alerts |
[production] |
| 15:55 |
<_joe_> |
restarting pybal on lvs2006, then 2003 for picking up parsoid-php |
[production] |
| 15:32 |
<marostegui> |
Enable slow query log 1/20 on db1089 (enwiki) T223151 |
[production] |
| 14:40 |
<ema@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 14:39 |
<ema@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 14:38 |
<ema@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) |
[production] |
| 14:37 |
<ema@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 14:36 |
<ema@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 14:35 |
<ema@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 14:19 |
<bblack> |
repooling esams |
[production] |
| 14:00 |
<hashar> |
Restarting CI Jenkins |
[production] |
| 13:57 |
<_joe_> |
manually changing the symlinked deployed version of parsoid on wtp1025 T236275 |
[production] |
| 13:35 |
<XioNoX> |
migrate esams mgmt to new mgmt router |
[production] |
| 13:34 |
<effie> |
disable puppet on mwdebug1002 - T214734 |
[production] |
| 13:13 |
<ssastry@deploy1001> |
Finished deploy [parsoid/deploy@451db1e]: Updating Parsoid to 5521ea74; Dummy Parsoid deploy to debug Parsoid/PHP deployment issues (duration: 08m 44s) |
[production] |
| 13:07 |
<liw@deploy1001> |
Synchronized php: group1 wikis to 1.35.0-wmf.3 (duration: 01m 00s) |
[production] |
| 13:05 |
<liw@deploy1001> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.35.0-wmf.3 |
[production] |
| 13:04 |
<ssastry@deploy1001> |
Started deploy [parsoid/deploy@451db1e]: Updating Parsoid to 5521ea74; Dummy Parsoid deploy to debug Parsoid/PHP deployment issues |
[production] |
| 12:37 |
<effie> |
Depool mwdebug1002 - T214734 |
[production] |
| 12:31 |
<vgutierrez> |
restarting ats-tls on cache text nodes - T233274 |
[production] |
| 12:27 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Remove db1130 from the special slaves group on s5 and leave it back with its original pooling options T223151', diff saved to https://phabricator.wikimedia.org/P9454 and previous config saved to /var/cache/conftool/dbconfig/20191023-122708-marostegui.json |
[production] |
| 11:26 |
<XioNoX> |
powering down cr1-esams |
[production] |
| 11:24 |
<Urbanecm> |
EU SWAT done |
[production] |
| 11:22 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InterwikiSortOrders.php: SWAT: e21054e: Add Balinese to interwiki sort orders (T234768) (duration: 01m 01s) |
[production] |
| 11:18 |
<Urbanecm> |
mwscript updateArticleCount.php --wiki=frwikiquote --update (T236212) |
[production] |
| 11:16 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: 0889da0: Add custom Minerva wordmark for Hebrew wikivoyage (2/2; T234278) (duration: 01m 01s) |
[production] |
| 11:09 |
<urbanecm@deploy1001> |
Synchronized static/images/mobile/copyright: SWAT: 0889da0: Add custom Minerva wordmark for Hebrew wikivoyage (1/2; T234278) (duration: 01m 01s) |
[production] |
| 11:06 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: cf8e2f1: Set $wgArticleCountMethod to any for frwikiquote (T236212) (duration: 01m 12s) |
[production] |
| 10:46 |
<ema> |
cp-ats: rolling ATS backend restart to apply https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/545522/ T233274 |
[production] |
| 10:13 |
<jynus> |
reverting dbtree revision to HEAD~1 T224589 |
[production] |
| 10:11 |
<jynus> |
deploying new version of dbtree T224589 |
[production] |
| 10:04 |
<ema> |
cp1075: ats-backend-restart to test https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/545508/ |
[production] |
| 09:42 |
<godog> |
bounce burrow-logging-eqiad.service on kafkamon1001 |
[production] |
| 09:40 |
<godog> |
roll restart logstash to pick up new rsyslog-notice partitions |
[production] |
| 09:31 |
<godog> |
bump rsyslog-notice topic to 6 partitions |
[production] |
| 09:00 |
<moritzm> |
rebooting logstash2021 for some firmware tests |
[production] |
| 08:59 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 08:59 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:54 |
<moritzm> |
installing systemd bugfix update on mw canaries |
[production] |
| 08:50 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 08:50 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:42 |
<godog> |
roll restart rsyslog on cirrus and wqds hosts to pick up changes to logback topic partitions |
[production] |
| 08:28 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db2091:3312 after table compression', diff saved to https://phabricator.wikimedia.org/P9452 and previous config saved to /var/cache/conftool/dbconfig/20191023-082826-marostegui.json |
[production] |
| 08:23 |
<godog> |
roll restart logstash in codfw/eqiad to pick up new kafka partitions |
[production] |
| 08:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Change weights to x100 on s8 eqiad - T231018', diff saved to https://phabricator.wikimedia.org/P9451 and previous config saved to /var/cache/conftool/dbconfig/20191023-082246-marostegui.json |
[production] |
| 08:11 |
<godog> |
kafka-logging eqiad set 12 partitions for ^mwlog- ^logback- and eqiad.client.error topics |
[production] |