|
2021-07-21
§
|
| 22:37 |
<ryankemper> |
T280382 `sudo -i cookbook sre.wdqs.data-transfer --source wdqs1006.eqiad.wmnet --dest wdqs1010.eqiad.wmnet --reason "transferring fresh categories journal to resolve categories update lag unknown alert status" --blazegraph_instance categories --without-lvs` on `ryankemper@cumin1001` tmux session `wdqs` |
[production] |
| 22:37 |
<ryankemper@cumin1001> |
START - Cookbook sre.wdqs.data-transfer |
[production] |
| 22:36 |
<ryankemper@cumin1001> |
END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) |
[production] |
| 22:36 |
<ryankemper@cumin1001> |
START - Cookbook sre.wdqs.data-transfer |
[production] |
| 22:29 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 22:24 |
<pt1979@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 20:27 |
<dancy> |
testing upcoming Scap release on beta |
[production] |
| 18:27 |
<ryankemper> |
T281327 [Elastic] `sudo -i wmf-auto-reimage-host -p T281327 elastic2043.codfw.wmnet` on `ryankemper@cumin2001` tmux session `reimage_elastic2043` |
[production] |
| 18:21 |
<ryankemper> |
[WDQS] Restarted `wdqs-updater` on `wdqs1004` |
[production] |
| 18:19 |
<XioNoX> |
cr2-codfw> request chassis fpc slot 0 restart |
[production] |
| 18:17 |
<ryankemper> |
[WDQS] Depooled `wdqs1004` and restarted `wdqs-blazegraph` |
[production] |
| 18:17 |
<herron@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on logstash2003.codfw.wmnet with reason: REIMAGE |
[production] |
| 18:16 |
<ryankemper> |
[Elastic] Depooled `wdqs1004` and restarted `wdqs-blazegraph` |
[production] |
| 18:16 |
<urbanecm@deploy1002> |
Synchronized php-1.37.0-wmf.15/extensions/DiscussionTools/modules/ReplyLinksController.js: 1453831db13e17e550a86dd99d09dc26eeb242b1: Do not teardown newtopictool interface if it was not setup (T287035) (duration: 01m 04s) |
[production] |
| 18:15 |
<urbanecm@deploy1002> |
Synchronized php-1.37.0-wmf.14/extensions/DiscussionTools/modules/ReplyLinksController.js: aca510b773a67d24452731d5d6a33952c57592b8: Do not teardown newtopictool interface if it was not setup (T287035) (duration: 01m 05s) |
[production] |
| 18:14 |
<herron@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on logstash2003.codfw.wmnet with reason: REIMAGE |
[production] |
| 17:12 |
<herron@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on logstash2002.codfw.wmnet with reason: REIMAGE |
[production] |
| 17:10 |
<herron@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on logstash2002.codfw.wmnet with reason: REIMAGE |
[production] |
| 16:49 |
<urbanecm> |
[urbanecm@mwmaint2002 ~]$ time /usr/local/bin/mw-cli-wrapper /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblists/growthexperiments.dblist extensions/GrowthExperiments/maintenance/updateMenteeData.php # T285811 |
[production] |
| 16:32 |
<lucaswerkmeister-wmde@deploy1002> |
Synchronized php-1.37.0-wmf.15/extensions/WikibaseLexeme/WikibaseLexeme.entitytypes.repo.php: Backport: [[gerrit:705912|Define PREFETCHING_TERM_LOOKUP for all types in client and repo (T287085)]] (2/2) – I think this might be the fastest way to fix the errors (duration: 01m 05s) |
[production] |
| 16:27 |
<lucaswerkmeister-wmde@deploy1002> |
Synchronized php-1.37.0-wmf.15/extensions/WikibaseLexeme/WikibaseLexeme.entitytypes.php: restore previous state after previous scap failed on canaries with seemingly legitimate error (duration: 01m 04s) |
[production] |
| 16:23 |
<lucaswerkmeister-wmde@deploy1002> |
scap failed: average error rate on 2/6 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/83629bcb5560d11e61d3085c89dd9ed6 for details) |
[production] |
| 16:19 |
<herron@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on logstash2001.codfw.wmnet with reason: REIMAGE |
[production] |
| 16:17 |
<herron@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on logstash2001.codfw.wmnet with reason: REIMAGE |
[production] |
| 16:04 |
<jbond> |
upload cas_6.3.2-1+wmf10u1 to apt |
[production] |
| 15:45 |
<volans@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:05:00 on 10.3.0.1,cumin1001.mgmt,debmonitor.wikimedia.org with reason: testing new feature |
[production] |
| 15:45 |
<volans@cumin2002> |
START - Cookbook sre.hosts.downtime for 0:05:00 on 10.3.0.1,cumin1001.mgmt,debmonitor.wikimedia.org with reason: testing new feature |
[production] |
| 15:35 |
<dzahn@cumin1001> |
conftool action : set/weight=10; selector: name=mw143[7-8].eqiad.wmnet |
[production] |
| 15:34 |
<dzahn@cumin1001> |
conftool action : set/weight=10; selector: name=mw143[7-8].eqiad.wmnet,service=canary |
[production] |
| 15:17 |
<moritzm> |
installing intel-microcode security updates on stretch |
[production] |
| 15:11 |
<moritzm> |
installing apt bugfix updates from Buster 10.10 point release |
[production] |
| 14:51 |
<reedy@deploy1002> |
Finished scap: Fix some VE translation issues for T286679 (duration: 04m 45s) |
[production] |
| 14:46 |
<reedy@deploy1002> |
Started scap: Fix some VE translation issues for T286679 |
[production] |
| 14:40 |
<papaul> |
powerdown ms-be2038 for BBU replacement |
[production] |
| 14:33 |
<dzahn@cumin1001> |
conftool action : set/weight=10; selector: name=mw143[7-8].eqiad.wmnet,service=canary |
[production] |
| 14:19 |
<hashar@deploy1002> |
Finished deploy [gerrit/gerrit@a5c9d35]: Update its-phabricator: Urlencode POST to conduit # T280197 (duration: 00m 09s) |
[production] |
| 14:19 |
<hashar@deploy1002> |
Started deploy [gerrit/gerrit@a5c9d35]: Update its-phabricator: Urlencode POST to conduit # T280197 |
[production] |
| 14:07 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mw1294.eqiad.wmnet |
[production] |
| 14:07 |
<Jeff_Green> |
authdns-update to remove deprecated records related to fundraising.wikimedia.org |
[production] |
| 14:01 |
<moritzm> |
imported systemd 241-5~bpo9+wmf1 to component/systemd241 T287036 |
[production] |
| 13:52 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts mw1294.eqiad.wmnet |
[production] |
| 13:49 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mw1293.eqiad.wmnet |
[production] |
| 13:39 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts mw1293.eqiad.wmnet |
[production] |
| 13:20 |
<jiji@cumin1001> |
conftool action : set/pooled=true; selector: name=eqiad,dnsdisc=mwdebug |
[production] |
| 13:15 |
<jiji@cumin1001> |
conftool action : set/pooled=true; selector: name=codfw,dnsdisc=mwdebug |
[production] |
| 13:10 |
<hashar@deploy1002> |
Synchronized php: group1 wikis to 1.37.0-wmf.15 (duration: 01m 13s) |
[production] |
| 13:09 |
<godog> |
apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/705705 to puppetdb hosts |
[production] |
| 13:09 |
<hashar@deploy1002> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.37.0-wmf.15 |
[production] |
| 13:09 |
<mutante> |
mw1293, mw1294 - formerly jobrunner canaries, depooled, replaced by new jobrunner canaries mw1437, mw1438 |
[production] |
| 13:08 |
<dzahn@cumin1001> |
conftool action : set/pooled=inactive; selector: name=mw129[34].eqiad.wmnet |
[production] |