5351-5400 of 10000 results (56ms)
2021-07-21 §
22:41 <ryankemper> T280382 `sudo -i cookbook sre.wdqs.data-transfer --source wdqs1006.eqiad.wmnet --dest wdqs1009.eqiad.wmnet --reason "transferring fresh categories journal to resolve categories update lag unknown alert status" --blazegraph_instance categories --without-lvs` on `ryankemper@cumin1001` tmux session `wdqs` [production]
22:41 <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
22:40 <ryankemper@cumin1001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
22:37 <ryankemper> T280382 `sudo -i cookbook sre.wdqs.data-transfer --source wdqs1006.eqiad.wmnet --dest wdqs1010.eqiad.wmnet --reason "transferring fresh categories journal to resolve categories update lag unknown alert status" --blazegraph_instance categories --without-lvs` on `ryankemper@cumin1001` tmux session `wdqs` [production]
22:37 <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
22:36 <ryankemper@cumin1001> END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) [production]
22:36 <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
22:29 <pt1979@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
22:24 <pt1979@cumin2002> START - Cookbook sre.dns.netbox [production]
20:27 <dancy> testing upcoming Scap release on beta [production]
18:27 <ryankemper> T281327 [Elastic] `sudo -i wmf-auto-reimage-host -p T281327 elastic2043.codfw.wmnet` on `ryankemper@cumin2001` tmux session `reimage_elastic2043` [production]
18:21 <ryankemper> [WDQS] Restarted `wdqs-updater` on `wdqs1004` [production]
18:19 <XioNoX> cr2-codfw> request chassis fpc slot 0 restart [production]
18:17 <ryankemper> [WDQS] Depooled `wdqs1004` and restarted `wdqs-blazegraph` [production]
18:17 <herron@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on logstash2003.codfw.wmnet with reason: REIMAGE [production]
18:16 <ryankemper> [Elastic] Depooled `wdqs1004` and restarted `wdqs-blazegraph` [production]
18:16 <urbanecm@deploy1002> Synchronized php-1.37.0-wmf.15/extensions/DiscussionTools/modules/ReplyLinksController.js: 1453831db13e17e550a86dd99d09dc26eeb242b1: Do not teardown newtopictool interface if it was not setup (T287035) (duration: 01m 04s) [production]
18:15 <urbanecm@deploy1002> Synchronized php-1.37.0-wmf.14/extensions/DiscussionTools/modules/ReplyLinksController.js: aca510b773a67d24452731d5d6a33952c57592b8: Do not teardown newtopictool interface if it was not setup (T287035) (duration: 01m 05s) [production]
18:14 <herron@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on logstash2003.codfw.wmnet with reason: REIMAGE [production]
17:12 <herron@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on logstash2002.codfw.wmnet with reason: REIMAGE [production]
17:10 <herron@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on logstash2002.codfw.wmnet with reason: REIMAGE [production]
16:49 <urbanecm> [urbanecm@mwmaint2002 ~]$ time /usr/local/bin/mw-cli-wrapper /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblists/growthexperiments.dblist extensions/GrowthExperiments/maintenance/updateMenteeData.php # T285811 [production]
16:32 <lucaswerkmeister-wmde@deploy1002> Synchronized php-1.37.0-wmf.15/extensions/WikibaseLexeme/WikibaseLexeme.entitytypes.repo.php: Backport: [[gerrit:705912|Define PREFETCHING_TERM_LOOKUP for all types in client and repo (T287085)]] (2/2) – I think this might be the fastest way to fix the errors (duration: 01m 05s) [production]
16:27 <lucaswerkmeister-wmde@deploy1002> Synchronized php-1.37.0-wmf.15/extensions/WikibaseLexeme/WikibaseLexeme.entitytypes.php: restore previous state after previous scap failed on canaries with seemingly legitimate error (duration: 01m 04s) [production]
16:23 <lucaswerkmeister-wmde@deploy1002> scap failed: average error rate on 2/6 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/83629bcb5560d11e61d3085c89dd9ed6 for details) [production]
16:19 <herron@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on logstash2001.codfw.wmnet with reason: REIMAGE [production]
16:17 <herron@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on logstash2001.codfw.wmnet with reason: REIMAGE [production]
16:04 <jbond> upload cas_6.3.2-1+wmf10u1 to apt [production]
15:45 <volans@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:05:00 on 10.3.0.1,cumin1001.mgmt,debmonitor.wikimedia.org with reason: testing new feature [production]
15:45 <volans@cumin2002> START - Cookbook sre.hosts.downtime for 0:05:00 on 10.3.0.1,cumin1001.mgmt,debmonitor.wikimedia.org with reason: testing new feature [production]
15:35 <dzahn@cumin1001> conftool action : set/weight=10; selector: name=mw143[7-8].eqiad.wmnet [production]
15:34 <dzahn@cumin1001> conftool action : set/weight=10; selector: name=mw143[7-8].eqiad.wmnet,service=canary [production]
15:17 <moritzm> installing intel-microcode security updates on stretch [production]
15:11 <moritzm> installing apt bugfix updates from Buster 10.10 point release [production]
14:51 <reedy@deploy1002> Finished scap: Fix some VE translation issues for T286679 (duration: 04m 45s) [production]
14:46 <reedy@deploy1002> Started scap: Fix some VE translation issues for T286679 [production]
14:40 <papaul> powerdown ms-be2038 for BBU replacement [production]
14:33 <dzahn@cumin1001> conftool action : set/weight=10; selector: name=mw143[7-8].eqiad.wmnet,service=canary [production]
14:19 <hashar@deploy1002> Finished deploy [gerrit/gerrit@a5c9d35]: Update its-phabricator: Urlencode POST to conduit # T280197 (duration: 00m 09s) [production]
14:19 <hashar@deploy1002> Started deploy [gerrit/gerrit@a5c9d35]: Update its-phabricator: Urlencode POST to conduit # T280197 [production]
14:07 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mw1294.eqiad.wmnet [production]
14:07 <Jeff_Green> authdns-update to remove deprecated records related to fundraising.wikimedia.org [production]
14:01 <moritzm> imported systemd 241-5~bpo9+wmf1 to component/systemd241 T287036 [production]
13:52 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission for hosts mw1294.eqiad.wmnet [production]
13:49 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mw1293.eqiad.wmnet [production]
13:39 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission for hosts mw1293.eqiad.wmnet [production]
13:20 <jiji@cumin1001> conftool action : set/pooled=true; selector: name=eqiad,dnsdisc=mwdebug [production]
13:15 <jiji@cumin1001> conftool action : set/pooled=true; selector: name=codfw,dnsdisc=mwdebug [production]
13:10 <hashar@deploy1002> Synchronized php: group1 wikis to 1.37.0-wmf.15 (duration: 01m 13s) [production]
13:09 <godog> apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/705705 to puppetdb hosts [production]