|
2021-07-21
§
|
| 22:40 |
<ryankemper@cumin1001> |
END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) |
[production] |
| 22:37 |
<ryankemper> |
T280382 `sudo -i cookbook sre.wdqs.data-transfer --source wdqs1006.eqiad.wmnet --dest wdqs1010.eqiad.wmnet --reason "transferring fresh categories journal to resolve categories update lag unknown alert status" --blazegraph_instance categories --without-lvs` on `ryankemper@cumin1001` tmux session `wdqs` |
[production] |
| 22:37 |
<ryankemper@cumin1001> |
START - Cookbook sre.wdqs.data-transfer |
[production] |
| 22:36 |
<ryankemper@cumin1001> |
END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) |
[production] |
| 22:36 |
<ryankemper@cumin1001> |
START - Cookbook sre.wdqs.data-transfer |
[production] |
| 22:29 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 22:24 |
<pt1979@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 21:06 |
<brennen> |
gitlab1001: running ansible for logging typo fix (T274462) |
[releng] |
| 20:36 |
<dancy> |
Newest scap deployed to beta cluster |
[releng] |
| 20:27 |
<dancy> |
testing upcoming Scap release on beta |
[production] |
| 20:01 |
<bstorm> |
deployed new maintain-kubeusers to toolforge T285011 |
[tools] |
| 19:55 |
<bstorm> |
deployed new rbac for maintain-kubeusers changes T285011 |
[tools] |
| 19:53 |
<bstorm> |
deployed new maintain-kubeusers T285011 |
[paws] |
| 19:52 |
<bstorm> |
deployed new rbac for maintain-kubeusers changes T285011 |
[paws] |
| 19:06 |
<brennen> |
gitlab1001: running ansible to deploy nginx logging and status changes (T274462, T275170) |
[releng] |
| 18:27 |
<ryankemper> |
T281327 [Elastic] `sudo -i wmf-auto-reimage-host -p T281327 elastic2043.codfw.wmnet` on `ryankemper@cumin2001` tmux session `reimage_elastic2043` |
[production] |
| 18:21 |
<ryankemper> |
[WDQS] Restarted `wdqs-updater` on `wdqs1004` |
[production] |
| 18:19 |
<XioNoX> |
cr2-codfw> request chassis fpc slot 0 restart |
[production] |
| 18:17 |
<ryankemper> |
[WDQS] Depooled `wdqs1004` and restarted `wdqs-blazegraph` |
[production] |
| 18:17 |
<herron@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on logstash2003.codfw.wmnet with reason: REIMAGE |
[production] |
| 18:16 |
<ryankemper> |
[Elastic] Depooled `wdqs1004` and restarted `wdqs-blazegraph` |
[production] |
| 18:16 |
<urbanecm@deploy1002> |
Synchronized php-1.37.0-wmf.15/extensions/DiscussionTools/modules/ReplyLinksController.js: 1453831db13e17e550a86dd99d09dc26eeb242b1: Do not teardown newtopictool interface if it was not setup (T287035) (duration: 01m 04s) |
[production] |
| 18:15 |
<urbanecm@deploy1002> |
Synchronized php-1.37.0-wmf.14/extensions/DiscussionTools/modules/ReplyLinksController.js: aca510b773a67d24452731d5d6a33952c57592b8: Do not teardown newtopictool interface if it was not setup (T287035) (duration: 01m 05s) |
[production] |
| 18:14 |
<herron@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on logstash2003.codfw.wmnet with reason: REIMAGE |
[production] |
| 17:12 |
<herron@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on logstash2002.codfw.wmnet with reason: REIMAGE |
[production] |
| 17:10 |
<herron@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on logstash2002.codfw.wmnet with reason: REIMAGE |
[production] |
| 17:10 |
<majavah> |
deploying calico v3.18.4 T280342 |
[tools] |
| 16:59 |
<majavah> |
deploying calico v3.18.4 T280342 |
[paws] |
| 16:49 |
<urbanecm> |
[urbanecm@mwmaint2002 ~]$ time /usr/local/bin/mw-cli-wrapper /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblists/growthexperiments.dblist extensions/GrowthExperiments/maintenance/updateMenteeData.php # T285811 |
[production] |
| 16:46 |
<dancy> |
restarting Gerrit to fix plugins |
[releng] |
| 16:32 |
<lucaswerkmeister-wmde@deploy1002> |
Synchronized php-1.37.0-wmf.15/extensions/WikibaseLexeme/WikibaseLexeme.entitytypes.repo.php: Backport: [[gerrit:705912|Define PREFETCHING_TERM_LOOKUP for all types in client and repo (T287085)]] (2/2) – I think this might be the fastest way to fix the errors (duration: 01m 05s) |
[production] |
| 16:27 |
<lucaswerkmeister-wmde@deploy1002> |
Synchronized php-1.37.0-wmf.15/extensions/WikibaseLexeme/WikibaseLexeme.entitytypes.php: restore previous state after previous scap failed on canaries with seemingly legitimate error (duration: 01m 04s) |
[production] |
| 16:23 |
<lucaswerkmeister-wmde@deploy1002> |
scap failed: average error rate on 2/6 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/83629bcb5560d11e61d3085c89dd9ed6 for details) |
[production] |
| 16:19 |
<herron@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on logstash2001.codfw.wmnet with reason: REIMAGE |
[production] |
| 16:17 |
<herron@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on logstash2001.codfw.wmnet with reason: REIMAGE |
[production] |
| 16:07 |
<dancy> |
Updating plugins on releases-jenkins |
[releng] |
| 16:04 |
<jbond> |
upload cas_6.3.2-1+wmf10u1 to apt |
[production] |
| 15:52 |
<majavah> |
add my key to passwords::root::extra_keys |
[paws] |
| 15:45 |
<volans@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:05:00 on 10.3.0.1,cumin1001.mgmt,debmonitor.wikimedia.org with reason: testing new feature |
[production] |
| 15:45 |
<volans@cumin2002> |
START - Cookbook sre.hosts.downtime for 0:05:00 on 10.3.0.1,cumin1001.mgmt,debmonitor.wikimedia.org with reason: testing new feature |
[production] |
| 15:35 |
<dzahn@cumin1001> |
conftool action : set/weight=10; selector: name=mw143[7-8].eqiad.wmnet |
[production] |
| 15:34 |
<dzahn@cumin1001> |
conftool action : set/weight=10; selector: name=mw143[7-8].eqiad.wmnet,service=canary |
[production] |
| 15:17 |
<moritzm> |
installing intel-microcode security updates on stretch |
[production] |
| 15:11 |
<moritzm> |
installing apt bugfix updates from Buster 10.10 point release |
[production] |
| 15:00 |
<majavah> |
starting kubernetes upgrades T280302 |
[paws] |
| 15:00 |
<urbanecm> |
deployment-prep: Change password for `Martin Urbanec` at votewiki |
[releng] |
| 14:51 |
<reedy@deploy1002> |
Finished scap: Fix some VE translation issues for T286679 (duration: 04m 45s) |
[production] |
| 14:46 |
<reedy@deploy1002> |
Started scap: Fix some VE translation issues for T286679 |
[production] |
| 14:40 |
<papaul> |
powerdown ms-be2038 for BBU replacement |
[production] |
| 14:35 |
<majavah> |
updating systemd on toolforge stretch bastions T287036 |
[tools] |