|
2024-04-24
§
|
| 20:12 |
<mutante> |
- puppet run on deploy-1006 deployment_server - scap init duplicate declaration fixed after adding "profile::mediawiki::scap_client::is_master: true" to new deploy-prefix Hiera - thanks to Jaime Nuche per comment on gerrit |
[devtools] |
| 19:46 |
<mutante> |
creating new prefix 'deploy' to apply needed Hiera keys to deployment hosts based on host name prefix (both deploy1004, deploy1006 and future deploy*) |
[devtools] |
| 19:41 |
<mutante> |
deleting instance puppetmaster-1001 that was > 4 years old, on buster and I had shutdown a couple days ago. replaced by puppetmaster-1003 (bookworm, puppetserver) T360964 T360470 |
[devtools] |
| 19:32 |
<dzahn@cumin2002> |
START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: security release T363349 |
[production] |
| 19:27 |
<cstone> |
payments-wiki upgraded from 1895e43b to c7ab847d |
[production] |
| 19:26 |
<wmbot~anticomposite@tools-bastion-13> |
rm -r venv/ venv-k8s-py37/ venv-py3/ venv-py37-k8s/ # Remove old unused python 2.7, 3.5, and 3.7 venvs (T130031) |
[tools.stewardbots] |
| 19:26 |
<wmbot~anticomposite@tools-bastion-13> |
rm -r venv/ venv-k8s-py37/ venv-py3/ venv-py37-k8s/ |
[tools.stewardbots] |
| 19:21 |
<mutante> |
gitlab-prod-1002; gitlab-runner-1003; gitlab-runner-1002 - apt-get update && apt-get upgrade |
[devtools] |
| 19:15 |
<bking@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 19:15 |
<bking@deploy1002> |
helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 19:14 |
<eevans@cumin1002> |
END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:restbase-eqiad: Apply truststore changes — T352647 - eevans@cumin1002 |
[production] |
| 19:08 |
<inflatador> |
bking@deploy1002 stop `consumer-cloudelastic` release to test alerting T359213 |
[production] |
| 19:07 |
<wmbot~anticomposite@tools-bastion-13> |
toolforge jobs delete sbar # This job never worked as configured and won't work as intended. |
[tools.stewardbots] |
| 19:07 |
<bking@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 19:06 |
<bking@deploy1002> |
helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 19:03 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Depooling db2153 (T352010)', diff saved to https://phabricator.wikimedia.org/P61181 and previous config saved to /var/cache/conftool/dbconfig/20240424-190237-ladsgroup.json |
[production] |
| 19:03 |
<ladsgroup@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2153.codfw.wmnet with reason: Maintenance |
[production] |
| 19:02 |
<ladsgroup@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2153.codfw.wmnet with reason: Maintenance |
[production] |
| 19:02 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2146 (T352010)', diff saved to https://phabricator.wikimedia.org/P61180 and previous config saved to /var/cache/conftool/dbconfig/20240424-190214-ladsgroup.json |
[production] |
| 18:57 |
<amastilovic@deploy1002> |
Finished deploy [airflow-dags/analytics@3f994d5]: (no justification provided) (duration: 00m 28s) |
[production] |
| 18:57 |
<amastilovic@deploy1002> |
Started deploy [airflow-dags/analytics@3f994d5]: (no justification provided) |
[production] |
| 18:47 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2146', diff saved to https://phabricator.wikimedia.org/P61179 and previous config saved to /var/cache/conftool/dbconfig/20240424-184707-ladsgroup.json |
[production] |
| 18:37 |
<wmbot~anticomposite@tools-bastion-13> |
Deploy 2f528f2..c8354da (T329327) |
[tools.stewardbots] |
| 18:32 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2146', diff saved to https://phabricator.wikimedia.org/P61178 and previous config saved to /var/cache/conftool/dbconfig/20240424-183200-ladsgroup.json |
[production] |
| 18:20 |
<brennen@deploy1002> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.43.0-wmf.2 refs T361396 |
[production] |
| 18:16 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2146 (T352010)', diff saved to https://phabricator.wikimedia.org/P61177 and previous config saved to /var/cache/conftool/dbconfig/20240424-181653-ladsgroup.json |
[production] |
| 18:14 |
<cstone> |
payments-wiki upgraded from fb0367a4 to 1895e43b |
[production] |
| 18:12 |
<wmbot~taavi@tools-bastion-12> |
kubectl apply -f redirect.yaml |
[tools.libraryupgrader] |
| 18:03 |
<brennen> |
train 1.43.0-wmf.2 (T361396) status: no current blockers, rolling to group1 |
[production] |
| 17:41 |
<btullis@cumin1002> |
END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) |
[production] |
| 17:20 |
<btullis@cumin1002> |
START - Cookbook sre.wdqs.restart |
[production] |
| 17:05 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Depooling db2154 (T352010)', diff saved to https://phabricator.wikimedia.org/P61176 and previous config saved to /var/cache/conftool/dbconfig/20240424-170444-ladsgroup.json |
[production] |
| 17:04 |
<ladsgroup@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2154.codfw.wmnet with reason: Maintenance |
[production] |
| 17:04 |
<ladsgroup@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2154.codfw.wmnet with reason: Maintenance |
[production] |
| 17:04 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2152 (T352010)', diff saved to https://phabricator.wikimedia.org/P61175 and previous config saved to /var/cache/conftool/dbconfig/20240424-170421-ladsgroup.json |
[production] |
| 17:03 |
<hmonroy@deploy1002> |
Finished scap: Backport for [[gerrit:1023501|[hewiki] enable CodeMirrorV6 and CodeMirrorLineNumberingNamespaces (T357795 T347211)]] (duration: 20m 36s) |
[production] |
| 16:51 |
<eevans@cumin1002> |
START - Cookbook sre.cassandra.roll-restart for nodes matching A:restbase-eqiad: Apply truststore changes — T352647 - eevans@cumin1002 |
[production] |
| 16:50 |
<hmonroy@deploy1002> |
musikanimal and hmonroy: Continuing with sync |
[production] |
| 16:49 |
<btullis@cumin1002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host stat1010.eqiad.wmnet |
[production] |
| 16:49 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2152', diff saved to https://phabricator.wikimedia.org/P61174 and previous config saved to /var/cache/conftool/dbconfig/20240424-164914-ladsgroup.json |
[production] |
| 16:45 |
<hmonroy@deploy1002> |
musikanimal and hmonroy: Backport for [[gerrit:1023501|[hewiki] enable CodeMirrorV6 and CodeMirrorLineNumberingNamespaces (T357795 T347211)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 16:43 |
<elukey@cumin1002> |
END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:restbase-codfw: Deploy new TLS Truststore for PKI - elukey@cumin1002 |
[production] |
| 16:42 |
<hmonroy@deploy1002> |
Started scap: Backport for [[gerrit:1023501|[hewiki] enable CodeMirrorV6 and CodeMirrorLineNumberingNamespaces (T357795 T347211)]] |
[production] |
| 16:42 |
<btullis@cumin1002> |
START - Cookbook sre.hosts.reboot-single for host stat1010.eqiad.wmnet |
[production] |
| 16:38 |
<bd808> |
Added jhsoby as a co-maintainer. He kindly volunteered to help with restarts when the bot is double posting to Telegram (T305487) |
[tools.bridgebot] |
| 16:34 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2152', diff saved to https://phabricator.wikimedia.org/P61173 and previous config saved to /var/cache/conftool/dbconfig/20240424-163407-ladsgroup.json |
[production] |
| 16:19 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2152 (T352010)', diff saved to https://phabricator.wikimedia.org/P61172 and previous config saved to /var/cache/conftool/dbconfig/20240424-161859-ladsgroup.json |
[production] |
| 16:11 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1190 (re)pooling @ 100%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61171 and previous config saved to /var/cache/conftool/dbconfig/20240424-161112-arnaudb.json |
[production] |
| 16:05 |
<sukhe> |
running authdns-update |
[production] |
| 16:04 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . |
[production] |