|
2024-12-10
ยง
|
| 21:27 |
<cjming@deploy2002> |
bvibber, cjming: Backport for [[gerrit:1101600|LanguageConverter: Ignore content inside <math> and <svg> elements (T381617)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 21:23 |
<cjming@deploy2002> |
Started scap sync-world: Backport for [[gerrit:1101600|LanguageConverter: Ignore content inside <math> and <svg> elements (T381617)]] |
[production] |
| 21:22 |
<ryankemper@cumin2002> |
END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T376150, xfer wdqs scholarly 2023(public)->2026(internal)) xfer scholarly_articles from wdqs2023.codfw.wmnet -> wdqs2026.codfw.wmnet w/ force delete existing files, repooling source-only afterwards |
[production] |
| 21:00 |
<mforns> |
re-ran commons_impact_metrics_monthly 2024-11 and cleared dependent DAGs which had timed out |
[analytics] |
| 20:56 |
<mforns> |
deployed airflow-analytics to fix the commons impact metrics pipeline |
[analytics] |
| 20:55 |
<mforns@deploy2002> |
Finished deploy [airflow-dags/analytics@2af4e1a]: Fix for the Commons Impact Metrics job (duration: 01m 38s) |
[production] |
| 20:54 |
<mforns@deploy2002> |
Started deploy [airflow-dags/analytics@2af4e1a]: Fix for the Commons Impact Metrics job |
[production] |
| 20:50 |
<mforns> |
finished deployment of refinery to fix Commons Impact Metrics job |
[analytics] |
| 20:47 |
<mforns@deploy2002> |
Finished deploy [analytics/refinery@25c1946] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@25c1946c] (duration: 00m 27s) |
[production] |
| 20:46 |
<mforns@deploy2002> |
Started deploy [analytics/refinery@25c1946] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@25c1946c] |
[production] |
| 20:46 |
<mforns@deploy2002> |
Finished deploy [analytics/refinery@25c1946] (thin): Regular analytics weekly train THIN [analytics/refinery@25c1946c] (duration: 00m 31s) |
[production] |
| 20:45 |
<mforns@deploy2002> |
Started deploy [analytics/refinery@25c1946] (thin): Regular analytics weekly train THIN [analytics/refinery@25c1946c] |
[production] |
| 20:45 |
<mforns@deploy2002> |
Finished deploy [analytics/refinery@25c1946]: Regular analytics weekly train [analytics/refinery@25c1946c] (duration: 13m 12s) |
[production] |
| 20:38 |
<ryankemper@cumin2002> |
START - Cookbook sre.wdqs.data-transfer (T376150, xfer wdqs scholarly 2023(public)->2026(internal)) xfer scholarly_articles from wdqs2023.codfw.wmnet -> wdqs2026.codfw.wmnet w/ force delete existing files, repooling source-only afterwards |
[production] |
| 20:38 |
<ryankemper@cumin2002> |
END (ERROR) - Cookbook sre.wdqs.data-transfer (exit_code=97) (T376150, xfer wdqs scholarly 2023(public)->2026(internal)) xfer scholarly_articles from wdqs2023.codfw.wmnet -> wdqs2026.codfw.wmnet, repooling source-only afterwards |
[production] |
| 20:37 |
<ryankemper@cumin2002> |
START - Cookbook sre.wdqs.data-transfer (T376150, xfer wdqs scholarly 2023(public)->2026(internal)) xfer scholarly_articles from wdqs2023.codfw.wmnet -> wdqs2026.codfw.wmnet, repooling source-only afterwards |
[production] |
| 20:32 |
<mforns@deploy2002> |
Started deploy [analytics/refinery@25c1946]: Regular analytics weekly train [analytics/refinery@25c1946c] |
[production] |
| 20:31 |
<mforns> |
starting deployment of refinery to fix Commons Impact Metrics job |
[analytics] |
| 20:28 |
<jhathaway@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on ms-be1088.eqiad.wmnet with reason: T381919 |
[production] |
| 20:28 |
<jhathaway@cumin1002> |
START - Cookbook sre.hosts.downtime for 4:00:00 on ms-be1088.eqiad.wmnet with reason: T381919 |
[production] |
| 20:04 |
<jhathaway@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on ms-be1088.eqiad.wmnet with reason: T381919 |
[production] |
| 20:04 |
<jhathaway@cumin1002> |
START - Cookbook sre.hosts.downtime for 3:00:00 on ms-be1088.eqiad.wmnet with reason: T381919 |
[production] |
| 18:35 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2158 (re)pooling @ 100%: Pooling in production', diff saved to https://phabricator.wikimedia.org/P71693 and previous config saved to /var/cache/conftool/dbconfig/20241210-183545-root.json |
[production] |
| 18:20 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2158 (re)pooling @ 75%: Pooling in production', diff saved to https://phabricator.wikimedia.org/P71692 and previous config saved to /var/cache/conftool/dbconfig/20241210-182040-root.json |
[production] |
| 18:05 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2158 (re)pooling @ 50%: Pooling in production', diff saved to https://phabricator.wikimedia.org/P71691 and previous config saved to /var/cache/conftool/dbconfig/20241210-180534-root.json |
[production] |
| 18:02 |
<dbrant@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/mobileapps: apply |
[production] |
| 18:02 |
<dbrant@deploy2002> |
helmfile [codfw] START helmfile.d/services/mobileapps: apply |
[production] |
| 18:02 |
<dbrant@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply |
[production] |
| 18:01 |
<elukey@cumin1002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1002" |
[production] |
| 18:01 |
<dbrant@deploy2002> |
helmfile [eqiad] START helmfile.d/services/mobileapps: apply |
[production] |
| 18:00 |
<dbrant@deploy2002> |
helmfile [staging] DONE helmfile.d/services/mobileapps: apply |
[production] |
| 18:00 |
<dbrant@deploy2002> |
helmfile [staging] START helmfile.d/services/mobileapps: apply |
[production] |
| 17:55 |
<hnowlan@deploy1003> |
helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 17:54 |
<hnowlan@deploy1003> |
helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 17:50 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2158 (re)pooling @ 25%: Pooling in production', diff saved to https://phabricator.wikimedia.org/P71690 and previous config saved to /var/cache/conftool/dbconfig/20241210-175029-root.json |
[production] |
| 17:47 |
<hnowlan@deploy1003> |
helmfile [codfw] DONE helmfile.d/services/mw-videoscaler: apply |
[production] |
| 17:47 |
<hnowlan@deploy1003> |
helmfile [codfw] START helmfile.d/services/mw-videoscaler: apply |
[production] |
| 17:42 |
<hnowlan@deploy1003> |
helmfile [codfw] DONE helmfile.d/services/mw-videoscaler: apply |
[production] |
| 17:42 |
<hnowlan@deploy1003> |
helmfile [codfw] START helmfile.d/services/mw-videoscaler: apply |
[production] |
| 17:42 |
<hnowlan@deploy1003> |
helmfile [eqiad] DONE helmfile.d/services/mw-videoscaler: apply |
[production] |
| 17:41 |
<hnowlan@deploy1003> |
helmfile [eqiad] START helmfile.d/services/mw-videoscaler: apply |
[production] |
| 17:35 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2158 (re)pooling @ 10%: Pooling in production', diff saved to https://phabricator.wikimedia.org/P71688 and previous config saved to /var/cache/conftool/dbconfig/20241210-173524-root.json |
[production] |
| 17:30 |
<elukey@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-lab1002.eqiad.wmnet with reason: host reimage |
[production] |
| 17:29 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2158.codfw.wmnet with reason: maintenance |
[production] |
| 17:28 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 1:00:00 on db2158.codfw.wmnet with reason: maintenance |
[production] |
| 17:25 |
<elukey@cumin1002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on ml-lab1002.eqiad.wmnet with reason: host reimage |
[production] |
| 17:24 |
<herron@cumin1002> |
dbctl commit (dc=all): 'depooling db2158 T381901', diff saved to https://phabricator.wikimedia.org/P71687 and previous config saved to /var/cache/conftool/dbconfig/20241210-172424-herron.json |
[production] |
| 17:18 |
<hnowlan@deploy1003> |
helmfile [eqiad] DONE helmfile.d/services/mw-videoscaler: apply |
[production] |
| 17:18 |
<hnowlan@deploy1003> |
helmfile [eqiad] START helmfile.d/services/mw-videoscaler: apply |
[production] |
| 17:13 |
<elukey@cumin1002> |
START - Cookbook sre.hosts.reimage for host ml-lab1002.eqiad.wmnet with OS bookworm |
[production] |