|
2024-12-10
ยง
|
| 22:10 |
<cjming@deploy2002> |
cwhite, cjming: Continuing with sync |
[production] |
| 22:08 |
<cjming@deploy2002> |
cwhite, cjming: Backport for [[gerrit:1100864|Disable stats collection when WMF_MAINTENANCE_OFFLINE is set (T380609)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 22:04 |
<cjming@deploy2002> |
Started scap sync-world: Backport for [[gerrit:1100864|Disable stats collection when WMF_MAINTENANCE_OFFLINE is set (T380609)]] |
[production] |
| 21:59 |
<cjming@deploy2002> |
Finished scap sync-world: Backport for [[gerrit:1101840|Beta Cluster: Enable MetricsPlatform extension on all wikis (T381849 T381853)]] (duration: 10m 50s) |
[production] |
| 21:56 |
<jhathaway@cumin1002> |
START - Cookbook sre.hosts.reimage for host ms-be1088.eqiad.wmnet with OS bookworm |
[production] |
| 21:53 |
<cjming@deploy2002> |
cjming, phuedx: Continuing with sync |
[production] |
| 21:52 |
<cjming@deploy2002> |
cjming, phuedx: Backport for [[gerrit:1101840|Beta Cluster: Enable MetricsPlatform extension on all wikis (T381849 T381853)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 21:48 |
<cjming@deploy2002> |
Started scap sync-world: Backport for [[gerrit:1101840|Beta Cluster: Enable MetricsPlatform extension on all wikis (T381849 T381853)]] |
[production] |
| 21:47 |
<eileen> |
ivicrm upgraded from f9c89e50 to 3ef855ca |
[production] |
| 21:47 |
<cjming@deploy2002> |
Finished scap sync-world: Backport for [[gerrit:1101875|Reader Survey: Increase coverage (T378660)]] (duration: 10m 02s) |
[production] |
| 21:41 |
<cjming@deploy2002> |
cjming, dani: Continuing with sync |
[production] |
| 21:41 |
<cjming@deploy2002> |
cjming, dani: Backport for [[gerrit:1101875|Reader Survey: Increase coverage (T378660)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 21:37 |
<cjming@deploy2002> |
Started scap sync-world: Backport for [[gerrit:1101875|Reader Survey: Increase coverage (T378660)]] |
[production] |
| 21:35 |
<cjming@deploy2002> |
Finished scap sync-world: Backport for [[gerrit:1101600|LanguageConverter: Ignore content inside <math> and <svg> elements (T381617)]] (duration: 11m 55s) |
[production] |
| 21:30 |
<cjming@deploy2002> |
bvibber, cjming: Continuing with sync |
[production] |
| 21:27 |
<cjming@deploy2002> |
bvibber, cjming: Backport for [[gerrit:1101600|LanguageConverter: Ignore content inside <math> and <svg> elements (T381617)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 21:23 |
<cjming@deploy2002> |
Started scap sync-world: Backport for [[gerrit:1101600|LanguageConverter: Ignore content inside <math> and <svg> elements (T381617)]] |
[production] |
| 21:22 |
<ryankemper@cumin2002> |
END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T376150, xfer wdqs scholarly 2023(public)->2026(internal)) xfer scholarly_articles from wdqs2023.codfw.wmnet -> wdqs2026.codfw.wmnet w/ force delete existing files, repooling source-only afterwards |
[production] |
| 21:00 |
<mforns> |
re-ran commons_impact_metrics_monthly 2024-11 and cleared dependent DAGs which had timed out |
[analytics] |
| 20:56 |
<mforns> |
deployed airflow-analytics to fix the commons impact metrics pipeline |
[analytics] |
| 20:55 |
<mforns@deploy2002> |
Finished deploy [airflow-dags/analytics@2af4e1a]: Fix for the Commons Impact Metrics job (duration: 01m 38s) |
[production] |
| 20:54 |
<mforns@deploy2002> |
Started deploy [airflow-dags/analytics@2af4e1a]: Fix for the Commons Impact Metrics job |
[production] |
| 20:50 |
<mforns> |
finished deployment of refinery to fix Commons Impact Metrics job |
[analytics] |
| 20:47 |
<mforns@deploy2002> |
Finished deploy [analytics/refinery@25c1946] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@25c1946c] (duration: 00m 27s) |
[production] |
| 20:46 |
<mforns@deploy2002> |
Started deploy [analytics/refinery@25c1946] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@25c1946c] |
[production] |
| 20:46 |
<mforns@deploy2002> |
Finished deploy [analytics/refinery@25c1946] (thin): Regular analytics weekly train THIN [analytics/refinery@25c1946c] (duration: 00m 31s) |
[production] |
| 20:45 |
<mforns@deploy2002> |
Started deploy [analytics/refinery@25c1946] (thin): Regular analytics weekly train THIN [analytics/refinery@25c1946c] |
[production] |
| 20:45 |
<mforns@deploy2002> |
Finished deploy [analytics/refinery@25c1946]: Regular analytics weekly train [analytics/refinery@25c1946c] (duration: 13m 12s) |
[production] |
| 20:38 |
<ryankemper@cumin2002> |
START - Cookbook sre.wdqs.data-transfer (T376150, xfer wdqs scholarly 2023(public)->2026(internal)) xfer scholarly_articles from wdqs2023.codfw.wmnet -> wdqs2026.codfw.wmnet w/ force delete existing files, repooling source-only afterwards |
[production] |
| 20:38 |
<ryankemper@cumin2002> |
END (ERROR) - Cookbook sre.wdqs.data-transfer (exit_code=97) (T376150, xfer wdqs scholarly 2023(public)->2026(internal)) xfer scholarly_articles from wdqs2023.codfw.wmnet -> wdqs2026.codfw.wmnet, repooling source-only afterwards |
[production] |
| 20:37 |
<ryankemper@cumin2002> |
START - Cookbook sre.wdqs.data-transfer (T376150, xfer wdqs scholarly 2023(public)->2026(internal)) xfer scholarly_articles from wdqs2023.codfw.wmnet -> wdqs2026.codfw.wmnet, repooling source-only afterwards |
[production] |
| 20:32 |
<mforns@deploy2002> |
Started deploy [analytics/refinery@25c1946]: Regular analytics weekly train [analytics/refinery@25c1946c] |
[production] |
| 20:31 |
<mforns> |
starting deployment of refinery to fix Commons Impact Metrics job |
[analytics] |
| 20:28 |
<jhathaway@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on ms-be1088.eqiad.wmnet with reason: T381919 |
[production] |
| 20:28 |
<jhathaway@cumin1002> |
START - Cookbook sre.hosts.downtime for 4:00:00 on ms-be1088.eqiad.wmnet with reason: T381919 |
[production] |
| 20:04 |
<jhathaway@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on ms-be1088.eqiad.wmnet with reason: T381919 |
[production] |
| 20:04 |
<jhathaway@cumin1002> |
START - Cookbook sre.hosts.downtime for 3:00:00 on ms-be1088.eqiad.wmnet with reason: T381919 |
[production] |
| 18:35 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2158 (re)pooling @ 100%: Pooling in production', diff saved to https://phabricator.wikimedia.org/P71693 and previous config saved to /var/cache/conftool/dbconfig/20241210-183545-root.json |
[production] |
| 18:20 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2158 (re)pooling @ 75%: Pooling in production', diff saved to https://phabricator.wikimedia.org/P71692 and previous config saved to /var/cache/conftool/dbconfig/20241210-182040-root.json |
[production] |
| 18:05 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2158 (re)pooling @ 50%: Pooling in production', diff saved to https://phabricator.wikimedia.org/P71691 and previous config saved to /var/cache/conftool/dbconfig/20241210-180534-root.json |
[production] |
| 18:02 |
<dbrant@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/mobileapps: apply |
[production] |
| 18:02 |
<dbrant@deploy2002> |
helmfile [codfw] START helmfile.d/services/mobileapps: apply |
[production] |
| 18:02 |
<dbrant@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply |
[production] |
| 18:01 |
<elukey@cumin1002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1002" |
[production] |
| 18:01 |
<dbrant@deploy2002> |
helmfile [eqiad] START helmfile.d/services/mobileapps: apply |
[production] |
| 18:00 |
<dbrant@deploy2002> |
helmfile [staging] DONE helmfile.d/services/mobileapps: apply |
[production] |
| 18:00 |
<dbrant@deploy2002> |
helmfile [staging] START helmfile.d/services/mobileapps: apply |
[production] |
| 17:55 |
<hnowlan@deploy1003> |
helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 17:54 |
<hnowlan@deploy1003> |
helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 17:50 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2158 (re)pooling @ 25%: Pooling in production', diff saved to https://phabricator.wikimedia.org/P71690 and previous config saved to /var/cache/conftool/dbconfig/20241210-175029-root.json |
[production] |