|
2025-10-09
§
|
| 07:20 |
<ryankemper@cumin2002> |
END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T405978, transfer to freshly reimaged host) xfer wikidata_main from wdqs1017.eqiad.wmnet -> wdqs1018.eqiad.wmnet w/ force delete existing files, repooling both afterwards |
[production] |
| 07:17 |
<kharlan@deploy2002> |
Started scap sync-world: Backport for [[gerrit:1194666|hCaptcha: Provide capabilities for failing over to alternate CAPTCHA type (T404204)]] |
[production] |
| 07:15 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on es[1029,1034].eqiad.wmnet with reason: Cloning |
[production] |
| 07:14 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depool es1034 and es1029 T406488', diff saved to https://phabricator.wikimedia.org/P83697 and previous config saved to /var/cache/conftool/dbconfig/20251009-071430-marostegui.json |
[production] |
| 07:05 |
<moritzm> |
installing Redis security updates |
[production] |
| 06:53 |
<jelto@cumin1003> |
START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab |
[production] |
| 06:50 |
<ryankemper@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1019.eqiad.wmnet with OS bullseye |
[production] |
| 06:48 |
<jelto@cumin1003> |
END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Upgrade GitLab replica |
[production] |
| 06:39 |
<jelto@cumin1003> |
START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Upgrade GitLab replica |
[production] |
| 06:31 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'db2155 (re)pooling @ 100%: 10', diff saved to https://phabricator.wikimedia.org/P83696 and previous config saved to /var/cache/conftool/dbconfig/20251009-063106-root.json |
[production] |
| 06:28 |
<ryankemper@cumin2002> |
conftool action : set/pooled=no:weight=10; selector: name=wdqs1019.* |
[production] |
| 06:28 |
<ryankemper@cumin2002> |
START - Cookbook sre.wdqs.data-transfer (T405978, transfer to freshly reimaged host) xfer wikidata_main from wdqs1020.eqiad.wmnet -> wdqs1019.eqiad.wmnet w/ force delete existing files, repooling both afterwards |
[production] |
| 06:27 |
<ryankemper@cumin2002> |
START - Cookbook sre.wdqs.data-transfer (T405978, transfer to freshly reimaged host) xfer wikidata_main from wdqs1017.eqiad.wmnet -> wdqs1018.eqiad.wmnet w/ force delete existing files, repooling both afterwards |
[production] |
| 06:27 |
<ryankemper@cumin2002> |
END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) (T405978, transfer to freshly reimaged host) xfer wikidata_main from wdqs1017.eqiad.wmnet -> wdqs1018.eqiad.wmnet w/ force delete existing files, repooling both afterwards |
[production] |
| 06:27 |
<ryankemper@cumin2002> |
START - Cookbook sre.wdqs.data-transfer (T405978, transfer to freshly reimaged host) xfer wikidata_main from wdqs1017.eqiad.wmnet -> wdqs1018.eqiad.wmnet w/ force delete existing files, repooling both afterwards |
[production] |
| 06:26 |
<ryankemper@deploy2002> |
Finished deploy [wdqs/wdqs@fea7794]: deploy to fresh wdqs-main host (duration: 00m 13s) |
[production] |
| 06:26 |
<ryankemper@deploy2002> |
Started deploy [wdqs/wdqs@fea7794]: deploy to fresh wdqs-main host |
[production] |
| 06:26 |
<ryankemper@deploy2002> |
Finished deploy [wdqs/wdqs@fea7794]: deploy to fresh wdqs-main host (duration: 00m 14s) |
[production] |
| 06:26 |
<ryankemper@deploy2002> |
Started deploy [wdqs/wdqs@fea7794]: deploy to fresh wdqs-main host |
[production] |
| 06:22 |
<marostegui@cumin1003> |
END (PASS) - Cookbook sre.mysql.clone_es (exit_code=0) of es1027.eqiad.wmnet onto es1050.eqiad.wmnet |
[production] |
| 06:22 |
<marostegui@cumin1003> |
END (PASS) - Cookbook sre.mysql.pool (exit_code=0) es1027 gradually with 4 steps - Pool es1027.eqiad.wmnet in after cloning |
[production] |
| 06:16 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'db2155 (re)pooling @ 75%: 10', diff saved to https://phabricator.wikimedia.org/P83694 and previous config saved to /var/cache/conftool/dbconfig/20251009-061600-root.json |
[production] |
| 06:01 |
<marostegui@cumin1003> |
END (PASS) - Cookbook sre.mysql.clone_es (exit_code=0) of es1030.eqiad.wmnet onto es1053.eqiad.wmnet |
[production] |
| 06:01 |
<marostegui@cumin1003> |
END (PASS) - Cookbook sre.mysql.pool (exit_code=0) es1030 gradually with 4 steps - Pool es1030.eqiad.wmnet in after cloning |
[production] |
| 06:00 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'db2155 (re)pooling @ 50%: 10', diff saved to https://phabricator.wikimedia.org/P83691 and previous config saved to /var/cache/conftool/dbconfig/20251009-060054-root.json |
[production] |
| 05:45 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'db2155 (re)pooling @ 25%: 10', diff saved to https://phabricator.wikimedia.org/P83688 and previous config saved to /var/cache/conftool/dbconfig/20251009-054548-root.json |
[production] |
| 05:43 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Add es1050 and es1053 depooled T406488', diff saved to https://phabricator.wikimedia.org/P83687 and previous config saved to /var/cache/conftool/dbconfig/20251009-054347-marostegui.json |
[production] |
| 05:37 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depool db2155 for migration to mariadb 10.11', diff saved to https://phabricator.wikimedia.org/P83686 and previous config saved to /var/cache/conftool/dbconfig/20251009-053730-marostegui.json |
[production] |
| 05:37 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2155.codfw.wmnet with reason: Maintenance |
[production] |
| 05:36 |
<marostegui@cumin1003> |
START - Cookbook sre.mysql.pool es1027 gradually with 4 steps - Pool es1027.eqiad.wmnet in after cloning |
[production] |
| 05:16 |
<marostegui@cumin1003> |
START - Cookbook sre.mysql.pool es1030 gradually with 4 steps - Pool es1030.eqiad.wmnet in after cloning |
[production] |
| 04:13 |
<eileen> |
civicrm upgraded from 6f24d513 to 132211d5 |
[production] |
| 02:11 |
<dzahn@cumin2002> |
END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: security release 20251008 |
[production] |
| 02:02 |
<dzahn@cumin2002> |
START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: security release 20251008 |
[production] |
| 01:54 |
<mutante> |
[wdqs1020:~] $ sudo systemctl restart wdqs-blazegraph |
[production] |
| 01:32 |
<eileen> |
civicrm upgraded from 4c13f904 to 6f24d513 |
[production] |
| 01:18 |
<eileen> |
civicrm upgraded from 2c6fedc8 to 4c13f904 |
[production] |
| 01:15 |
<mwpresync@deploy2002> |
Finished scap build-images: Publishing wmf/next image (duration: 14m 20s) |
[production] |
| 01:00 |
<mwpresync@deploy2002> |
Started scap build-images: Publishing wmf/next image |
[production] |
|
2025-10-08
§
|
| 23:58 |
<ryankemper@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1019.eqiad.wmnet with reason: host reimage |
[production] |
| 23:54 |
<ryankemper@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1018.eqiad.wmnet with reason: host reimage |
[production] |
| 23:50 |
<ryankemper@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1019.eqiad.wmnet with reason: host reimage |
[production] |
| 23:50 |
<ryankemper@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1018.eqiad.wmnet with reason: host reimage |
[production] |
| 22:09 |
<ryankemper@cumin2002> |
END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T405978, transfer to freshly reimaged host) xfer scholarly_articles from wdqs2016.codfw.wmnet -> wdqs2017.codfw.wmnet w/ force delete existing files, repooling source-only afterwards |
[production] |
| 21:47 |
<wmbot~dcaro@acme> |
END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-76 |
[tools] |
| 21:25 |
<ryankemper@cumin2002> |
START - Cookbook sre.hosts.reimage for host wdqs1019.eqiad.wmnet with OS bullseye |
[production] |
| 21:19 |
<wmbot~dcaro@acme> |
START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-76 |
[tools] |
| 21:19 |
<ryankemper@cumin2002> |
START - Cookbook sre.wdqs.data-transfer (T405978, transfer to freshly reimaged host) xfer scholarly_articles from wdqs2016.codfw.wmnet -> wdqs2017.codfw.wmnet w/ force delete existing files, repooling source-only afterwards |
[production] |
| 21:19 |
<ryankemper@cumin2002> |
END (ERROR) - Cookbook sre.wdqs.data-transfer (exit_code=97) (T405978, transfer to freshly reimaged host) xfer scholarly_articles from wdqs2016.codfw.wmnet -> wdqs2017.codfw.wmnet w/ force delete existing files, repooling source-only afterwards |
[production] |
| 21:18 |
<ryankemper@cumin2002> |
START - Cookbook sre.wdqs.data-transfer (T405978, transfer to freshly reimaged host) xfer scholarly_articles from wdqs2016.codfw.wmnet -> wdqs2017.codfw.wmnet w/ force delete existing files, repooling source-only afterwards |
[production] |