|
2024-03-13
§
|
| 17:47 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1182 (re)pooling @ 25%: After recloning db1246', diff saved to https://phabricator.wikimedia.org/P58783 and previous config saved to /var/cache/conftool/dbconfig/20240313-174740-root.json |
[production] |
| 17:44 |
<logmsgbot> |
@deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 17:43 |
<logmsgbot> |
@deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 17:35 |
<logmsgbot> |
@deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 17:34 |
<logmsgbot> |
@deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 17:32 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1182 (re)pooling @ 10%: After recloning db1246', diff saved to https://phabricator.wikimedia.org/P58782 and previous config saved to /var/cache/conftool/dbconfig/20240313-173234-root.json |
[production] |
| 17:32 |
<James_F> |
Zuul: Make 'standalone' jobs run consecutively, not concurrently |
[releng] |
| 17:26 |
<logmsgbot> |
@deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 17:26 |
<logmsgbot> |
@deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 17:17 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1182 (re)pooling @ 5%: After recloning db1246', diff saved to https://phabricator.wikimedia.org/P58781 and previous config saved to /var/cache/conftool/dbconfig/20240313-171728-root.json |
[production] |
| 17:02 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1182 (re)pooling @ 1%: After recloning db1246', diff saved to https://phabricator.wikimedia.org/P58779 and previous config saved to /var/cache/conftool/dbconfig/20240313-170222-root.json |
[production] |
| 17:00 |
<elukey@deploy2002> |
helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'readability' for release 'main' . |
[production] |
| 16:52 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1182.eqiad.wmnet onto db1246.eqiad.wmnet |
[production] |
| 16:48 |
<root@cumin1002> |
END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) status all services in all: None - None |
[production] |
| 16:48 |
<root@cumin1002> |
START - Cookbook sre.discovery.datacenter status all services in all: None - None |
[production] |
| 16:33 |
<eevans@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 30 days, 0:00:00 on restbase1021.eqiad.wmnet with reason: Decommissioning — T354561 |
[production] |
| 16:33 |
<eevans@cumin1002> |
START - Cookbook sre.hosts.downtime for 30 days, 0:00:00 on restbase1021.eqiad.wmnet with reason: Decommissioning — T354561 |
[production] |
| 16:32 |
<root@cumin1002> |
END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) status all services in all: None - None |
[production] |
| 16:32 |
<root@cumin1002> |
START - Cookbook sre.discovery.datacenter status all services in all: None - None |
[production] |
| 16:21 |
<wmbot~bd808@tools-sgebastion-11> |
Changed branch protections on https://gitlab.wikimedia.org/toolforge-repos/wikibugs2 to disallow push to main by all. fa4fbd54 was accidentally pushed to main rather than a feature branch. |
[tools.wikibugs] |
| 16:18 |
<wmbot~bd808@tools-sgebastion-11> |
Restared gerrit task to pick up fa4fbd54 fix for T360038 |
[tools.wikibugs] |
| 16:15 |
<aborrero@cloudcumin1001> |
END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 (T359638) |
[toolsbeta] |
| 16:15 |
<aborrero@cloudcumin1001> |
START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 (T359638) |
[toolsbeta] |
| 16:14 |
<aborrero@cloudcumin1001> |
END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 (T359638) |
[toolsbeta] |
| 16:14 |
<aborrero@cloudcumin1001> |
START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 (T359638) |
[toolsbeta] |
| 16:08 |
<wmbot~bd808@tools-sgebastion-11> |
Restared gerrit task to pick up 82caa471 fix for T360038 |
[tools.wikibugs] |
| 16:07 |
<logmsgbot> |
@deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 16:07 |
<logmsgbot> |
@deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 16:01 |
<hashar@deploy2002> |
Synchronized php: group1 wikis to 1.42.0-wmf.22 refs T354440 (duration: 11m 46s) |
[production] |
| 15:49 |
<aborrero@cloudcumin1001> |
END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.23 to 1.24 (T359638) |
[toolsbeta] |
| 15:49 |
<hashar@deploy2002> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.42.0-wmf.22 refs T354440 |
[production] |
| 15:48 |
<aborrero@cloudcumin1001> |
START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.23 to 1.24 (T359638) |
[toolsbeta] |
| 15:32 |
<logmsgbot> |
lucaswerkmeister-wmde@deploy2002 Finished scap: Backport for [[gerrit:1010581|WikiModule: Fix pages merging (T360014)]] (duration: 13m 27s) |
[production] |
| 15:27 |
<marostegui@cumin1002> |
START - Cookbook sre.mysql.clone of db1182.eqiad.wmnet onto db1246.eqiad.wmnet |
[production] |
| 15:25 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1246.eqiad.wmnet with OS bookworm |
[production] |
| 15:25 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - marostegui@cumin1002" |
[production] |
| 15:24 |
<marostegui@cumin1002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - marostegui@cumin1002" |
[production] |
| 15:22 |
<logmsgbot> |
lucaswerkmeister-wmde@deploy2002 lucaswerkmeister-wmde: Continuing with sync |
[production] |
| 15:21 |
<logmsgbot> |
lucaswerkmeister-wmde@deploy2002 lucaswerkmeister-wmde: Backport for [[gerrit:1010581|WikiModule: Fix pages merging (T360014)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 15:19 |
<logmsgbot> |
lucaswerkmeister-wmde@deploy2002 Started scap: Backport for [[gerrit:1010581|WikiModule: Fix pages merging (T360014)]] |
[production] |
| 15:04 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1246.eqiad.wmnet with reason: host reimage |
[production] |
| 15:04 |
<wmbot~taavi@tools-sgebastion-11> |
toolforge jobs restart gerrit |
[tools.wikibugs] |
| 15:03 |
<wmbot~lucaswerkmeister@tools-sgebastion-10> |
toolforge jobs restart irc |
[tools.wikibugs] |
| 15:01 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db1246.eqiad.wmnet with reason: host reimage |
[production] |
| 14:56 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1182.eqiad.wmnet with reason: Cloning db1246 |
[production] |
| 14:56 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1182.eqiad.wmnet with reason: Cloning db1246 |
[production] |
| 14:56 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool db1182 to clone db1246', diff saved to https://phabricator.wikimedia.org/P58778 and previous config saved to /var/cache/conftool/dbconfig/20240313-145603-marostegui.json |
[production] |
| 14:49 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.reimage for host db1246.eqiad.wmnet with OS bookworm |
[production] |
| 14:49 |
<marostegui@cumin1002> |
END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host db1246.eqiad.wmnet with OS bookworm |
[production] |
| 14:33 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.reimage for host db1246.eqiad.wmnet with OS bookworm |
[production] |