|
2024-12-06
§
|
| 05:53 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on db1154.eqiad.wmnet with reason: Alter table |
[production] |
| 05:50 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es2044 (re)pooling @ 1%: Pooling in production', diff saved to https://phabricator.wikimedia.org/P71624 and previous config saved to /var/cache/conftool/dbconfig/20241206-055047-root.json |
[production] |
| 05:44 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Add es2044 to dbctl depooled T381259', diff saved to https://phabricator.wikimedia.org/P71623 and previous config saved to /var/cache/conftool/dbconfig/20241206-054457-marostegui.json |
[production] |
| 05:40 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es2023 (re)pooling @ 25%: 10', diff saved to https://phabricator.wikimedia.org/P71622 and previous config saved to /var/cache/conftool/dbconfig/20241206-054010-root.json |
[production] |
| 01:50 |
<amastilovic@deploy2002> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply |
[production] |
| 01:49 |
<amastilovic@deploy2002> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply |
[production] |
| 01:49 |
<amastilovic@deploy2002> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply |
[production] |
| 01:48 |
<amastilovic@deploy2002> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply |
[production] |
| 01:47 |
<amastilovic@deploy2002> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply |
[production] |
| 01:47 |
<amastilovic@deploy2002> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply |
[production] |
| 01:47 |
<amastilovic@deploy2002> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply |
[production] |
| 01:47 |
<amastilovic@deploy2002> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply |
[production] |
| 01:32 |
<TimStarling> |
on mwmaint2002: deleting [[MediaWiki:Sitesupport-url]] pages per T379205 |
[production] |
| 01:16 |
<amastilovic@deploy2002> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply |
[production] |
| 01:15 |
<amastilovic@deploy2002> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply |
[production] |
| 00:19 |
<urbanecm> |
mwmaint2002: foreachwikiindblist growthexperiments extensions/GrowthExperiments/maintenance/revalidateLinkRecommendations.php --all --verbose # T380455 |
[production] |
| 00:19 |
<urbanecm> |
Delete previously-started mwscript-k8s instances of revalidateLinkRecommendations.php (T380455) |
[production] |
|
2024-12-05
§
|
| 23:26 |
<jhathaway> |
looking at puppet failures on an-workers |
[production] |
| 23:23 |
<urbanecm> |
Start revalidateLinkRecommendations.php for Add Link-enabled wikis via mwscript-k8s (T380455) |
[production] |
| 22:53 |
<bking@cumin2002> |
END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T376150, initialize wdqs internal main tier) xfer wikidata_main from wdqs2021.codfw.wmnet -> wdqs2020.codfw.wmnet w/ force delete existing files, repooling both afterwards |
[production] |
| 22:00 |
<bking@cumin2002> |
START - Cookbook sre.wdqs.data-transfer (T376150, initialize wdqs internal main tier) xfer wikidata_main from wdqs2021.codfw.wmnet -> wdqs2020.codfw.wmnet w/ force delete existing files, repooling both afterwards |
[production] |
| 21:24 |
<cdanis@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/chart-renderer: apply |
[production] |
| 21:23 |
<cdanis@deploy2002> |
helmfile [eqiad] START helmfile.d/services/chart-renderer: apply |
[production] |
| 21:21 |
<cdanis@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/chart-renderer: apply |
[production] |
| 21:21 |
<cdanis@deploy2002> |
helmfile [codfw] START helmfile.d/services/chart-renderer: apply |
[production] |
| 21:20 |
<cdanis@deploy2002> |
helmfile [staging] DONE helmfile.d/services/chart-renderer: apply |
[production] |
| 21:19 |
<cdanis@deploy2002> |
helmfile [staging] START helmfile.d/services/chart-renderer: apply |
[production] |
| 20:27 |
<bking@cumin2002> |
END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T376150, initialize wdqs internal main tier) xfer wikidata_main from wdqs2021.codfw.wmnet -> wdqs2019.codfw.wmnet w/ force delete existing files, repooling both afterwards |
[production] |
| 19:34 |
<bking@cumin2002> |
START - Cookbook sre.wdqs.data-transfer (T376150, initialize wdqs internal main tier) xfer wikidata_main from wdqs2021.codfw.wmnet -> wdqs2019.codfw.wmnet w/ force delete existing files, repooling both afterwards |
[production] |
| 19:28 |
<jhuneidi@deploy2002> |
rebuilt and synchronized wikiversions files: group2 to 1.44.0-wmf.6 refs T375665 |
[production] |
| 18:44 |
<jhancock@cumin2002> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host es1043.eqiad.wmnet with OS bookworm |
[production] |
| 18:13 |
<jelto@cumin1002> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1048-1049,1051].eqiad.wmnet |
[production] |
| 18:13 |
<jelto@cumin1002> |
START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1048-1049,1051].eqiad.wmnet |
[production] |
| 18:00 |
<jhancock@cumin2002> |
START - Cookbook sre.hosts.reimage for host es1043.eqiad.wmnet with OS bookworm |
[production] |
| 18:00 |
<jhancock@cumin2002> |
END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host es1043.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED |
[production] |
| 17:59 |
<jelto> |
homer 'cr*eqiad*' commit 'T377876' |
[production] |
| 17:57 |
<jelto@cumin1002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1051.eqiad.wmnet with OS bookworm |
[production] |
| 17:53 |
<jhancock@cumin2002> |
START - Cookbook sre.hosts.provision for host es1043.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED |
[production] |
| 17:53 |
<jelto@cumin1002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1050.eqiad.wmnet with OS bookworm |
[production] |
| 17:41 |
<pt1979@cumin1002> |
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudelastic1011.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART |
[production] |
| 17:40 |
<pt1979@cumin1002> |
START - Cookbook sre.hosts.provision for host cloudelastic1011.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART |
[production] |
| 17:39 |
<jelto@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1051.eqiad.wmnet with reason: host reimage |
[production] |
| 17:36 |
<jelto@cumin1002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1051.eqiad.wmnet with reason: host reimage |
[production] |
| 17:36 |
<jhathaway> |
upgrading facter on bullseye puppet nodes |
[production] |
| 17:34 |
<jelto@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1050.eqiad.wmnet with reason: host reimage |
[production] |
| 17:30 |
<jelto@cumin1002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1050.eqiad.wmnet with reason: host reimage |
[production] |
| 17:25 |
<ladsgroup@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1125.eqiad.wmnet with reason: Test setup should not alert |
[production] |
| 17:25 |
<ladsgroup@cumin1002> |
START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on db1125.eqiad.wmnet with reason: Test setup should not alert |
[production] |
| 17:21 |
<jclark@cumin1002> |
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudelastic1011.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART |
[production] |
| 17:21 |
<jclark@cumin1002> |
START - Cookbook sre.hosts.provision for host cloudelastic1011.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART |
[production] |