|
2026-03-10
ยง
|
| 15:55 |
<brouberol@deploy2002> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-fr-tech: apply |
[production] |
| 15:55 |
<brouberol@deploy2002> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-fr-tech: apply |
[production] |
| 15:55 |
<jynus@deploy2002> |
helmfile [codfw] START helmfile.d/services/mw-cron: apply |
[production] |
| 15:48 |
<brouberol@deploy2002> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 15:46 |
<brouberol@deploy2002> |
helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 15:39 |
<elukey@cumin1003> |
END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-serve1015.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART |
[production] |
| 15:34 |
<elukey@cumin1003> |
START - Cookbook sre.hosts.provision for host ml-serve1015.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART |
[production] |
| 15:28 |
<brouberol@dns1004> |
END - running authdns-update |
[production] |
| 15:27 |
<brouberol@dns1004> |
START - running authdns-update |
[production] |
| 15:10 |
<swfrench@cumin2002> |
END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "Deploy: Fix an edge case in validation of a new object - swfrench@cumin2002" |
[production] |
| 15:10 |
<swfrench@cumin2002> |
END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Deploy: Fix an edge case in validation of a new object - swfrench@cumin2002 |
[production] |
| 15:09 |
<swfrench@cumin2002> |
START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Deploy: Fix an edge case in validation of a new object - swfrench@cumin2002 |
[production] |
| 15:09 |
<swfrench@cumin2002> |
START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "Deploy: Fix an edge case in validation of a new object - swfrench@cumin2002" |
[production] |
| 15:05 |
<elukey@cumin1003> |
END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-serve1014.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART |
[production] |
| 14:59 |
<elukey@cumin1003> |
START - Cookbook sre.hosts.provision for host ml-serve1014.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART |
[production] |
| 14:58 |
<sukhe> |
sudo cumin -b1 -s15 "C:bird" "run-puppet-agent --enable 'merging CR 1238007; add function return type'" |
[production] |
| 14:58 |
<elukey@cumin1003> |
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-serve1014.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART |
[production] |
| 14:58 |
<elukey@cumin1003> |
START - Cookbook sre.hosts.provision for host ml-serve1014.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART |
[production] |
| 14:58 |
<sukhe> |
sudo cumin -b1 -s15 "C:bird" "run-puppet-agent 'merging CR 1238007; add function return type'" |
[production] |
| 14:51 |
<elukey@cumin1003> |
END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host thanos-be1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL |
[production] |
| 14:44 |
<elukey@cumin1003> |
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-serve1014.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART |
[production] |
| 14:44 |
<elukey@cumin1003> |
START - Cookbook sre.hosts.provision for host ml-serve1014.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART |
[production] |
| 14:42 |
<sukhe> |
sudo cumin "C:bird" "disable-puppet 'merging CR 1238007; add function return type'" |
[production] |
| 14:41 |
<elukey@cumin1003> |
END (PASS) - Cookbook sre.hosts.powercycle (exit_code=0) for host ml-serve1014 |
[production] |
| 14:39 |
<elukey@cumin1003> |
START - Cookbook sre.hosts.provision for host thanos-be1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL |
[production] |
| 14:36 |
<elukey@cumin1003> |
START - Cookbook sre.hosts.powercycle for host ml-serve1014 |
[production] |
| 14:36 |
<elukey@cumin1003> |
END (FAIL) - Cookbook sre.hosts.powercycle (exit_code=99) for host ml-serve1014 |
[production] |
| 14:36 |
<elukey@cumin1003> |
START - Cookbook sre.hosts.powercycle for host ml-serve1014 |
[production] |
| 14:12 |
<otto@deploy2002> |
Finished scap sync-world: Backport for [[gerrit:1249367|stream: mediawiki.page_edit_type_simple.dev0 (T351225)]] (duration: 11m 05s) |
[production] |
| 14:08 |
<otto@deploy2002> |
akhatun, otto: Continuing with sync |
[production] |
| 14:02 |
<otto@deploy2002> |
akhatun, otto: Backport for [[gerrit:1249367|stream: mediawiki.page_edit_type_simple.dev0 (T351225)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. |
[production] |
| 14:01 |
<otto@deploy2002> |
Started scap sync-world: Backport for [[gerrit:1249367|stream: mediawiki.page_edit_type_simple.dev0 (T351225)]] |
[production] |
| 13:49 |
<vriley@cumin1003> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host contint1003.wikimedia.org with OS trixie |
[production] |
| 13:43 |
<javiermonton@deploy2002> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply |
[production] |
| 13:42 |
<javiermonton@deploy2002> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply |
[production] |
| 13:28 |
<vgutierrez> |
testing acme-chief 0.39 in acmechief-test2001 - T419352 |
[production] |
| 13:27 |
<vgutierrez> |
upload acme-chief 0.39 to bookworm-wikimedia (apt.wm.o) - T419352 |
[production] |
| 13:16 |
<jiji@cumin1003> |
END (FAIL) - Cookbook sre.memcached.roll-reboot-restart (exit_code=1) rolling restart_daemons on A:memcached-canary |
[production] |
| 13:16 |
<jiji@cumin1003> |
START - Cookbook sre.memcached.roll-reboot-restart rolling restart_daemons on A:memcached-canary |
[production] |
| 13:12 |
<mszwarc@deploy2002> |
Finished scap sync-world: Backport for [[gerrit:1249903|Require 2FA from 6 other user groups ($wgRestrictedGroups) (T418580)]], [[gerrit:1249035|kaiwiki: add logo, stiename, projectnamespace and timezone (T414237)]] (duration: 08m 45s) |
[production] |
| 13:08 |
<mszwarc@deploy2002> |
mszwarc, anzx: Continuing with sync |
[production] |
| 13:05 |
<mszwarc@deploy2002> |
mszwarc, anzx: Backport for [[gerrit:1249903|Require 2FA from 6 other user groups ($wgRestrictedGroups) (T418580)]], [[gerrit:1249035|kaiwiki: add logo, stiename, projectnamespace and timezone (T414237)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. |
[production] |
| 13:03 |
<mszwarc@deploy2002> |
Started scap sync-world: Backport for [[gerrit:1249903|Require 2FA from 6 other user groups ($wgRestrictedGroups) (T418580)]], [[gerrit:1249035|kaiwiki: add logo, stiename, projectnamespace and timezone (T414237)]] |
[production] |
| 13:03 |
<jiji@cumin1003> |
END (PASS) - Cookbook sre.memcached.roll-reboot-restart (exit_code=0) rolling reboot on A:memcached-gutter-eqiad |
[production] |
| 12:57 |
<jclark@cumin1003> |
START - Cookbook sre.hosts.reimage for host ml-serve1015.eqiad.wmnet with OS bookworm |
[production] |
| 12:56 |
<jiji@cumin1003> |
START - Cookbook sre.memcached.roll-reboot-restart rolling reboot on A:memcached-gutter-eqiad |
[production] |
| 12:51 |
<jclark@cumin1003> |
START - Cookbook sre.hosts.reimage for host ml-serve1014.eqiad.wmnet with OS bookworm |
[production] |
| 12:50 |
<jclark@cumin1003> |
END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ml-serve1014 |
[production] |
| 12:50 |
<jclark@cumin1003> |
START - Cookbook sre.network.configure-switch-interfaces for host ml-serve1014 |
[production] |
| 12:49 |
<jclark@cumin1003> |
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-serve1015.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART |
[production] |