|
2023-05-31
ยง
|
| 15:45 |
<vgutierrez> |
cp2035 depooled as puppet is unable to run due to ipmi issues - T337247 |
[production] |
| 15:42 |
<brett> |
Maglev LVS scheduler rollout began IN PROGRESS, not finished - T263797 |
[production] |
| 15:42 |
<brett> |
Maglev LVS scheduler rollout finished in codfw - T263797 |
[production] |
| 15:40 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on cp2035.codfw.wmnet with reason: ipmi/mgmt console issues |
[production] |
| 15:40 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on cp2035.codfw.wmnet with reason: ipmi/mgmt console issues |
[production] |
| 15:39 |
<vgutierrez@cumin1001> |
END (FAIL) - Cookbook sre.cdn.run-puppet-restart-varnish (exit_code=1) rolling custom on A:cp-text_codfw |
[production] |
| 14:55 |
<klausman@deploy1002> |
helmfile [staging] DONE helmfile.d/services/api-gateway: apply |
[production] |
| 14:55 |
<klausman@deploy1002> |
helmfile [staging] START helmfile.d/services/api-gateway: apply |
[production] |
| 14:54 |
<klausman@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply |
[production] |
| 14:54 |
<klausman@deploy1002> |
helmfile [eqiad] START helmfile.d/services/api-gateway: apply |
[production] |
| 14:50 |
<klausman@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/api-gateway: apply |
[production] |
| 14:50 |
<klausman@deploy1002> |
helmfile [codfw] START helmfile.d/services/api-gateway: apply |
[production] |
| 14:44 |
<bking@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/rdf-streaming-updater: apply |
[production] |
| 14:43 |
<bking@deploy1002> |
helmfile [eqiad] START helmfile.d/services/rdf-streaming-updater: apply |
[production] |
| 14:41 |
<klausman@deploy1002> |
helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' . |
[production] |
| 14:27 |
<bking@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/rdf-streaming-updater: apply |
[production] |
| 14:27 |
<bking@deploy1002> |
helmfile [codfw] START helmfile.d/services/rdf-streaming-updater: apply |
[production] |
| 14:25 |
<bking@deploy1002> |
helmfile [codfw] START helmfile.d/services/rdf-streaming-updater: apply |
[production] |
| 14:25 |
<bking@deploy1002> |
helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply |
[production] |
| 14:25 |
<bking@deploy1002> |
helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply |
[production] |
| 14:14 |
<urbanecm@deploy1002> |
Finished scap: Backport for [[gerrit:924941|NewImpact: Cache empty user impact on account creation (T337320)]], [[gerrit:924940|Personalized praise: Fix first-ever notifications (T322452)]] (duration: 07m 26s) |
[production] |
| 14:08 |
<urbanecm@deploy1002> |
urbanecm: Backport for [[gerrit:924941|NewImpact: Cache empty user impact on account creation (T337320)]], [[gerrit:924940|Personalized praise: Fix first-ever notifications (T322452)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet |
[production] |
| 14:07 |
<urbanecm@deploy1002> |
Started scap: Backport for [[gerrit:924941|NewImpact: Cache empty user impact on account creation (T337320)]], [[gerrit:924940|Personalized praise: Fix first-ever notifications (T322452)]] |
[production] |
| 14:02 |
<urbanecm@deploy1002> |
Finished scap: Backport for [[gerrit:924939|Personalized praise: Fix first-ever notifications (T322452)]], [[gerrit:924576|DeleteAction: Replace remaining OOUI fields (T337809)]] (duration: 11m 11s) |
[production] |
| 14:02 |
<hnowlan@cumin1001> |
END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P{lvs1019*,lvs2009*} and A:lvs (T329049) |
[production] |
| 13:58 |
<otto@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-page-content-change-enrich: apply |
[production] |
| 13:57 |
<otto@deploy1002> |
helmfile [codfw] START helmfile.d/services/mw-page-content-change-enrich: apply |
[production] |
| 13:56 |
<jayme@deploy1002> |
helmfile [eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 13:56 |
<jayme@deploy1002> |
helmfile [eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 13:55 |
<jayme@deploy1002> |
helmfile [eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 13:55 |
<jayme@deploy1002> |
helmfile [eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 13:55 |
<jayme@deploy1002> |
helmfile [eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 13:55 |
<jayme@deploy1002> |
helmfile [eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 13:54 |
<jayme@deploy1002> |
helmfile [codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 13:53 |
<jayme@deploy1002> |
helmfile [codfw] START helmfile.d/admin 'apply'. |
[production] |
| 13:53 |
<jayme@deploy1002> |
helmfile [codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 13:53 |
<jayme@deploy1002> |
helmfile [codfw] START helmfile.d/admin 'apply'. |
[production] |
| 13:53 |
<klausman@deploy1002> |
helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . |
[production] |
| 13:53 |
<urbanecm@deploy1002> |
daimona and urbanecm: Backport for [[gerrit:924939|Personalized praise: Fix first-ever notifications (T322452)]], [[gerrit:924576|DeleteAction: Replace remaining OOUI fields (T337809)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet |
[production] |
| 13:52 |
<jayme@deploy1002> |
helmfile [codfw] START helmfile.d/admin 'apply'. |
[production] |
| 13:51 |
<urbanecm@deploy1002> |
Started scap: Backport for [[gerrit:924939|Personalized praise: Fix first-ever notifications (T322452)]], [[gerrit:924576|DeleteAction: Replace remaining OOUI fields (T337809)]] |
[production] |
| 13:46 |
<hnowlan@cumin1001> |
START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P{lvs1019*,lvs2009*} and A:lvs (T329049) |
[production] |
| 13:44 |
<otto@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-page-content-change-enrich: apply |
[production] |
| 13:44 |
<otto@deploy1002> |
helmfile [codfw] START helmfile.d/services/mw-page-content-change-enrich: apply |
[production] |
| 13:42 |
<otto@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-page-content-change-enrich: apply |
[production] |
| 13:41 |
<otto@deploy1002> |
helmfile [codfw] START helmfile.d/services/mw-page-content-change-enrich: apply |
[production] |
| 13:41 |
<urbanecm@deploy1002> |
Finished scap: Backport for [[gerrit:923645|Enable wgMinervaEnableSiteNotice for bnwikiquote (T337683)]] (duration: 10m 01s) |
[production] |
| 13:39 |
<ottomata> |
destroy mw-page-content-change-enrich deployment in dse-k8s-eqiad in order to deploy in wikikube - T330507 |
[production] |
| 13:35 |
<godog> |
rm cadvisor.service symlink/alias and restart kubelet on affected hosts - T337836 |
[production] |
| 13:33 |
<urbanecm@deploy1002> |
mdsshakil and urbanecm: Backport for [[gerrit:923645|Enable wgMinervaEnableSiteNotice for bnwikiquote (T337683)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet |
[production] |