|
2024-03-27
ยง
|
| 15:51 |
<bking@cumin2002> |
START - Cookbook sre.hosts.downtime for 12 days, 0:00:00 on elastic2038.codfw.wmnet with reason: T358882 |
[production] |
| 15:51 |
<arnaudb@cumin1002> |
END (ERROR) - Cookbook sre.mysql.clone (exit_code=97) Will create a clone of db2115.codfw.wmnet onto db2215.codfw.wmnet |
[production] |
| 15:51 |
<jayme@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/changeprop: apply |
[production] |
| 15:51 |
<claime> |
50% of backend RESTbase traffic to mw-api-int - T358213 |
[production] |
| 15:50 |
<jayme@deploy1002> |
helmfile [eqiad] START helmfile.d/services/changeprop: apply |
[production] |
| 15:50 |
<jayme@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/changeprop: apply |
[production] |
| 15:50 |
<jayme@deploy1002> |
helmfile [eqiad] START helmfile.d/services/changeprop: apply |
[production] |
| 15:43 |
<jayme@deploy1002> |
helmfile [staging] DONE helmfile.d/services/changeprop: apply |
[production] |
| 15:43 |
<jayme@deploy1002> |
helmfile [staging] START helmfile.d/services/changeprop: apply |
[production] |
| 15:35 |
<jnuche@deploy1002> |
Finished deploy [releng/jenkins-deploy@1a343bf] (releasing): deploying fix for T361084 to all targets (duration: 01m 03s) |
[production] |
| 15:33 |
<jnuche@deploy1002> |
Started deploy [releng/jenkins-deploy@1a343bf] (releasing): deploying fix for T361084 to all targets |
[production] |
| 15:33 |
<jnuche@deploy1002> |
Finished deploy [releng/jenkins-deploy@1a343bf] (releasing): deploying fix for T361084 to all targets (duration: 00m 19s) |
[production] |
| 15:33 |
<jnuche@deploy1002> |
Started deploy [releng/jenkins-deploy@1a343bf] (releasing): deploying fix for T361084 to all targets |
[production] |
| 15:23 |
<brouberol@cumin2002> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts an-tool1009.eqiad.wmnet |
[production] |
| 15:23 |
<brouberol@cumin2002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 15:23 |
<brouberol@cumin2002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: an-tool1009.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - brouberol@cumin2002" |
[production] |
| 15:23 |
<logmsgbot> |
andrewtavis-wmde@deploy1002 Finished deploy [airflow-dags/wmde@36dee63]: (no justification provided) (duration: 00m 08s) |
[production] |
| 15:23 |
<logmsgbot> |
andrewtavis-wmde@deploy1002 Started deploy [airflow-dags/wmde@36dee63]: (no justification provided) |
[production] |
| 15:21 |
<brouberol@cumin2002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: an-tool1009.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - brouberol@cumin2002" |
[production] |
| 15:19 |
<brouberol@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 15:17 |
<claime> |
enabling and running puppet on P:restbase - T358213 |
[production] |
| 15:14 |
<brouberol@cumin2002> |
START - Cookbook sre.hosts.decommission for hosts an-tool1009.eqiad.wmnet |
[production] |
| 15:14 |
<claime> |
enabling and running puppet on restbase1035.eqiad.wmnet - T358213 |
[production] |
| 15:12 |
<jnuche@deploy1002> |
Finished deploy [releng/jenkins-deploy@1a343bf] (releasing): testing fix for T361084 (duration: 00m 20s) |
[production] |
| 15:12 |
<jnuche@deploy1002> |
Started deploy [releng/jenkins-deploy@1a343bf] (releasing): testing fix for T361084 |
[production] |
| 15:11 |
<claime> |
enabling and running puppet on restbase2021.codfw.wmnet - T358213 |
[production] |
| 15:08 |
<claime> |
Disabling puppet on P:restbase - T358213 |
[production] |
| 14:54 |
<cgoubert@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/proton: apply |
[production] |
| 14:52 |
<cgoubert@deploy1002> |
helmfile [codfw] START helmfile.d/services/proton: apply |
[production] |
| 14:45 |
<effie> |
Day 8: Pool codfw for user traffic - T357547 |
[production] |
| 14:42 |
<jiji@cumin1002> |
END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) status all services in all: None - None |
[production] |
| 14:42 |
<jiji@cumin1002> |
START - Cookbook sre.discovery.datacenter status all services in all: None - None |
[production] |
| 14:40 |
<jiji@cumin1002> |
END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) pool all active/active services in codfw: Pool active/active services on codfw - T357547 |
[production] |
| 14:21 |
<jiji@cumin1002> |
START - Cookbook sre.discovery.datacenter pool all active/active services in codfw: Pool active/active services on codfw - T357547 |
[production] |
| 14:19 |
<effie> |
Day 8: Pool active/active services on codfw - T357547 |
[production] |
| 14:17 |
<Dreamy_Jazz> |
Afternoon UTC backport window done (extended by 17 mins) |
[production] |
| 14:17 |
<dreamyjazz@deploy1002> |
Finished scap: Backport for [[gerrit:1014559|Prevent new user names matching the temporary account pattern (T361021 T349506)]] (duration: 14m 28s) |
[production] |
| 14:09 |
<jiji@cumin1002> |
END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) status all services in all: None - None |
[production] |
| 14:09 |
<jiji@cumin1002> |
START - Cookbook sre.discovery.datacenter status all services in all: None - None |
[production] |
| 14:06 |
<dreamyjazz@deploy1002> |
dreamyjazz and tchanders: Continuing with sync |
[production] |
| 14:05 |
<dreamyjazz@deploy1002> |
dreamyjazz and tchanders: Backport for [[gerrit:1014559|Prevent new user names matching the temporary account pattern (T361021 T349506)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 14:03 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2216 (re)pooling @ 100%: Post clone (dst)', diff saved to https://phabricator.wikimedia.org/P58957 and previous config saved to /var/cache/conftool/dbconfig/20240327-140334-arnaudb.json |
[production] |
| 14:02 |
<dreamyjazz@deploy1002> |
Started scap: Backport for [[gerrit:1014559|Prevent new user names matching the temporary account pattern (T361021 T349506)]] |
[production] |
| 13:59 |
<dreamyjazz@deploy1002> |
Finished scap: Backport for [[gerrit:1013101|Removing MachineVision events, extension is being sunsetted (T347970)]], [[gerrit:1013284|Sunsetting MachineVision extension, so remove config (T352884)]] (duration: 50m 49s) |
[production] |
| 13:59 |
<godog> |
bounce prometheus@k8s-aux in eqiad - T343529 |
[production] |
| 13:57 |
<btullis@cumin1002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1096.eqiad.wmnet with OS bullseye |
[production] |
| 13:48 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2216 (re)pooling @ 75%: Post clone (dst)', diff saved to https://phabricator.wikimedia.org/P58956 and previous config saved to /var/cache/conftool/dbconfig/20240327-134828-arnaudb.json |
[production] |
| 13:41 |
<dreamyjazz@deploy1002> |
dreamyjazz and cparle: Continuing with sync |
[production] |
| 13:41 |
<dreamyjazz@deploy1002> |
dreamyjazz and cparle: Backport for [[gerrit:1013101|Removing MachineVision events, extension is being sunsetted (T347970)]], [[gerrit:1013284|Sunsetting MachineVision extension, so remove config (T352884)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 13:33 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2216 (re)pooling @ 50%: Post clone (dst)', diff saved to https://phabricator.wikimedia.org/P58955 and previous config saved to /var/cache/conftool/dbconfig/20240327-133322-arnaudb.json |
[production] |