|
2023-06-05
ยง
|
| 13:44 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve1001.eqiad.wmnet with reason: Host under maintenance |
[production] |
| 13:44 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P48724 and previous config saved to /var/cache/conftool/dbconfig/20230605-134418-ladsgroup.json |
[production] |
| 13:44 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1002.eqiad.wmnet with reason: Host under maintenance |
[production] |
| 13:43 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1002.eqiad.wmnet with reason: Host under maintenance |
[production] |
| 13:43 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1143 (T335845)', diff saved to https://phabricator.wikimedia.org/P48723 and previous config saved to /var/cache/conftool/dbconfig/20230605-134313-ladsgroup.json |
[production] |
| 13:41 |
<otto@deploy1002> |
helmfile [staging] DONE helmfile.d/services/mw-page-content-change-enrich: apply |
[production] |
| 13:41 |
<otto@deploy1002> |
helmfile [staging] START helmfile.d/services/mw-page-content-change-enrich: apply |
[production] |
| 13:38 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P48722 and previous config saved to /var/cache/conftool/dbconfig/20230605-133805-ladsgroup.json |
[production] |
| 13:36 |
<sukhe@deploy1002> |
Locking from deployment [ALL REPOSITORIES]: LVS maintenance in codfw, blocking deploys T326767 |
[production] |
| 13:35 |
<sukhe@deploy1002> |
Unlocked for deployment [ALL REPOSITORIES]: LVS maintenance in codfw, blocking deploys T322937 (duration: 01m 06s) |
[production] |
| 13:35 |
<sukhe@deploy1002> |
Locking from deployment [ALL REPOSITORIES]: LVS maintenance in codfw, blocking deploys T322937 |
[production] |
| 13:35 |
<bblack@deploy1002> |
Unlocked for deployment [ALL REPOSITORIES]: temporary lock for LVS resarts in core DCs (duration: 05m 54s) |
[production] |
| 13:32 |
<bblack> |
lvs1* (eqiad) - restart pybal for T334703 IPs |
[production] |
| 13:29 |
<bblack> |
lvs2* (codfw) - restart pybal for T334703 IPs |
[production] |
| 13:29 |
<bblack@deploy1002> |
Locking from deployment [ALL REPOSITORIES]: temporary lock for LVS resarts in core DCs |
[production] |
| 13:29 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1168 (T336886)', diff saved to https://phabricator.wikimedia.org/P48721 and previous config saved to /var/cache/conftool/dbconfig/20230605-132911-ladsgroup.json |
[production] |
| 13:28 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P48720 and previous config saved to /var/cache/conftool/dbconfig/20230605-132807-ladsgroup.json |
[production] |
| 13:27 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1168 (T336886)', diff saved to https://phabricator.wikimedia.org/P48719 and previous config saved to /var/cache/conftool/dbconfig/20230605-132703-ladsgroup.json |
[production] |
| 13:26 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1168.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:26 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1168.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:26 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1165 (T336886)', diff saved to https://phabricator.wikimedia.org/P48718 and previous config saved to /var/cache/conftool/dbconfig/20230605-132642-ladsgroup.json |
[production] |
| 13:25 |
<hashar> |
Restarted Zuul due to stall ssh connection # T309376 |
[production] |
| 13:25 |
<bblack> |
lvs3* (esams) - restart pybal for T334703 IPs |
[production] |
| 13:23 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P48717 and previous config saved to /var/cache/conftool/dbconfig/20230605-132259-ladsgroup.json |
[production] |
| 13:19 |
<bblack> |
lvs5* (eqsin) - restart pybal for T334703 IPs |
[production] |
| 13:17 |
<Lucas_WMDE> |
UTC afternoon backport+config window done |
[production] |
| 13:15 |
<bblack> |
lvs6* (drmrs) - restart pybal for T334703 IPs |
[production] |
| 13:14 |
<lucaswerkmeister-wmde@deploy1002> |
Finished scap: Backport for [[gerrit:927127|Make outreachwiki a multilingual Wikidata client (T171140)]] (duration: 10m 06s) |
[production] |
| 13:13 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P48716 and previous config saved to /var/cache/conftool/dbconfig/20230605-131301-ladsgroup.json |
[production] |
| 13:11 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P48715 and previous config saved to /var/cache/conftool/dbconfig/20230605-131136-ladsgroup.json |
[production] |
| 13:09 |
<bblack> |
lvs4* (ulsfo) - restart pybal for T334703 IPs |
[production] |
| 13:07 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1166 (T336886)', diff saved to https://phabricator.wikimedia.org/P48714 and previous config saved to /var/cache/conftool/dbconfig/20230605-130753-ladsgroup.json |
[production] |
| 13:05 |
<lucaswerkmeister-wmde@deploy1002> |
lucaswerkmeister-wmde: Backport for [[gerrit:927127|Make outreachwiki a multilingual Wikidata client (T171140)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet |
[production] |
| 13:04 |
<lucaswerkmeister-wmde@deploy1002> |
Started scap: Backport for [[gerrit:927127|Make outreachwiki a multilingual Wikidata client (T171140)]] |
[production] |
| 13:02 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1166 (T336886)', diff saved to https://phabricator.wikimedia.org/P48713 and previous config saved to /var/cache/conftool/dbconfig/20230605-130228-ladsgroup.json |
[production] |
| 13:02 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1166.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:02 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1166.eqiad.wmnet with reason: Maintenance |
[production] |
| 12:57 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1143 (T335845)', diff saved to https://phabricator.wikimedia.org/P48712 and previous config saved to /var/cache/conftool/dbconfig/20230605-125754-ladsgroup.json |
[production] |
| 12:56 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-eqiad |
[production] |
| 12:56 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P48711 and previous config saved to /var/cache/conftool/dbconfig/20230605-125630-ladsgroup.json |
[production] |
| 12:52 |
<jmm@cumin2002> |
START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-eqiad |
[production] |
| 12:51 |
<Amir1> |
killed prioritizeFilesWithTemplate.php, stopping depool maint. |
[production] |
| 12:49 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-codfw |
[production] |
| 12:44 |
<jmm@cumin2002> |
START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-codfw |
[production] |
| 12:44 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1143 (T335845)', diff saved to https://phabricator.wikimedia.org/P48710 and previous config saved to /var/cache/conftool/dbconfig/20230605-124444-ladsgroup.json |
[production] |
| 12:44 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1143.eqiad.wmnet with reason: Maintenance |
[production] |
| 12:44 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1143.eqiad.wmnet with reason: Maintenance |
[production] |
| 12:43 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe |
[production] |
| 12:41 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1165 (T336886)', diff saved to https://phabricator.wikimedia.org/P48709 and previous config saved to /var/cache/conftool/dbconfig/20230605-124124-ladsgroup.json |
[production] |
| 12:39 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1165 (T336886)', diff saved to https://phabricator.wikimedia.org/P48708 and previous config saved to /var/cache/conftool/dbconfig/20230605-123915-ladsgroup.json |
[production] |