|
2024-09-15
§
|
| 19:25 |
<lucaswerkmeister> |
commented out OAUTH section in config.yaml, should use envvars instead |
[tools.quickcategories] |
| 19:25 |
<lucaswerkmeister> |
python3 -c 'import yaml; print(yaml.safe_dump(yaml.safe_load(open("config.yaml"))["OAUTH"]["consumer_secret"]))' | toolforge envvars create TOOL_OAUTH__CONSUMER_SECRET |
[tools.quickcategories] |
| 19:25 |
<lucaswerkmeister> |
python3 -c 'import yaml; print(yaml.safe_dump(yaml.safe_load(open("config.yaml"))["OAUTH"]["consumer_key"]))' | toolforge envvars create TOOL_OAUTH__CONSUMER_KEY |
[tools.quickcategories] |
| 14:12 |
<wmbot~lucaswerkmeister@tools-bastion-13> |
commented out SUMMARY_BATCH_LINK in config.yaml, TOOL_SUMMARY_BATCH_LINK envvar is now used instead |
[tools.quickcategories] |
| 14:11 |
<lucaswerkmeister> |
python3 -c 'import yaml; print(yaml.safe_dump(yaml.safe_load(open("config.yaml"))["SUMMARY_BATCH_LINK"]))' | toolforge envvars create TOOL_SUMMARY_BATCH_LINK |
[tools.quickcategories] |
| 12:44 |
<wmbot~lucaswerkmeister@tools-bastion-13> |
removed SUMMARY_SUFFIX from config.yaml, unused since da859e4b81 five years ago |
[tools.quickcategories] |
| 12:05 |
<mvernon@cumin2002> |
END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-codfw |
[production] |
| 12:03 |
<mvernon@cumin2002> |
START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-codfw |
[production] |
|
2024-09-14
§
|
| 18:33 |
<wmbot~lucaswerkmeister@tools-bastion-13> |
commented out SECRET_KEY in config.yaml, TOOL_SECRET_KEY envvar is now used instead |
[tools.quickcategories] |
| 18:31 |
<lucaswerkmeister> |
toolforge envvars create TOOL_SECRET_KEY "$(python3 -c 'import yaml; print(yaml.safe_load(open("config.yaml"))["SECRET_KEY"])')" |
[tools.quickcategories] |
| 18:26 |
<wmbot~lucaswerkmeister@tools-bastion-13> |
deployed c2dfddb8b1 (load config from TOOL_* envvars in addition to config.yaml; no envvars actually added yet) |
[tools.quickcategories] |
| 18:01 |
<wmbot~samtar@tools-bastion-13> |
webservice restart |
[tools.svgtranslate] |
| 17:49 |
<wmbot~lucaswerkmeister@tools-bastion-13> |
deployed feae0e7e66 (refactor config) |
[tools.quickcategories] |
| 13:44 |
<wmbot~lucaswerkmeister@tools-bastion-13> |
deployed 6d6aed57e0 (update dependencies); includes a restart of the background runner which probably “fixes” T374152 for now |
[tools.quickcategories] |
| 11:14 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P69122 and previous config saved to /var/cache/conftool/dbconfig/20240914-111434-ladsgroup.json |
[production] |
| 11:04 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P69121 and previous config saved to /var/cache/conftool/dbconfig/20240914-110416-ladsgroup.json |
[production] |
| 10:49 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P69120 and previous config saved to /var/cache/conftool/dbconfig/20240914-104909-ladsgroup.json |
[production] |
| 10:34 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1172 (T367856)', diff saved to https://phabricator.wikimedia.org/P69119 and previous config saved to /var/cache/conftool/dbconfig/20240914-103402-ladsgroup.json |
[production] |
|
2024-09-13
§
|
| 21:24 |
<dzahn@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:10:00 on vrts2001.codfw.wmnet with reason: nftables migration |
[production] |
| 21:24 |
<dzahn@cumin2002> |
START - Cookbook sre.hosts.downtime for 0:10:00 on vrts2001.codfw.wmnet with reason: nftables migration |
[production] |
| 21:14 |
<dwisehaupt@cumin1002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 21:14 |
<dwisehaupt@cumin1002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: decommissioning frban2001 - dwisehaupt@cumin1002" |
[production] |
| 21:14 |
<dwisehaupt@cumin1002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: decommissioning frban2001 - dwisehaupt@cumin1002" |
[production] |
| 21:11 |
<dwisehaupt@cumin1002> |
START - Cookbook sre.dns.netbox |
[production] |
| 19:52 |
<bking@cumin2002> |
END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for wdqs[2021-2024].codfw.wmnet |
[production] |
| 19:52 |
<bking@cumin2002> |
START - Cookbook sre.hosts.remove-downtime for wdqs[2021-2024].codfw.wmnet |
[production] |
| 19:52 |
<bking@cumin2002> |
END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for wdqs[1021-1024].eqiad.wmnet |
[production] |
| 19:52 |
<bking@cumin2002> |
START - Cookbook sre.hosts.remove-downtime for wdqs[1021-1024].eqiad.wmnet |
[production] |
| 17:38 |
<vgutierrez@cumin1002> |
END (PASS) - Cookbook sre.cdn.transfer-purged-positions (exit_code=0) rolling custom on P{cp[2028-2034,2038-2042].codfw.wmnet,cp[5017,5019-5020,5023,5027-5028,5030].eqsin.wmnet,cp[4038-4052].ulsfo.wmnet} and A:cp |
[production] |
| 16:59 |
<swfrench@cumin2002> |
END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker2123.codfw.wmnet |
[production] |
| 16:59 |
<swfrench@cumin2002> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2123.codfw.wmnet |
[production] |
| 16:59 |
<swfrench@cumin2002> |
START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2123.codfw.wmnet |
[production] |
| 16:57 |
<swfrench-wmf> |
running homer cr*codfw* commit 'T372878' |
[production] |
| 16:56 |
<bking@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on wdqs[1021-1024].eqiad.wmnet with reason: T373935 |
[production] |
| 16:56 |
<bking@cumin2002> |
START - Cookbook sre.hosts.downtime for 3:00:00 on wdqs[1021-1024].eqiad.wmnet with reason: T373935 |
[production] |
| 16:55 |
<bking@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on wdqs[2021-2024].codfw.wmnet with reason: T373791 |
[production] |
| 16:55 |
<bking@cumin2002> |
START - Cookbook sre.hosts.downtime for 3:00:00 on wdqs[2021-2024].codfw.wmnet with reason: T373791 |
[production] |
| 16:52 |
<swfrench@cumin2002> |
END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker2122.codfw.wmnet |
[production] |
| 16:52 |
<swfrench@cumin2002> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2122.codfw.wmnet |
[production] |
| 16:52 |
<swfrench@cumin2002> |
START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2122.codfw.wmnet |
[production] |
| 16:50 |
<swfrench@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2123.codfw.wmnet with OS bullseye |
[production] |
| 16:46 |
<swfrench@cumin2002> |
END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker2121.codfw.wmnet |
[production] |
| 16:46 |
<swfrench@cumin2002> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2121.codfw.wmnet |
[production] |
| 16:46 |
<swfrench@cumin2002> |
START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2121.codfw.wmnet |
[production] |
| 16:43 |
<swfrench@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2122.codfw.wmnet with OS bullseye |
[production] |
| 16:38 |
<swfrench-wmf> |
running homer lsw1-b3-codfw* commit 'T372878' |
[production] |
| 16:35 |
<swfrench@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2121.codfw.wmnet with OS bullseye |
[production] |
| 16:32 |
<wmbot~lucaswerkmeister@tools-bastion-13> |
webservice restart # kubectl get events has messages about failed liveness probes, “will be restarted”, but it doesn’t seem to help 😔 |
[tools.sal] |
| 16:31 |
<swfrench@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2123.codfw.wmnet with reason: host reimage |
[production] |
| 16:27 |
<swfrench@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2123.codfw.wmnet with reason: host reimage |
[production] |