|
2026-02-17
§
|
| 14:58 |
<sukhe> |
running authdns-update after magru depool |
[production] |
| 13:08 |
<cgoubert@cumin1003> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1030.eqiad.wmnet |
[production] |
| 13:08 |
<cgoubert@cumin1003> |
START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1030.eqiad.wmnet |
[production] |
| 13:05 |
<cgoubert@cumin1003> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1030.eqiad.wmnet |
[production] |
| 13:04 |
<cgoubert@cumin1003> |
START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1030.eqiad.wmnet |
[production] |
| 13:00 |
<ayounsi@cumin1003> |
END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool site magru [reason: no reason specified, T416442] |
[production] |
| 13:00 |
<ayounsi@cumin1003> |
START - Cookbook sre.dns.admin DNS admin: depool site magru [reason: no reason specified, T416442] |
[production] |
| 09:44 |
<jayme@cumin1003> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2001.codfw.wmnet |
[production] |
| 09:44 |
<jayme@cumin1003> |
START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2001.codfw.wmnet |
[production] |
| 08:30 |
<jayme@cumin1003> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet |
[production] |
| 08:30 |
<jayme@cumin1003> |
START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet |
[production] |
|
2026-02-10
§
|
| 23:04 |
<ryankemper> |
`[opensearch-ipoid]` T416345 Restarted (~16 mins ago) `opensearch-ipoid-masters-1` to force it to schedule to a k8s node with 10G networking, hopefully this helps w/ latency |
[production] |
| 18:37 |
<sukhe@cumin1003> |
END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool site magru [reason: no reason specified, ] |
[production] |
| 18:36 |
<sukhe@cumin1003> |
START - Cookbook sre.dns.admin DNS admin: depool site magru [reason: no reason specified, ] |
[production] |
| 18:12 |
<sukhe@cumin1003> |
END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool site magru [reason: no reason specified, ] |
[production] |
| 18:12 |
<sukhe@cumin1003> |
START - Cookbook sre.dns.admin DNS admin: depool site magru [reason: no reason specified, ] |
[production] |
| 17:26 |
<jasmine@cumin2002> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2116-2123,2216-2241].codfw.wmnet |
[production] |
| 17:25 |
<jasmine@cumin2002> |
START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2116-2123,2216-2241].codfw.wmnet |
[production] |
| 14:02 |
<ayounsi@cumin1003> |
END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool site drmrs [reason: no reason specified, ] |
[production] |
| 14:02 |
<ayounsi@cumin1003> |
START - Cookbook sre.dns.admin DNS admin: depool site drmrs [reason: no reason specified, ] |
[production] |
| 07:01 |
<marostegui@cumin1003> |
END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1169: Upgrade mariadb |
[production] |
| 07:01 |
<marostegui@cumin1003> |
START - Cookbook sre.mysql.depool depool db1169: Upgrade mariadb |
[production] |
| 06:40 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depool db2218 T416555', diff saved to https://phabricator.wikimedia.org/P88742 and previous config saved to /var/cache/conftool/dbconfig/20260210-064045-marostegui.json |
[production] |
|
2026-02-09
§
|
| 17:32 |
<jasmine@cumin2002> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2052-2054,2063,2079-2084,2096-2101].codfw.wmnet |
[production] |
| 17:31 |
<jasmine@cumin2002> |
START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2052-2054,2063,2079-2084,2096-2101].codfw.wmnet |
[production] |
| 11:29 |
<jayme@cumin1003> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2003.codfw.wmnet |
[production] |
| 11:29 |
<jayme@cumin1003> |
START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2003.codfw.wmnet |
[production] |
| 11:23 |
<jayme@cumin1003> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2001.codfw.wmnet |
[production] |
| 11:23 |
<jayme@cumin1003> |
START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2001.codfw.wmnet |
[production] |
| 11:13 |
<jayme@cumin1003> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet |
[production] |
| 11:10 |
<jayme@cumin1003> |
START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet |
[production] |
| 11:10 |
<jayme@cumin1003> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2003.codfw.wmnet |
[production] |
| 11:10 |
<jayme@cumin1003> |
START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2003.codfw.wmnet |
[production] |
| 10:49 |
<marostegui@cumin1003> |
END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1033: Will be depooled |
[production] |
| 10:49 |
<marostegui@cumin1003> |
START - Cookbook sre.mysql.depool depool es1033: Will be depooled |
[production] |
| 09:41 |
<jayme> |
kubectl delete node wikikube-worker2019.codfw.wmnet - T409102 |
[production] |
| 06:19 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depool db2203 T416554', diff saved to https://phabricator.wikimedia.org/P88725 and previous config saved to /var/cache/conftool/dbconfig/20260209-061904-marostegui.json |
[production] |