1-50 of 10000 results (113ms)
2026-07-29 ยง
12:08 <cwilliams@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1196.eqiad.wmnet with reason: Maintenance [production]
12:07 <root@cumin1003> END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1195: Maintenance [production]
12:04 <cwilliams@cumin1003> dbctl commit (dc=all): 'Depooling db1222 (T431660)', diff saved to https://phabricator.wikimedia.org/P95530 and previous config saved to /var/cache/conftool/dbconfig/20260729-120424-cwilliams.json [production]
12:04 <cwilliams@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1222.eqiad.wmnet with reason: Maintenance [production]
12:00 <elukey@cumin1003> END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1098 hosts [production]
12:00 <marostegui> Rename tables T425074 [production]
12:00 <btullis@cumin1003> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: an-test-coord1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1003" [production]
11:58 <root@cumin1003> END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1197: Maintenance [production]
11:55 <btullis@cumin1003> START - Cookbook sre.dns.netbox [production]
11:52 <elukey@deploy1003> helmfile [codfw] DONE helmfile.d/services/proton: sync [production]
11:51 <btullis@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply [production]
11:51 <elukey@deploy1003> helmfile [codfw] START helmfile.d/services/proton: sync [production]
11:51 <btullis@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply [production]
11:50 <btullis@cumin1003> START - Cookbook sre.hosts.decommission for hosts an-test-coord1001.eqiad.wmnet [production]
11:50 <elukey@deploy1003> helmfile [staging] DONE helmfile.d/services/proton: sync [production]
11:49 <elukey@deploy1003> helmfile [staging] START helmfile.d/services/proton: sync [production]
11:35 <root@cumin1003> START - Cookbook sre.mysql.pool pool db1241: Maintenance [production]
11:33 <root@cumin1003> START - Cookbook sre.mysql.pool pool db1212: Maintenance [production]
11:30 <brouberol@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply [production]
11:29 <brouberol@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply [production]
11:29 <cwilliams@cumin1003> dbctl commit (dc=all): 'Depooling db1241 (T431660)', diff saved to https://phabricator.wikimedia.org/P95520 and previous config saved to /var/cache/conftool/dbconfig/20260729-112918-cwilliams.json [production]
11:29 <cwilliams@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1241.eqiad.wmnet with reason: Maintenance [production]
11:29 <root@cumin1003> END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1238: Maintenance [production]
11:28 <brouberol@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply [production]
11:27 <brouberol@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply [production]
11:27 <cwilliams@cumin1003> dbctl commit (dc=all): 'Depooling db1212 (T431660)', diff saved to https://phabricator.wikimedia.org/P95517 and previous config saved to /var/cache/conftool/dbconfig/20260729-112727-cwilliams.json [production]
11:27 <cwilliams@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: Maintenance [production]
11:26 <cwilliams@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1212.eqiad.wmnet with reason: Maintenance [production]
11:26 <root@cumin1003> END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1198: Maintenance [production]
11:23 <bwojtowicz@deploy1003> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [production]
11:22 <bwojtowicz@deploy1003> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . [production]
11:21 <root@cumin1003> START - Cookbook sre.mysql.pool pool db1195: Maintenance [production]
11:14 <cwilliams@cumin1003> dbctl commit (dc=all): 'Depooling db1195 (T431660)', diff saved to https://phabricator.wikimedia.org/P95514 and previous config saved to /var/cache/conftool/dbconfig/20260729-111450-cwilliams.json [production]
11:14 <cwilliams@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1195.eqiad.wmnet with reason: Maintenance [production]
11:14 <root@cumin1003> END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1186: Maintenance [production]
11:13 <btullis@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply [production]
11:12 <btullis@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [production]
11:05 <ozge@deploy1003> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [production]
10:54 <marostegui> Dropping renamed tables T425066 [production]
10:41 <root@cumin1003> START - Cookbook sre.mysql.pool pool db1238: Maintenance [production]
10:39 <root@cumin1003> START - Cookbook sre.mysql.pool pool db1198: Maintenance [production]
10:39 <Amir1> ran https://phabricator.wikimedia.org/T432509#12149723 in production (T432509) [production]
10:38 <root@cumin1003> START - Cookbook sre.mysql.pool pool db1197: Maintenance [production]
10:35 <cwilliams@cumin1003> dbctl commit (dc=all): 'Depooling db1238 (T431660)', diff saved to https://phabricator.wikimedia.org/P95501 and previous config saved to /var/cache/conftool/dbconfig/20260729-103532-cwilliams.json [production]
10:35 <cwilliams@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1238.eqiad.wmnet with reason: Maintenance [production]
10:35 <root@cumin1003> END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1221: Maintenance [production]
10:33 <cwilliams@cumin1003> dbctl commit (dc=all): 'Depooling db1198 (T431660)', diff saved to https://phabricator.wikimedia.org/P95499 and previous config saved to /var/cache/conftool/dbconfig/20260729-103330-cwilliams.json [production]
10:33 <cwilliams@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1198.eqiad.wmnet with reason: Maintenance [production]
10:33 <root@cumin1003> END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1175: Maintenance [production]
10:32 <cwilliams@cumin1003> dbctl commit (dc=all): 'Depooling db1197 (T431660)', diff saved to https://phabricator.wikimedia.org/P95496 and previous config saved to /var/cache/conftool/dbconfig/20260729-103217-cwilliams.json [production]