51-100 of 10000 results (116ms)
2026-07-29 §
08:45 <mvernon@cumin2003> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2072.codfw.wmnet with OS trixie [production]
08:45 <cwilliams@cumin1003> dbctl commit (dc=all): 'Depooling db1199 (T431660)', diff saved to https://phabricator.wikimedia.org/P95464 and previous config saved to /var/cache/conftool/dbconfig/20260729-084534-cwilliams.json [production]
08:45 <ayounsi@cumin1003> END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool magru [reason: router upgrade, T431750] [production]
08:45 <cwilliams@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1199.eqiad.wmnet with reason: Maintenance [production]
08:45 <ayounsi@cumin1003> START - Cookbook sre.dns.admin DNS admin: depool magru [reason: router upgrade, T431750] [production]
08:45 <root@cumin1003> END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1190: Maintenance [production]
08:44 <cwilliams@cumin1003> dbctl commit (dc=all): 'Depooling db1182 (T431660)', diff saved to https://phabricator.wikimedia.org/P95462 and previous config saved to /var/cache/conftool/dbconfig/20260729-084436-cwilliams.json [production]
08:44 <cwilliams@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1182.eqiad.wmnet with reason: Maintenance [production]
08:44 <root@cumin1003> END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1156: Maintenance [production]
08:44 <cwilliams@cumin1003> dbctl commit (dc=all): 'Depooling db1166 (T431660)', diff saved to https://phabricator.wikimedia.org/P95460 and previous config saved to /var/cache/conftool/dbconfig/20260729-084400-cwilliams.json [production]
08:43 <cwilliams@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1166.eqiad.wmnet with reason: Maintenance [production]
08:43 <root@cumin1003> END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1157: Maintenance [production]
08:41 <cwilliams@cumin1003> dbctl commit (dc=all): 'Depooling db1169 (T431660)', diff saved to https://phabricator.wikimedia.org/P95458 and previous config saved to /var/cache/conftool/dbconfig/20260729-084147-cwilliams.json [production]
08:41 <cwilliams@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1169.eqiad.wmnet with reason: Maintenance [production]
08:41 <root@cumin1003> END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1163: Maintenance [production]
08:30 <btullis@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 11 hosts with reason: Replacing the namenodes [production]
08:23 <mvernon@cumin2003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2072.codfw.wmnet with reason: host reimage [production]
08:20 <elukey@cumin1003> START - Cookbook sre.hosts.bmc-user-mgmt for 1098 hosts [production]
08:19 <mvernon@cumin2003> START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2072.codfw.wmnet with reason: host reimage [production]
07:58 <mvernon@cumin2003> START - Cookbook sre.hosts.reimage for host ms-be2072.codfw.wmnet with OS trixie [production]
07:57 <root@cumin1003> START - Cookbook sre.mysql.pool pool db1190: Maintenance [production]
07:55 <root@cumin1003> START - Cookbook sre.mysql.pool pool db1156: Maintenance [production]
07:55 <root@cumin1003> START - Cookbook sre.mysql.pool pool db1157: Maintenance [production]
07:53 <root@cumin1003> START - Cookbook sre.mysql.pool pool db1163: Maintenance [production]
07:49 <cwilliams@cumin1003> dbctl commit (dc=all): 'Depooling db1190 (T431660)', diff saved to https://phabricator.wikimedia.org/P95444 and previous config saved to /var/cache/conftool/dbconfig/20260729-074930-cwilliams.json [production]
07:49 <cwilliams@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1190.eqiad.wmnet with reason: Maintenance [production]
07:49 <cwilliams@cumin1003> dbctl commit (dc=all): 'Depooling db1157 (T431660)', diff saved to https://phabricator.wikimedia.org/P95443 and previous config saved to /var/cache/conftool/dbconfig/20260729-074914-cwilliams.json [production]
07:49 <cwilliams@cumin1003> dbctl commit (dc=all): 'Depooling db1156 (T431660)', diff saved to https://phabricator.wikimedia.org/P95442 and previous config saved to /var/cache/conftool/dbconfig/20260729-074906-cwilliams.json [production]
07:49 <cwilliams@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance [production]
07:48 <cwilliams@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: Maintenance [production]
07:48 <cwilliams@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1156.eqiad.wmnet with reason: Maintenance [production]
07:46 <cwilliams@cumin1003> dbctl commit (dc=all): 'Depooling db1163 (T431660)', diff saved to https://phabricator.wikimedia.org/P95441 and previous config saved to /var/cache/conftool/dbconfig/20260729-074652-cwilliams.json [production]
07:46 <cwilliams@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1163.eqiad.wmnet with reason: Maintenance [production]
07:46 <ayounsi@cumin1003> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2034.codfw.wmnet [production]
07:42 <ayounsi@cumin1003> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2034.codfw.wmnet [production]
07:42 <marostegui@cumin1003> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2232.codfw.wmnet with OS trixie [production]
07:34 <trueg@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply [production]
07:34 <trueg@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply [production]
07:33 <trueg@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply [production]
07:31 <trueg@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply [production]
07:19 <marostegui@cumin1003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2232.codfw.wmnet with reason: host reimage [production]
07:15 <marostegui@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on db2232.codfw.wmnet with reason: host reimage [production]
06:58 <marostegui@cumin1003> START - Cookbook sre.hosts.reimage for host db2232.codfw.wmnet with OS trixie [production]
06:58 <marostegui@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db[2160,2232].codfw.wmnet with reason: Reimage [production]
06:26 <marostegui@cumin1003> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1164.eqiad.wmnet with OS trixie [production]
06:05 <marostegui@cumin1003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1164.eqiad.wmnet with reason: host reimage [production]
06:01 <marostegui@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on db1164.eqiad.wmnet with reason: host reimage [production]
05:47 <marostegui@cumin1003> START - Cookbook sre.hosts.reimage for host db1164.eqiad.wmnet with OS trixie [production]
05:46 <marostegui@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1164.eqiad.wmnet with reason: Reimage [production]
2026-07-28 §
22:50 <jiji@cumin1003> END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1138.eqiad.wmnet [production]