1-50 of 10000 results (118ms)
2026-06-30 §
02:11 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2194 (T410589)', diff saved to https://phabricator.wikimedia.org/P94565 and previous config saved to /var/cache/conftool/dbconfig/20260630-021149-ladsgroup.json [production]
02:09 <jasmine@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1163.eqiad.wmnet with reason: host reimage [production]
02:07 <mwpresync@deploy1003> Finished scap build-images: Publishing wmf/next image (duration: 06m 45s) [production]
02:04 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Depooling db2194 (T410589)', diff saved to https://phabricator.wikimedia.org/P94564 and previous config saved to /var/cache/conftool/dbconfig/20260630-020416-ladsgroup.json [production]
02:04 <ladsgroup@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2194.codfw.wmnet with reason: Maintenance [production]
02:03 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2190 (T410589)', diff saved to https://phabricator.wikimedia.org/P94563 and previous config saved to /var/cache/conftool/dbconfig/20260630-020353-ladsgroup.json [production]
02:00 <mwpresync@deploy1003> Started scap build-images: Publishing wmf/next image [production]
01:54 <jasmine@cumin2002> END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1163 [production]
01:54 <jasmine@cumin2002> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1163 [production]
01:53 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2190', diff saved to https://phabricator.wikimedia.org/P94562 and previous config saved to /var/cache/conftool/dbconfig/20260630-015345-ladsgroup.json [production]
01:53 <jasmine@cumin2002> START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1163 [production]
01:53 <jasmine@cumin2002> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1163.eqiad.wmnet 55.48.64.10.in-addr.arpa 5.5.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [production]
01:52 <jasmine@cumin2002> START - Cookbook sre.dns.wipe-cache wikikube-worker1163.eqiad.wmnet 55.48.64.10.in-addr.arpa 5.5.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [production]
01:52 <jasmine@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
01:52 <jasmine@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1163 - jasmine@cumin2002" [production]
01:52 <jasmine@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1163 - jasmine@cumin2002" [production]
01:47 <jasmine@cumin2002> START - Cookbook sre.dns.netbox [production]
01:46 <jasmine@cumin2002> START - Cookbook sre.hosts.move-vlan for host wikikube-worker1163 [production]
01:45 <jasmine@cumin2002> START - Cookbook sre.hosts.reimage for host wikikube-worker1163.eqiad.wmnet with OS trixie [production]
01:45 <jasmine@cumin2002> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1163.eqiad.wmnet [production]
01:44 <jasmine@cumin2002> START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1163.eqiad.wmnet [production]
01:44 <jasmine@cumin2002> START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1163.eqiad.wmnet [production]
01:43 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2190', diff saved to https://phabricator.wikimedia.org/P94561 and previous config saved to /var/cache/conftool/dbconfig/20260630-014337-ladsgroup.json [production]
01:37 <ryankemper@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2063.codfw.wmnet with OS trixie [production]
01:33 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2190 (T410589)', diff saved to https://phabricator.wikimedia.org/P94560 and previous config saved to /var/cache/conftool/dbconfig/20260630-013329-ladsgroup.json [production]
01:26 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Depooling db2190 (T410589)', diff saved to https://phabricator.wikimedia.org/P94559 and previous config saved to /var/cache/conftool/dbconfig/20260630-012605-ladsgroup.json [production]
01:25 <ladsgroup@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2190.codfw.wmnet with reason: Maintenance [production]
01:25 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2177 (T410589)', diff saved to https://phabricator.wikimedia.org/P94558 and previous config saved to /var/cache/conftool/dbconfig/20260630-012540-ladsgroup.json [production]
01:18 <ryankemper@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2063.codfw.wmnet with reason: host reimage [production]
01:15 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P94557 and previous config saved to /var/cache/conftool/dbconfig/20260630-011533-ladsgroup.json [production]
01:12 <ryankemper@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2063.codfw.wmnet with reason: host reimage [production]
01:05 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P94556 and previous config saved to /var/cache/conftool/dbconfig/20260630-010525-ladsgroup.json [production]
00:55 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2177 (T410589)', diff saved to https://phabricator.wikimedia.org/P94555 and previous config saved to /var/cache/conftool/dbconfig/20260630-005517-ladsgroup.json [production]
00:53 <ryankemper@cumin2002> START - Cookbook sre.hosts.reimage for host cirrussearch2063.codfw.wmnet with OS trixie [production]
00:47 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Depooling db2177 (T410589)', diff saved to https://phabricator.wikimedia.org/P94554 and previous config saved to /var/cache/conftool/dbconfig/20260630-004721-ladsgroup.json [production]
00:47 <ladsgroup@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2177.codfw.wmnet with reason: Maintenance [production]
00:46 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2156 (T410589)', diff saved to https://phabricator.wikimedia.org/P94553 and previous config saved to /var/cache/conftool/dbconfig/20260630-004657-ladsgroup.json [production]
00:36 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P94552 and previous config saved to /var/cache/conftool/dbconfig/20260630-003650-ladsgroup.json [production]
00:26 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P94551 and previous config saved to /var/cache/conftool/dbconfig/20260630-002642-ladsgroup.json [production]
00:16 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2156 (T410589)', diff saved to https://phabricator.wikimedia.org/P94550 and previous config saved to /var/cache/conftool/dbconfig/20260630-001634-ladsgroup.json [production]
00:08 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Depooling db2156 (T410589)', diff saved to https://phabricator.wikimedia.org/P94549 and previous config saved to /var/cache/conftool/dbconfig/20260630-000844-ladsgroup.json [production]
00:08 <ladsgroup@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2156.codfw.wmnet with reason: Maintenance [production]
2026-06-29 §
23:38 <inflatador> bking@localhost raise the number of incoming shard recoveries from 4 to 7 on all search_codfw endpoints T429844 [production]
21:30 <bking@cumin2003> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cirrussearch2091.codfw.wmnet with OS trixie [production]
21:23 <bking@cumin2003> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cirrussearch2089.codfw.wmnet'] [production]
21:21 <maryum> Deployed security fix for T430548 [production]
21:17 <bking@cumin2003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2073.codfw.wmnet with reason: host reimage [production]
21:13 <bking@cumin2003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2091.codfw.wmnet with reason: host reimage [production]
21:12 <bking@cumin2003> START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2073.codfw.wmnet with reason: host reimage [production]
21:12 <bking@cumin2003> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cirrussearch2089.codfw.wmnet'] [production]