1-50 of 10000 results (16ms)
2026-02-14 §
06:30 <wmbot~chlod@tools-bastion-15> bump to 2 replicas [tools.copyvios]
03:37 <brett> Import lua5.4-maxminddb 0.1.1~deb13u1 into trixie-wikimedia (T401832) [production]
02:14 <mwpresync@deploy2002> Finished scap build-images: Publishing wmf/next image (duration: 13m 02s) [production]
02:01 <mwpresync@deploy2002> Started scap build-images: Publishing wmf/next image [production]
01:12 <mutante> cumin1003 - race condition between puppet and systemd - puppet fails because userdel fails. userdel fails because there is still a pid used by the user. that process is /lib/systemdl/system --user - ran "loginctl terminate-user"; killed tmux PID; ran puppet again T417465 [production]
00:01 <mutante> codesearch9: systemctl restart logrotate with new config without * - T413739 [codesearch]
2026-02-13 §
23:25 <brett> Import prometheus-varnishkafka-exporter 0.1~deb13u1 into trixie-wikimedia (T401832) [production]
23:01 <brett> Import varnishkafka 1.2.0~deb13+wmf1 into trixie-wikimedia (T401832) [production]
22:48 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Depooling db1214 (T410589)', diff saved to https://phabricator.wikimedia.org/P88814 and previous config saved to /var/cache/conftool/dbconfig/20260213-224820-ladsgroup.json [production]
22:48 <ladsgroup@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1214.eqiad.wmnet with reason: Maintenance [production]
22:48 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1203 (T410589)', diff saved to https://phabricator.wikimedia.org/P88813 and previous config saved to /var/cache/conftool/dbconfig/20260213-224806-ladsgroup.json [production]
22:37 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1203', diff saved to https://phabricator.wikimedia.org/P88812 and previous config saved to /var/cache/conftool/dbconfig/20260213-223758-ladsgroup.json [production]
22:30 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mwlog2003.codfw.wmnet with OS bookworm [production]
22:30 <pt1979@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002" [production]
22:29 <sukhe@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on cp2043.codfw.wmnet with reason: host not provisioned yet [production]
22:28 <sukhe> disable puppet on cp2043: test host, wmfuniq service broken (needs upgrade for trixie) [production]
22:27 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1203', diff saved to https://phabricator.wikimedia.org/P88811 and previous config saved to /var/cache/conftool/dbconfig/20260213-222749-ladsgroup.json [production]
22:25 <pt1979@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002" [production]
22:17 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1203 (T410589)', diff saved to https://phabricator.wikimedia.org/P88810 and previous config saved to /var/cache/conftool/dbconfig/20260213-221741-ladsgroup.json [production]
22:03 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mwlog2003.codfw.wmnet with reason: host reimage [production]
22:01 <pt1979@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on mwlog2003.codfw.wmnet with reason: host reimage [production]
21:58 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host backup2015.codfw.wmnet with OS trixie [production]
21:40 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host mwlog2003.codfw.wmnet with OS bookworm [production]
21:39 <pt1979@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host mwlog2003.codfw.wmnet with OS bookworm [production]
21:39 <Krinkle> Manually pruning codesearch9:/var/log/account/pacct, ref T417397 T413739 [codesearch]
21:29 <Krinkle> Reboot codesearch9 instance, ref T417397 [codesearch]
20:54 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host mwlog2003.codfw.wmnet with OS bookworm [production]
20:41 <jhancock@cumin2002> END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host cloudcephosd2008-dev [production]
20:41 <jhancock@cumin2002> START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd2008-dev [production]
20:41 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd2008-dev.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
20:41 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host cloudcephosd2008-dev.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
20:40 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcephosd2008-dev.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
20:40 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host cloudcephosd2008-dev.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
20:38 <jhancock@cumin2002> END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host cloudcephosd2008-dev [production]
20:38 <jhancock@cumin2002> START - Cookbook sre.network.configure-switch-interfaces for host cloudcephosd2008-dev [production]
20:38 <jhancock@cumin2002> START - Cookbook sre.hosts.reimage for host backup2015.codfw.wmnet with OS trixie [production]
20:38 <jhancock@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
20:38 <jhancock@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding cloudcephosd2008-dev to codfw - jhancock@cumin2002" [production]
20:38 <jhancock@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding cloudcephosd2008-dev to codfw - jhancock@cumin2002" [production]
20:38 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['backup2015'] [production]
20:37 <jhancock@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['backup2015'] [production]
20:36 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host backup2015.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
20:34 <jhancock@cumin2002> START - Cookbook sre.dns.netbox [production]
20:30 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host backup2015.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
19:55 <brett@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 365 days, 0:00:00 on cp[2043-2044].codfw.wmnet with reason: These are test instances, not prod yet [production]
19:30 <trueg@deploy2002> helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply [production]
19:30 <trueg@deploy2002> helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply [production]
17:26 <wmftkbot> Test Kitchen mw-user experiment (poll 129062) - adds: none; removes: none; fields: growthexperiments-revise-tone - xLab/MPIC/TK tips at https://w.wiki/FwuD [analytics]
17:01 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mwlog2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
16:52 <vgutierrez> upload lua5.4-maxminddb_0.1.1-1 to bullseye-wikimedia (apt.wm.o) - T417291 [production]