1-50 of 10000 results (6ms)
2026-09-16 ยง
17:51 <vriley@cumin2003> START - Cookbook sre.hosts.provision for host ms-be1099.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
17:46 <ladsgroup@deploy1003> helmfile [staging] DONE helmfile.d/services/thumbor: apply [production]
17:46 <ladsgroup@deploy1003> helmfile [staging] START helmfile.d/services/thumbor: apply [production]
17:45 <vriley@cumin1003> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db1245.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
17:45 <vriley@cumin1003> START - Cookbook sre.hosts.provision for host db1245.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
17:44 <vriley@cumin1003> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db1245.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
17:44 <vriley@cumin1003> START - Cookbook sre.hosts.provision for host db1245.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
17:35 <jclark@cumin1003> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul1006.eqiad.wmnet with OS trixie [production]
17:35 <jclark@cumin1003> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003" [production]
17:30 <vriley@cumin1003> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1099.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
17:30 <vriley@cumin1003> START - Cookbook sre.hosts.provision for host ms-be1099.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
17:29 <jclark@cumin1003> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003" [production]
17:27 <vriley@cumin1003> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1099.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
17:27 <vriley@cumin1003> START - Cookbook sre.hosts.provision for host ms-be1099.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
17:23 <vriley@cumin1003> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1099.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
17:23 <vriley@cumin1003> START - Cookbook sre.hosts.provision for host ms-be1099.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
17:20 <ladsgroup@deploy1003> helmfile [staging] DONE helmfile.d/services/thumbor: apply [production]
17:20 <ladsgroup@deploy1003> helmfile [staging] START helmfile.d/services/thumbor: apply [production]
17:14 <jhancock@cumin2003> START - Cookbook sre.hosts.provision for host db1245.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
17:13 <jclark@cumin1003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1006.eqiad.wmnet with reason: host reimage [production]
17:10 <vriley@cumin1003> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db1245.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
17:10 <vriley@cumin1003> START - Cookbook sre.hosts.provision for host db1245.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
17:10 <vriley@cumin1003> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db1245.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
17:10 <vriley@cumin1003> START - Cookbook sre.hosts.provision for host db1245.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
17:09 <vriley@cumin1003> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
17:07 <jclark@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1006.eqiad.wmnet with reason: host reimage [production]
17:07 <vriley@cumin1003> START - Cookbook sre.dns.netbox [production]
17:05 <vriley@cumin1003> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db1245.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
17:05 <vriley@cumin1003> START - Cookbook sre.hosts.provision for host db1245.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
16:52 <jclark@cumin1003> START - Cookbook sre.hosts.reimage for host zuul1006.eqiad.wmnet with OS trixie [production]
16:46 <cdobbins@cumin1004> conftool action : set/pooled=yes; selector: name=ncredir7003.magru.wmnet [production]
16:44 <cdobbins@cumin1004> conftool action : set/pooled=yes; selector: name=ncredir7003 [production]
16:19 <vriley@cumin1003> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1006.eqiad.wmnet with OS trixie [production]
15:42 <blake@deploy1003> helmfile [codfw] DONE helmfile.d/services/mw-cron: apply [production]
15:42 <blake@deploy1003> helmfile [codfw] START helmfile.d/services/mw-cron: apply [production]
15:42 <blake@deploy1003> helmfile [eqiad] DONE helmfile.d/services/mw-cron: apply [production]
15:42 <cdobbins@cumin1004> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir7003.magru.wmnet with OS trixie [production]
15:41 <blake@deploy1003> helmfile [eqiad] START helmfile.d/services/mw-cron: apply [production]
15:41 <andrewbogott> systemctl restart mariadb@x3.service mariadb@s3.service on clouddb1023, trying to resolve memory pressure T438200 [admin]
15:36 <jnuche@deploy1003> Finished scap sync-world: Backport for [[gerrit:1342279|Use parser output value instead of status (T438154)]] (duration: 33m 21s) [production]
15:35 <blake@deploy1003> helmfile [codfw] DONE helmfile.d/services/mw-cron: apply [production]
15:35 <blake@deploy1003> helmfile [codfw] START helmfile.d/services/mw-cron: apply [production]
15:24 <jnuche@deploy1003> jnuche, jforrester: Continuing with deployment [production]
15:23 <jnuche@deploy1003> jnuche, jforrester: Backport for [[gerrit:1342279|Use parser output value instead of status (T438154)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
15:19 <vriley@cumin1003> START - Cookbook sre.hosts.reimage for host zuul1006.eqiad.wmnet with OS trixie [production]
15:18 <cdobbins@cumin1004> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir7003.magru.wmnet with reason: host reimage [production]
15:14 <cdobbins@cumin1004> START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir7003.magru.wmnet with reason: host reimage [production]
15:13 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary (exit_code=0) on eqiad1, with recreate False, for hosts list: ['cloudvirt1067'] [cloudvirt-canary]
15:13 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary on eqiad1, with recreate False, for hosts list: ['cloudvirt1067'] [cloudvirt-canary]
15:03 <jnuche@deploy1003> Started scap sync-world: Backport for [[gerrit:1342279|Use parser output value instead of status (T438154)]] [production]