551-600 of 10000 results (17ms)
2025-07-01 ยง
20:36 <cjming@deploy1003> zhaofjx, cjming: Continuing with sync [production]
20:33 <cjming@deploy1003> zhaofjx, cjming: Backport for [[gerrit:1163483|zhwiki: Permissions change for abusefilter groups (T397788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
20:31 <cjming@deploy1003> Started scap sync-world: Backport for [[gerrit:1163483|zhwiki: Permissions change for abusefilter groups (T397788)]] [production]
20:26 <jhathaway@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [production]
20:26 <jhathaway@cumin2002> START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [production]
20:24 <eevans@cumin1003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage [production]
20:20 <eevans@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage [production]
20:20 <jhathaway@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [production]
20:20 <jhathaway@cumin2002> START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [production]
20:04 <eevans@cumin1003> START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye [production]
20:04 <eevans@cumin1003> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bullseye [production]
20:03 <jhathaway@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: T383173 [production]
20:02 <ejegg> disabled queue consumers for segment updates [production]
19:50 <jhathaway@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [production]
19:50 <jhathaway@cumin2002> START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [production]
19:47 <eevans@cumin1003> START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bullseye [production]
19:43 <andrew@cumin1003> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm [production]
19:42 <jhathaway@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [production]
19:37 <jhathaway@cumin2002> START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [production]
19:26 <kemayo@deploy1003> Finished scap sync-world: Backport for [[gerrit:1165589|Edit check: fix counter logging for SLO (T395444)]] (duration: 09m 07s) [production]
19:23 <andrew@cumin1003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage [production]
19:20 <kemayo@deploy1003> kemayo: Continuing with sync [production]
19:20 <andrew@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: host reimage [production]
19:19 <kemayo@deploy1003> kemayo: Backport for [[gerrit:1165589|Edit check: fix counter logging for SLO (T395444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
19:16 <kemayo@deploy1003> Started scap sync-world: Backport for [[gerrit:1165589|Edit check: fix counter logging for SLO (T395444)]] [production]
19:00 <andrew@cumin1003> START - Cookbook sre.hosts.reimage for host cloudcephmon2004-dev.codfw.wmnet with OS bookworm [production]
19:00 <jhathaway@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: T383173 [production]
18:55 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.ceph.upgrade_osds (exit_code=0) [admin]
18:30 <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.upgrade_osds [admin]
17:02 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet [production]
16:56 <andrew@cumin1003> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd2003-dev.codfw.wmnet [production]
16:56 <andrew@cumin1003> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
16:56 <andrew@cumin1003> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003" [production]
16:55 <andrew@cumin1003> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudcephosd2003-dev.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1003" [production]
16:51 <andrew@cumin1003> START - Cookbook sre.dns.netbox [production]
16:45 <andrew@cumin1003> START - Cookbook sre.hosts.decommission for hosts cloudcephosd2003-dev.codfw.wmnet [production]
16:44 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repool pc3 T378715', diff saved to https://phabricator.wikimedia.org/P78734 and previous config saved to /var/cache/conftool/dbconfig/20250701-164405-ladsgroup.json [production]
16:38 <dcaro@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers [tools]
16:37 <jgiannelos@deploy1003> helmfile [staging] DONE helmfile.d/services/mobileapps: apply [production]
16:37 <jgiannelos@deploy1003> helmfile [staging] START helmfile.d/services/mobileapps: apply [production]
16:28 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) [admin]
16:23 <dcaro@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers [tools]
16:13 <swfrench@deploy1003> Finished scap sync-world: Backport for [[gerrit:1152295|Remove title-case overrides for PHP 8.1 migration (T394556)]] (duration: 09m 21s) [production]
16:11 <inflatador> bking@prometheus1005:~$ sudo run-puppet-agent T398341 [production]
16:10 <jhancock@cumin1003> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
16:10 <dcaro@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers [toolsbeta]
16:08 <jhancock@cumin1003> START - Cookbook sre.dns.netbox [production]
16:07 <swfrench@deploy1003> swfrench: Continuing with sync [production]
16:07 <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.depool_and_destroy [admin]
16:06 <jhancock@cumin1003> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2013 [production]