2401-2450 of 10000 results (107ms)
2023-10-19 ยง
14:51 <jclark@cumin1001> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcontrol1008-dev.mgmt.eqiad.wmnet with reboot policy FORCED [production]
14:50 <jclark@cumin1001> START - Cookbook sre.hosts.provision for host cloudcontrol1008-dev.mgmt.eqiad.wmnet with reboot policy FORCED [production]
14:44 <elukey> powercycle titan1001 [production]
14:39 <jclark@cumin1001> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcontrol1008-dev.mgmt.eqiad.wmnet with reboot policy FORCED [production]
14:38 <jclark@cumin1001> START - Cookbook sre.hosts.provision for host cloudcontrol1008-dev.mgmt.eqiad.wmnet with reboot policy FORCED [production]
14:35 <kevinbazira@deploy2002> helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . [production]
14:34 <jclark@cumin1001> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcontrol1008-dev.mgmt.eqiad.wmnet with reboot policy FORCED [production]
14:34 <jclark@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudnet1007-dev.mgmt.eqiad.wmnet with reboot policy FORCED [production]
14:32 <jclark@cumin1001> START - Cookbook sre.hosts.provision for host cloudnet1008-dev.mgmt.eqiad.wmnet with reboot policy FORCED [production]
14:31 <jclark@cumin1001> START - Cookbook sre.hosts.provision for host cloudcontrol1009-dev.mgmt.eqiad.wmnet with reboot policy FORCED [production]
14:31 <jclark@cumin1001> START - Cookbook sre.hosts.provision for host cloudcontrol1010-dev.mgmt.eqiad.wmnet with reboot policy FORCED [production]
14:31 <jclark@cumin1001> START - Cookbook sre.hosts.provision for host cloudcontrol1008-dev.mgmt.eqiad.wmnet with reboot policy FORCED [production]
14:29 <jclark@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
14:28 <jclark@cumin1001> START - Cookbook sre.dns.netbox [production]
14:21 <jclark@cumin1001> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcontrol1008-dev.mgmt.eqiad.wmnet with reboot policy FORCED [production]
14:17 <jclark@cumin1001> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcontrol1010-dev.mgmt.eqiad.wmnet with reboot policy FORCED [production]
14:17 <jclark@cumin1001> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcontrol1009-dev.mgmt.eqiad.wmnet with reboot policy FORCED [production]
14:17 <jclark@cumin1001> START - Cookbook sre.hosts.provision for host cloudcontrol1008-dev.mgmt.eqiad.wmnet with reboot policy FORCED [production]
14:16 <jclark@cumin1001> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcontrol1008-dev.mgmt.eqiad.wmnet with reboot policy FORCED [production]
14:14 <jclark@cumin1001> START - Cookbook sre.hosts.provision for host cloudcontrol1010-dev.mgmt.eqiad.wmnet with reboot policy FORCED [production]
14:14 <jclark@cumin1001> START - Cookbook sre.hosts.provision for host cloudcontrol1008-dev.mgmt.eqiad.wmnet with reboot policy FORCED [production]
14:12 <jclark@cumin1001> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcontrol1010-dev.mgmt.eqiad.wmnet with reboot policy FORCED [production]
14:12 <jclark@cumin1001> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcontrol1009-dev.mgmt.eqiad.wmnet with reboot policy FORCED [production]
14:09 <jclark@cumin1001> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudcontrol1008-dev.mgmt.eqiad.wmnet with reboot policy FORCED [production]
14:05 <jclark@cumin1001> START - Cookbook sre.hosts.provision for host cloudnet1007-dev.mgmt.eqiad.wmnet with reboot policy FORCED [production]
14:04 <jclark@cumin1001> START - Cookbook sre.hosts.provision for host cloudcontrol1010-dev.mgmt.eqiad.wmnet with reboot policy FORCED [production]
14:03 <jclark@cumin1001> START - Cookbook sre.hosts.provision for host cloudcontrol1009-dev.mgmt.eqiad.wmnet with reboot policy FORCED [production]
14:03 <jclark@cumin1001> START - Cookbook sre.hosts.provision for host cloudcontrol1008-dev.mgmt.eqiad.wmnet with reboot policy FORCED [production]
14:01 <jclark@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
14:01 <jclark@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt cloudcontrol100[8-10]-dev cloudnet100[7-8]-dev - jclark@cumin1001" [production]
14:00 <jclark@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt cloudcontrol100[8-10]-dev cloudnet100[7-8]-dev - jclark@cumin1001" [production]
13:58 <jclark@cumin1001> START - Cookbook sre.dns.netbox [production]
13:48 <wmde-fisch@deploy2002> Finished scap: Backport for [[gerrit:966610|Revert "Revert "Workaround to center search terms label"" (T252346)]] (duration: 07m 50s) [production]
13:43 <wmde-fisch@deploy2002> wmde-fisch: Continuing with sync [production]
13:42 <wmde-fisch@deploy2002> wmde-fisch: Backport for [[gerrit:966610|Revert "Revert "Workaround to center search terms label"" (T252346)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
13:41 <wmde-fisch@deploy2002> Started scap: Backport for [[gerrit:966610|Revert "Revert "Workaround to center search terms label"" (T252346)]] [production]
13:00 <volans@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
13:00 <volans@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: noop - volans@cumin1001" [production]
12:59 <volans@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: noop - volans@cumin1001" [production]
12:52 <volans@cumin1001> START - Cookbook sre.dns.netbox [production]
12:50 <volans@cumin2002> END (ERROR) - Cookbook sre.dns.netbox (exit_code=97) [production]
12:50 <volans@cumin1001> END (ERROR) - Cookbook sre.dns.netbox (exit_code=97) [production]
12:50 <volans@cumin2002> START - Cookbook sre.dns.netbox [production]
12:50 <volans@cumin1001> START - Cookbook sre.dns.netbox [production]
11:47 <isaranto@deploy2002> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'llm' for release 'main' . [production]
11:46 <jnuche@deploy2002> Finished deploy [releng/jenkins-deploy@6f09297] (releasing): (no justification provided) (duration: 01m 08s) [production]
11:44 <jnuche@deploy2002> Started deploy [releng/jenkins-deploy@6f09297] (releasing): (no justification provided) [production]
11:30 <isaranto@deploy2002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'llm' for release 'main' . [production]
08:36 <isaranto@deploy2002> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'llm' for release 'main' . [production]
07:33 <volans@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1001.eqiad.wmnet with OS bullseye [production]