401-450 of 10000 results (75ms)
2023-09-01 ยง
09:38 <cmooney@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add management record for lsw1-a5-codfw - cmooney@cumin1001" [production]
09:38 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host rpki2002.codfw.wmnet [production]
09:38 <cmooney@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add management record for lsw1-a5-codfw - cmooney@cumin1001" [production]
09:37 <elukey@deploy1002> helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. [production]
09:36 <mvernon@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1059.eqiad.wmnet [production]
09:36 <elukey@deploy1002> helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. [production]
09:35 <cmooney@cumin1001> START - Cookbook sre.dns.netbox [production]
09:35 <cmooney@cumin1001> START - Cookbook sre.network.provision for device lsw1-a5-codfw.mgmt.codfw.wmnet [production]
09:35 <elukey@deploy1002> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. [production]
09:34 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki1001.eqiad.wmnet [production]
09:34 <elukey@deploy1002> helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. [production]
09:31 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host rpki1001.eqiad.wmnet [production]
09:29 <elukey@deploy1002> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. [production]
09:29 <elukey@deploy1002> helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. [production]
09:26 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2058.codfw.wmnet [production]
09:18 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ping1003.eqiad.wmnet [production]
09:14 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ping1003.eqiad.wmnet [production]
09:14 <mvernon@cumin1001> START - Cookbook sre.hosts.reboot-single for host ms-be1059.eqiad.wmnet [production]
09:14 <mvernon@cumin2002> START - Cookbook sre.hosts.reboot-single for host ms-be2058.codfw.wmnet [production]
09:08 <cmooney@cumin1001> END (PASS) - Cookbook sre.network.provision (exit_code=0) for device lsw1-a4-codfw.mgmt.codfw.wmnet [production]
09:04 <claime> Running puppet on 'A:cp-text and P{P:trafficserver::backend}' - T341780 [production]
09:02 <claime> Push 4% of global traffic to mw-on-k8s - T341780 [production]
09:00 <elukey@deploy1002> helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. [production]
08:59 <elukey@deploy1002> helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. [production]
08:54 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ping2003.codfw.wmnet [production]
08:51 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ping2003.codfw.wmnet [production]
08:50 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1003.eqiad.wmnet [production]
08:49 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on testreduce1002.eqiad.wmnet with reason: WIP [production]
08:49 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on testreduce1002.eqiad.wmnet with reason: WIP [production]
08:45 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host sretest1003.eqiad.wmnet [production]
08:40 <claime> Raised mw-web and mw-api-ext capacity by ~30% - T341780 [production]
08:39 <cgoubert@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-web: apply [production]
08:39 <cgoubert@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-web: apply [production]
08:38 <cgoubert@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-web: apply [production]
08:38 <cgoubert@deploy1002> helmfile [codfw] START helmfile.d/services/mw-web: apply [production]
08:37 <cmooney@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
08:37 <cmooney@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add management record for lsw1-a4-codfw - cmooney@cumin1001" [production]
08:37 <cgoubert@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply [production]
08:36 <cmooney@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add management record for lsw1-a4-codfw - cmooney@cumin1001" [production]
08:36 <cgoubert@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply [production]
08:35 <cgoubert@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply [production]
08:34 <cgoubert@deploy1002> helmfile [codfw] START helmfile.d/services/mw-api-ext: apply [production]
08:34 <cmooney@cumin1001> START - Cookbook sre.dns.netbox [production]
08:34 <cmooney@cumin1001> START - Cookbook sre.network.provision for device lsw1-a4-codfw.mgmt.codfw.wmnet [production]
08:33 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet [production]
08:33 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet [production]
08:30 <cmooney@cumin1001> END (PASS) - Cookbook sre.network.provision (exit_code=0) for device lsw1-a3-codfw.mgmt.codfw.wmnet [production]
08:26 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet [production]
08:21 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet [production]
07:58 <cmooney@cumin1001> END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [production]