751-800 of 10000 results (74ms)
2023-09-04 §
08:33 <jayme@cumin1001> END (FAIL) - Cookbook sre.k8s.reboot-nodes (exit_code=1) rolling reboot on A:wikikube-staging-worker-eqiad [production]
08:33 <jayme@cumin1001> START - Cookbook sre.hosts.reboot-single for host registry1003.eqiad.wmnet [production]
08:31 <jayme@cumin1001> START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad [production]
08:31 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts bast5003.wikimedia.org [production]
08:31 <jmm@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
08:31 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: bast5003.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" [production]
08:28 <jayme@cumin1001> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host kubestagemaster2002.codfw.wmnet [production]
08:27 <jmm@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: bast5003.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" [production]
08:25 <jayme@cumin1001> START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw [production]
08:23 <jayme@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet [production]
08:23 <jayme@cumin1001> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host kubestagemaster1002.eqiad.wmnet [production]
08:22 <jmm@cumin2002> START - Cookbook sre.dns.netbox [production]
08:19 <jayme@cumin1001> conftool action : set/pooled=true; selector: dnsdisc=helm-charts,name=eqiad [production]
08:18 <jayme@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum1001.eqiad.wmnet [production]
08:18 <jmm@cumin2002> START - Cookbook sre.hosts.decommission for hosts bast5003.wikimedia.org [production]
08:18 <jayme@cumin1001> START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet [production]
08:17 <jayme@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet [production]
08:17 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts bast4004.wikimedia.org [production]
08:17 <jmm@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
08:17 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: bast4004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" [production]
08:15 <jayme@cumin1001> START - Cookbook sre.hosts.reboot-single for host kubestagemaster2002.codfw.wmnet [production]
08:15 <jmm@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: bast4004.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" [production]
08:15 <jayme@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kubestagemaster2001.codfw.wmnet [production]
08:14 <elukey@cumin1001> START - Cookbook sre.ganeti.reboot-vm for VM orespoolcounter2003.codfw.wmnet [production]
08:14 <jayme@cumin1001> START - Cookbook sre.hosts.reboot-single for host chartmuseum1001.eqiad.wmnet [production]
08:14 <jayme@cumin1001> conftool action : set/pooled=false; selector: dnsdisc=helm-charts,name=eqiad [production]
08:14 <jayme@cumin1001> conftool action : set/pooled=true; selector: dnsdisc=helm-charts,name=codfw [production]
08:14 <jayme@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum2001.codfw.wmnet [production]
08:13 <jayme@cumin1001> START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet [production]
08:13 <jmm@cumin2002> START - Cookbook sre.dns.netbox [production]
08:11 <jayme@cumin1001> START - Cookbook sre.hosts.reboot-single for host kubestagemaster1002.eqiad.wmnet [production]
08:10 <jayme@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kubestagemaster1001.eqiad.wmnet [production]
08:10 <jayme@cumin1001> START - Cookbook sre.hosts.reboot-single for host chartmuseum2001.codfw.wmnet [production]
08:09 <jayme@cumin1001> conftool action : set/pooled=false; selector: dnsdisc=helm-charts,name=codfw [production]
08:08 <jmm@cumin2002> START - Cookbook sre.hosts.decommission for hosts bast4004.wikimedia.org [production]
08:03 <jayme@cumin1001> START - Cookbook sre.hosts.reboot-single for host kubestagemaster1001.eqiad.wmnet [production]
08:03 <jayme@cumin1001> START - Cookbook sre.hosts.reboot-single for host kubestagemaster2001.codfw.wmnet [production]
08:00 <elukey> restart kubelet on ml-serve1002 to check if stale prometheus metrics are the cause of the stop_container alert [production]
08:00 <elukey@cumin1001> END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM orespoolcounter2004.codfw.wmnet [production]
07:59 <elukey@cumin1001> START - Cookbook sre.ganeti.reboot-vm for VM orespoolcounter2004.codfw.wmnet [production]
07:35 <Emperor> restart tcpircbot-logmsgbot on alert1001 [production]
07:22 <moritzm> failover ganeti masters in drmrs to ganeti6001/ganeti6002 [production]
06:12 <XioNoX> push new pfw policies - T345288 [production]
2023-09-02 §
15:51 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db1128.eqiad.wmnet with reason: depooled after replica lag page, two days [production]
15:50 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db1128.eqiad.wmnet with reason: depooled after replica lag page, two days [production]
15:49 <sukhe@cumin2002> dbctl commit (dc=all): 'Depool db1128', diff saved to https://phabricator.wikimedia.org/P52244 and previous config saved to /var/cache/conftool/dbconfig/20230902-154903-sukhe.json [production]
05:45 <ariel@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dumpsdata1003.eqiad.wmnet [production]
05:39 <ariel@cumin1001> START - Cookbook sre.hosts.reboot-single for host dumpsdata1003.eqiad.wmnet [production]
05:38 <ariel@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host snapshot1008.eqiad.wmnet [production]
05:32 <ariel@cumin1001> START - Cookbook sre.hosts.reboot-single for host snapshot1008.eqiad.wmnet [production]