1151-1200 of 10000 results (25ms)
2024-09-26 §
13:36 <elukey@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
13:36 <elukey@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM aux-k8s-worker1003.eqiad.wmnet - elukey@cumin1002" [production]
13:36 <elukey@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM aux-k8s-worker1003.eqiad.wmnet - elukey@cumin1002" [production]
13:33 <elukey@cumin1002> START - Cookbook sre.dns.netbox [production]
13:32 <elukey@cumin1002> START - Cookbook sre.ganeti.makevm for new host aux-k8s-worker1003.eqiad.wmnet [production]
13:26 <elukey@cumin1002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host logging-hd2005.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
13:23 <elukey@cumin1002> START - Cookbook sre.hosts.provision for host logging-hd2005.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
13:21 <elukey@cumin1002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host logging-hd2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
13:17 <elukey@cumin1002> START - Cookbook sre.hosts.provision for host logging-hd2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
10:21 <elukey> start dry run of docker distribution GC on registry1004 (info in https://phabricator.wikimedia.org/T375645#10176397, you can find a root tmux session named as the task on the host to stop) [production]
09:51 <elukey@cumin1002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts registry2003.codfw.wmnet [production]
09:51 <elukey@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
09:51 <elukey@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: registry2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - elukey@cumin1002" [production]
09:50 <elukey@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: registry2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - elukey@cumin1002" [production]
09:47 <elukey@cumin1002> START - Cookbook sre.dns.netbox [production]
09:43 <elukey@cumin1002> START - Cookbook sre.hosts.decommission for hosts registry2003.codfw.wmnet [production]
09:41 <elukey@cumin1002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts registry1003.eqiad.wmnet [production]
09:41 <elukey@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
09:41 <elukey@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: registry1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - elukey@cumin1002" [production]
09:40 <elukey@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: registry1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - elukey@cumin1002" [production]
09:37 <elukey@cumin1002> START - Cookbook sre.dns.netbox [production]
09:33 <elukey@cumin1002> START - Cookbook sre.hosts.decommission for hosts registry1003.eqiad.wmnet [production]
09:30 <elukey@puppetserver1001> conftool action : set/pooled=inactive; selector: name=registry2003.codfw.wmnet,service=docker-registry,dc=eqiad [production]
09:30 <elukey@puppetserver1001> conftool action : set/pooled=inactive; selector: name=registry1003.eqiad.wmnet,service=docker-registry,dc=eqiad [production]
09:27 <elukey@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host registry2004.codfw.wmnet with OS bookworm [production]
09:11 <elukey@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on registry2004.codfw.wmnet with reason: host reimage [production]
09:08 <elukey@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on registry2004.codfw.wmnet with reason: host reimage [production]
08:49 <elukey@cumin1002> START - Cookbook sre.hosts.reimage for host registry2004.codfw.wmnet with OS bookworm [production]
08:10 <elukey@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host registry1004.eqiad.wmnet with OS bookworm [production]
07:37 <elukey@cumin1002> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on registry1004.eqiad.wmnet with reason: host reimage [production]
07:36 <elukey@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on registry1004.eqiad.wmnet with reason: host reimage [production]
07:20 <elukey@cumin1002> START - Cookbook sre.hosts.reimage for host registry1004.eqiad.wmnet with OS bookworm [production]
07:00 <elukey> apt-get clean on grafana2001 to free some space in the root partition [production]
2024-09-25 §
09:11 <elukey> set max-catalog-entries (changes the default catalog pagination) to 25 for docker-registry - T348876 [production]
08:06 <elukey> set max-catalog-entries (changes the default catalog pagination) to 50 for docker-registry - T348876 [production]
2024-09-24 §
12:32 <elukey@deploy1003> helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: sync [production]
12:32 <elukey@deploy1003> helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: sync [production]
12:31 <elukey@deploy1003> helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: sync [production]
12:30 <elukey@deploy1003> helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: sync [production]
12:29 <elukey@deploy1003> helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: sync [production]
12:29 <elukey@deploy1003> helmfile [staging] START helmfile.d/services/tegola-vector-tiles: sync [production]
08:36 <elukey@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on puppetmaster2001.codfw.wmnet with reason: WIP - working on puppet runs [production]
08:36 <elukey@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on puppetmaster2001.codfw.wmnet with reason: WIP - working on puppet runs [production]
2024-09-23 §
16:08 <elukey@puppetserver1001> conftool action : set/pooled=no; selector: name=registry1003.eqiad.wmnet,service=docker-registry,dc=eqiad [production]
16:08 <elukey@puppetserver1001> conftool action : set/weight=10; selector: name=registry1005.eqiad.wmnet,service=docker-registry,dc=eqiad [production]
16:08 <elukey@puppetserver1001> conftool action : set/pooled=yes; selector: name=registry1005.eqiad.wmnet,service=docker-registry,dc=eqiad [production]
16:04 <elukey@puppetserver1001> conftool action : set/pooled=true,weight=10; selector: name=registry1005.eqiad.wmnet,service=docker-registry,dc=eqiad [production]
14:50 <elukey@puppetserver1001> conftool action : set/pooled=true,weight=10; selector: name=registry1005.eqiad.wmnet,service=docker-registry,dc=eqiad [production]
14:49 <elukey@puppetserver1001> conftool action : set/pooled=true; selector: name=registry1005.eqiad.wmnet [production]
14:49 <elukey@puppetserver1001> conftool action : set/weight=10; selector: name=registry1005.eqiad.wmnet [production]