1751-1800 of 10000 results (77ms)
2023-08-29 ยง
22:50 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1199.eqiad.wmnet with reason: Maintenance [production]
22:50 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1199.eqiad.wmnet with reason: Maintenance [production]
22:50 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1190 (T343718)', diff saved to https://phabricator.wikimedia.org/P51960 and previous config saved to /var/cache/conftool/dbconfig/20230829-225010-ladsgroup.json [production]
22:40 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host kubernetes2026.mgmt.codfw.wmnet with reboot policy FORCED [production]
22:35 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P51959 and previous config saved to /var/cache/conftool/dbconfig/20230829-223504-ladsgroup.json [production]
22:19 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P51958 and previous config saved to /var/cache/conftool/dbconfig/20230829-221958-ladsgroup.json [production]
22:04 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1190 (T343718)', diff saved to https://phabricator.wikimedia.org/P51957 and previous config saved to /var/cache/conftool/dbconfig/20230829-220451-ladsgroup.json [production]
21:53 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host kubernetes2026.mgmt.codfw.wmnet with reboot policy FORCED [production]
21:50 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host kubernetes2026.mgmt.codfw.wmnet with reboot policy FORCED [production]
21:50 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host kubernetes2026.mgmt.codfw.wmnet with reboot policy FORCED [production]
21:49 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host kubernetes2027.mgmt.codfw.wmnet with reboot policy FORCED [production]
21:38 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host kubernetes2027.mgmt.codfw.wmnet with reboot policy FORCED [production]
21:37 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host kubernetes2026.mgmt.codfw.wmnet with reboot policy FORCED [production]
21:36 <eevans@cumin1001> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host restbase1030.eqiad.wmnet [production]
21:35 <jhancock@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
21:33 <jhancock@cumin2002> START - Cookbook sre.dns.netbox [production]
21:32 <jhancock@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
21:30 <jhancock@cumin2002> START - Cookbook sre.dns.netbox [production]
21:30 <jhancock@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
21:29 <jhancock@cumin2002> START - Cookbook sre.dns.netbox [production]
21:29 <jhancock@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
21:27 <jhancock@cumin2002> START - Cookbook sre.dns.netbox [production]
21:27 <jhancock@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
21:26 <jhancock@cumin2002> START - Cookbook sre.dns.netbox [production]
21:26 <jhancock@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
21:26 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2172 (T343718)', diff saved to https://phabricator.wikimedia.org/P51956 and previous config saved to /var/cache/conftool/dbconfig/20230829-212619-ladsgroup.json [production]
21:26 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2172.codfw.wmnet with reason: Maintenance [production]
21:26 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2172.codfw.wmnet with reason: Maintenance [production]
21:25 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2155 (T343718)', diff saved to https://phabricator.wikimedia.org/P51955 and previous config saved to /var/cache/conftool/dbconfig/20230829-212558-ladsgroup.json [production]
21:25 <jhancock@cumin2002> START - Cookbook sre.dns.netbox [production]
21:23 <jhancock@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
21:23 <eevans@cumin1001> START - Cookbook sre.hosts.reboot-single for host restbase1030.eqiad.wmnet [production]
21:22 <jhancock@cumin2002> START - Cookbook sre.dns.netbox [production]
21:21 <jhancock@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
21:20 <jhancock@cumin2002> START - Cookbook sre.dns.netbox [production]
21:20 <jhancock@cumin2002> END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [production]
21:20 <jhancock@cumin2002> END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [production]
21:20 <jhancock@cumin2002> END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [production]
21:20 <jhancock@cumin2002> END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [production]
21:20 <jhancock@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
21:19 <jhancock@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add newly racked kubernetes2026 hosts in codfw - jhancock@cumin2002" [production]
21:19 <jhancock@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add newly racked kubernetes2026 hosts in codfw - jhancock@cumin2002" [production]
21:18 <jhancock@cumin2002> END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [production]
21:14 <jhancock@cumin2002> START - Cookbook sre.dns.netbox [production]
21:13 <jhancock@cumin2002> START - Cookbook sre.dns.netbox [production]
21:13 <jhancock@cumin2002> START - Cookbook sre.dns.netbox [production]
21:13 <jhancock@cumin2002> START - Cookbook sre.dns.netbox [production]
21:13 <jhancock@cumin2002> START - Cookbook sre.dns.netbox [production]
21:13 <jhancock@cumin2002> START - Cookbook sre.dns.netbox [production]
21:10 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P51954 and previous config saved to /var/cache/conftool/dbconfig/20230829-211052-ladsgroup.json [production]