3851-3900 of 10000 results (84ms)
2022-10-26 §
06:20 <marostegui@cumin1001> dbctl commit (dc=all): 'es1029 (re)pooling @ 1%: After upgrade', diff saved to https://phabricator.wikimedia.org/P36370 and previous config saved to /var/cache/conftool/dbconfig/20221026-062056-root.json [production]
06:18 <oblivian@deploy1002> helmfile [staging] DONE helmfile.d/services/shellbox: apply [production]
06:10 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool es1029 es1030 es1031', diff saved to https://phabricator.wikimedia.org/P36369 and previous config saved to /var/cache/conftool/dbconfig/20221026-061044-root.json [production]
06:08 <oblivian@deploy1002> helmfile [staging] START helmfile.d/services/shellbox: apply [production]
2022-10-25 §
22:33 <robh@cumin2002> START - Cookbook sre.hosts.provision for host cp4052.mgmt.ulsfo.wmnet with reboot policy FORCED [production]
22:32 <robh@cumin2002> START - Cookbook sre.hosts.provision for host cp4051.mgmt.ulsfo.wmnet with reboot policy FORCED [production]
22:30 <robh@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cp4040.mgmt.ulsfo.wmnet with reboot policy FORCED [production]
22:29 <robh@cumin2002> START - Cookbook sre.hosts.provision for host cp4040.mgmt.ulsfo.wmnet with reboot policy FORCED [production]
22:28 <robh@cumin2002> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cp4040 [production]
22:28 <robh@cumin2002> START - Cookbook sre.network.configure-switch-interfaces for host cp4040 [production]
22:24 <robh@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host lvs4009.mgmt.ulsfo.wmnet with reboot policy FORCED [production]
22:24 <robh@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host lvs4010.mgmt.ulsfo.wmnet with reboot policy FORCED [production]
22:24 <robh@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cp4040.mgmt.ulsfo.wmnet with reboot policy FORCED [production]
22:20 <robh@cumin2002> START - Cookbook sre.hosts.provision for host lvs4010.mgmt.ulsfo.wmnet with reboot policy FORCED [production]
22:19 <robh@cumin2002> START - Cookbook sre.hosts.provision for host lvs4009.mgmt.ulsfo.wmnet with reboot policy FORCED [production]
22:18 <robh@cumin2002> START - Cookbook sre.hosts.provision for host cp4040.mgmt.ulsfo.wmnet with reboot policy FORCED [production]
22:18 <robh@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cp4040.mgmt.ulsfo.wmnet with reboot policy FORCED [production]
22:18 <robh@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cp4042.mgmt.ulsfo.wmnet with reboot policy FORCED [production]
22:17 <robh@cumin2002> START - Cookbook sre.hosts.provision for host cp4042.mgmt.ulsfo.wmnet with reboot policy FORCED [production]
22:17 <robh@cumin2002> START - Cookbook sre.hosts.provision for host cp4040.mgmt.ulsfo.wmnet with reboot policy FORCED [production]
22:17 <robh@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp4050.mgmt.ulsfo.wmnet with reboot policy FORCED [production]
22:17 <robh@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp4048.mgmt.ulsfo.wmnet with reboot policy FORCED [production]
22:16 <robh@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp4046.mgmt.ulsfo.wmnet with reboot policy FORCED [production]
22:04 <robh@cumin2002> START - Cookbook sre.hosts.provision for host cp4050.mgmt.ulsfo.wmnet with reboot policy FORCED [production]
22:04 <robh@cumin2002> START - Cookbook sre.hosts.provision for host cp4048.mgmt.ulsfo.wmnet with reboot policy FORCED [production]
22:03 <robh@cumin2002> START - Cookbook sre.hosts.provision for host cp4046.mgmt.ulsfo.wmnet with reboot policy FORCED [production]
22:02 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2175 (T321312)', diff saved to https://phabricator.wikimedia.org/P36368 and previous config saved to /var/cache/conftool/dbconfig/20221025-220249-ladsgroup.json [production]
22:01 <robh@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
22:01 <robh@cumin2002> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cp4051 [production]
22:00 <robh@cumin2002> START - Cookbook sre.network.configure-switch-interfaces for host cp4051 [production]
21:59 <robh@cumin2002> START - Cookbook sre.dns.netbox [production]
21:57 <robh@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp4044.mgmt.ulsfo.wmnet with reboot policy FORCED [production]
21:51 <robh@cumin2002> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts cp4035.ulsfo.wmnet [production]
21:51 <robh@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
21:50 <robh@cumin2002> START - Cookbook sre.dns.netbox [production]
21:50 <bking@cumin2002> END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) [production]
21:47 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P36367 and previous config saved to /var/cache/conftool/dbconfig/20221025-214743-ladsgroup.json [production]
21:47 <robh@cumin2002> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host lvs4010 [production]
21:46 <bking@cumin2002> START - Cookbook sre.wdqs.data-transfer [production]
21:46 <robh@cumin2002> START - Cookbook sre.network.configure-switch-interfaces for host lvs4010 [production]
21:46 <robh@cumin2002> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host lvs4009 [production]
21:45 <robh@cumin2002> START - Cookbook sre.network.configure-switch-interfaces for host lvs4009 [production]
21:43 <robh@cumin2002> START - Cookbook sre.hosts.provision for host cp4044.mgmt.ulsfo.wmnet with reboot policy FORCED [production]
21:43 <robh@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp4042.mgmt.ulsfo.wmnet with reboot policy FORCED [production]
21:42 <robh@cumin2002> START - Cookbook sre.hosts.decommission for hosts cp4035.ulsfo.wmnet [production]
21:41 <bking@cumin2002> END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) [production]
21:38 <robh@cumin2002> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts cp4033.ulsfo.wmnet [production]
21:38 <robh@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
21:37 <robh@cumin2002> START - Cookbook sre.dns.netbox [production]
21:37 <robh@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]