4251-4300 of 10000 results (89ms)
2023-07-18 ยง
16:24 <btullis@cumin1001> START - Cookbook sre.hosts.reimage for host analytics1075.eqiad.wmnet with OS bullseye [production]
16:03 <robh@cumin1001> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host lvs1015 [production]
16:03 <robh@cumin1001> START - Cookbook sre.network.configure-switch-interfaces for host lvs1015 [production]
16:03 <robh@cumin1001> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host lvs1014 [production]
16:03 <robh@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
16:03 <robh@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs10145 relocation - robh@cumin1001" [production]
16:03 <robh@cumin1001> START - Cookbook sre.network.configure-switch-interfaces for host lvs1014 [production]
16:02 <robh@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs10145 relocation - robh@cumin1001" [production]
16:01 <robh@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host lvs1013.eqiad.wmnet with OS bullseye [production]
16:00 <robh@cumin1001> START - Cookbook sre.dns.netbox [production]
15:44 <dancy@deploy1002> Started scap: testwikis wikis to 1.41.0-wmf.18 refs T340246 [production]
15:31 <fabfur@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts lvs1016.eqiad.wmnet [production]
15:31 <fabfur@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
15:31 <fabfur@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs1016.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - fabfur@cumin1001" [production]
15:31 <fabfur@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs1016.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - fabfur@cumin1001" [production]
15:28 <fabfur@cumin1001> START - Cookbook sre.dns.netbox [production]
15:23 <fabfur@cumin1001> START - Cookbook sre.hosts.decommission for hosts lvs1016.eqiad.wmnet [production]
15:22 <robh@cumin1001> START - Cookbook sre.hosts.reimage for host lvs1013.eqiad.wmnet with OS bullseye [production]
15:21 <robh@cumin1001> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host lvs1013 [production]
15:21 <robh@cumin1001> START - Cookbook sre.network.configure-switch-interfaces for host lvs1013 [production]
15:20 <robh@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host lvs1013.eqiad.wmnet with OS bullseye [production]
15:18 <robh@cumin1001> START - Cookbook sre.hosts.reimage for host lvs1013.eqiad.wmnet with OS bullseye [production]
15:08 <robh@cumin1001> END (ERROR) - Cookbook sre.hosts.provision (exit_code=97) for host lvs1013.mgmt.eqiad.wmnet with reboot policy FORCED [production]
15:02 <fabfur@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts lvs1015.eqiad.wmnet [production]
15:02 <fabfur@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
15:01 <fabfur@cumin1001> START - Cookbook sre.dns.netbox [production]
15:00 <robh@cumin1001> START - Cookbook sre.hosts.provision for host lvs1013.mgmt.eqiad.wmnet with reboot policy FORCED [production]
14:57 <robh@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
14:57 <robh@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs1013 relocation - robh@cumin1001" [production]
14:56 <robh@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs1013 relocation - robh@cumin1001" [production]
14:55 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1198', diff saved to https://phabricator.wikimedia.org/P49571 and previous config saved to /var/cache/conftool/dbconfig/20230718-145529-root.json [production]
14:54 <robh@cumin1001> START - Cookbook sre.dns.netbox [production]
14:54 <fabfur@cumin1001> START - Cookbook sre.hosts.decommission for hosts lvs1015.eqiad.wmnet [production]
14:45 <sukhe> dns2004 upgrade to pdns-rec 4.8.4: T341611 [production]
14:38 <ayounsi@cumin1001> END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin1001.eqiad.wmnet with reason: Release v0.6.3 - ayounsi@cumin1001 [production]
14:37 <fabfur@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts lvs1014.eqiad.wmnet [production]
14:37 <fabfur@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
14:37 <fabfur@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - fabfur@cumin1001" [production]
14:36 <fabfur@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - fabfur@cumin1001" [production]
14:36 <ayounsi@cumin1001> START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin1001.eqiad.wmnet with reason: Release v0.6.3 - ayounsi@cumin1001 [production]
14:35 <oblivian@deploy1002> helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply [production]
14:35 <oblivian@deploy1002> helmfile [eqiad] START helmfile.d/services/api-gateway: apply [production]
14:34 <fabfur@cumin1001> START - Cookbook sre.dns.netbox [production]
14:34 <oblivian@deploy1002> helmfile [codfw] DONE helmfile.d/services/api-gateway: apply [production]
14:33 <oblivian@deploy1002> helmfile [codfw] START helmfile.d/services/api-gateway: apply [production]
14:33 <oblivian@deploy1002> helmfile [staging] DONE helmfile.d/services/api-gateway: apply [production]
14:30 <oblivian@deploy1002> helmfile [staging] START helmfile.d/services/api-gateway: apply [production]
14:29 <fabfur@cumin1001> START - Cookbook sre.hosts.decommission for hosts lvs1014.eqiad.wmnet [production]
14:22 <fabfur@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts lvs1013.eqiad.wmnet [production]
14:22 <fabfur@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]