1901-1950 of 10000 results (67ms)
2022-06-29 ยง
19:28 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db2156.mgmt.codfw.wmnet with reboot policy FORCED [production]
19:26 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db2155.mgmt.codfw.wmnet with reboot policy FORCED [production]
19:23 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host db2156.mgmt.codfw.wmnet with reboot policy FORCED [production]
19:22 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db2156.mgmt.codfw.wmnet with reboot policy FORCED [production]
19:15 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host db2156.mgmt.codfw.wmnet with reboot policy FORCED [production]
19:15 <pt1979@cumin2002> END (ERROR) - Cookbook sre.hosts.provision (exit_code=97) for host db2156.mgmt.codfw.wmnet with reboot policy FORCED [production]
19:11 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host db2156.mgmt.codfw.wmnet with reboot policy FORCED [production]
19:10 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host db2155.mgmt.codfw.wmnet with reboot policy FORCED [production]
18:56 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db2153.mgmt.codfw.wmnet with reboot policy FORCED [production]
18:56 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db2154.mgmt.codfw.wmnet with reboot policy FORCED [production]
18:34 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
18:31 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
18:28 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host db2154.mgmt.codfw.wmnet with reboot policy FORCED [production]
18:28 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1049.mgmt.eqiad.wmnet with reboot policy FORCED [production]
18:28 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1051.mgmt.eqiad.wmnet with reboot policy FORCED [production]
18:28 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1048.mgmt.eqiad.wmnet with reboot policy FORCED [production]
18:28 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1052.mgmt.eqiad.wmnet with reboot policy FORCED [production]
18:28 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1053.mgmt.eqiad.wmnet with reboot policy FORCED [production]
18:28 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1050.mgmt.eqiad.wmnet with reboot policy FORCED [production]
18:27 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host db2153.mgmt.codfw.wmnet with reboot policy FORCED [production]
18:27 <bking@cumin1001> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: eqiad cluster restart to pickup swift-s3 plugin - bking@cumin1001 - T309648 [production]
18:26 <pt1979@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
18:24 <robh@cumin1001> START - Cookbook sre.hosts.provision for host dumpsdata1006.mgmt.eqiad.wmnet with reboot policy FORCED [production]
18:21 <pt1979@cumin2002> START - Cookbook sre.dns.netbox [production]
18:18 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
18:15 <robh@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host dumpsdata1006.eqiad.wmnet with OS bullseye [production]
18:15 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
18:14 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
18:14 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depool db1128', diff saved to https://phabricator.wikimedia.org/P30628 and previous config saved to /var/cache/conftool/dbconfig/20220629-181438-ladsgroup.json [production]
18:13 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
18:12 <robh@cumin1001> START - Cookbook sre.hosts.reimage for host dumpsdata1006.eqiad.wmnet with OS bullseye [production]
18:11 <cmjohnson@cumin1001> START - Cookbook sre.hosts.provision for host cloudvirt1050.mgmt.eqiad.wmnet with reboot policy FORCED [production]
18:11 <cmjohnson@cumin1001> START - Cookbook sre.hosts.provision for host cloudvirt1053.mgmt.eqiad.wmnet with reboot policy FORCED [production]
18:11 <cmjohnson@cumin1001> START - Cookbook sre.hosts.provision for host cloudvirt1052.mgmt.eqiad.wmnet with reboot policy FORCED [production]
18:11 <cmjohnson@cumin1001> START - Cookbook sre.hosts.provision for host cloudvirt1048.mgmt.eqiad.wmnet with reboot policy FORCED [production]
18:11 <cmjohnson@cumin1001> START - Cookbook sre.hosts.provision for host cloudvirt1049.mgmt.eqiad.wmnet with reboot policy FORCED [production]
18:11 <cmjohnson@cumin1001> START - Cookbook sre.hosts.provision for host cloudvirt1051.mgmt.eqiad.wmnet with reboot policy FORCED [production]
18:11 <dduvall@deploy1002> Synchronized php: group1 wikis to 1.39.0-wmf.18 refs T308071 (duration: 03m 35s) [production]
18:09 <robh@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host dumpsdata1007.eqiad.wmnet with OS bullseye [production]
18:08 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
18:08 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
18:08 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
18:07 <dduvall@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.39.0-wmf.18 refs T308071 [production]
18:07 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
18:06 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
18:02 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
17:55 <pt1979@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
17:51 <pt1979@cumin2002> START - Cookbook sre.dns.netbox [production]
17:34 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dumpsdata1007.eqiad.wmnet with reason: host reimage [production]
17:31 <robh@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on dumpsdata1007.eqiad.wmnet with reason: host reimage [production]