801-850 of 10000 results (51ms)
2022-06-09 ยง
20:52 <cmjohnson@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host an-worker1143.eqiad.wmnet with OS buster [production]
20:49 <bking@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host elastic2053.codfw.wmnet with OS bullseye [production]
20:47 <cmjohnson@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host an-worker1145.eqiad.wmnet with OS buster [production]
20:46 <cmjohnson@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host an-worker1146.eqiad.wmnet with OS buster [production]
20:46 <cmjohnson@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host an-worker1144.eqiad.wmnet with OS buster [production]
20:45 <cmjohnson@cumin1001> START - Cookbook sre.hosts.reimage for host an-worker1142.eqiad.wmnet with OS buster [production]
20:44 <bking@cumin1001> START - Cookbook sre.hosts.reimage for host elastic2053.codfw.wmnet with OS bullseye [production]
20:43 <thcipriani> end utc late backport window [production]
20:40 <cmjohnson@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1142.eqiad.wmnet with OS buster [production]
20:39 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs1016.mgmt.eqiad.wmnet with reboot policy FORCED [production]
20:39 <thcipriani@deploy1002> Synchronized php-1.39.0-wmf.15/extensions/GrowthExperiments/modules: Backport: [[gerrit:803969|Suggested edits: Fix loading states when fetching additional tasks (T309926)]] (duration: 03m 37s) [production]
20:38 <cmjohnson@cumin1001> START - Cookbook sre.hosts.reimage for host an-worker1146.eqiad.wmnet with OS buster [production]
20:37 <cmjohnson@cumin1001> START - Cookbook sre.hosts.reimage for host an-worker1145.eqiad.wmnet with OS buster [production]
20:36 <cmjohnson@cumin1001> START - Cookbook sre.hosts.reimage for host an-worker1144.eqiad.wmnet with OS buster [production]
20:35 <cmjohnson@cumin1001> START - Cookbook sre.hosts.reimage for host an-worker1143.eqiad.wmnet with OS buster [production]
20:34 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
20:33 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
20:33 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
20:32 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
20:26 <cmjohnson@cumin1001> START - Cookbook sre.hosts.reimage for host an-worker1142.eqiad.wmnet with OS buster [production]
20:23 <cmjohnson@cumin1001> START - Cookbook sre.hosts.provision for host wdqs1016.mgmt.eqiad.wmnet with reboot policy FORCED [production]
20:23 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs1015.mgmt.eqiad.wmnet with reboot policy FORCED [production]
20:18 <thcipriani> mwmaint1002:mwscript namespaceDupes.php kywiki --fix [production]
20:16 <thcipriani@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:803916|kywiki: Add $wgSitename, $wgMetaNamespace & $wgMetaNamespaceTalk (T309866)]] (duration: 03m 36s) [production]
20:11 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
20:11 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
20:11 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
20:08 <cmjohnson@cumin1001> START - Cookbook sre.hosts.provision for host wdqs1015.mgmt.eqiad.wmnet with reboot policy FORCED [production]
20:06 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
20:03 <bking@cumin1001> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host elastic2053.codfw.wmnet [production]
20:03 <bking@cumin1001> START - Cookbook sre.hosts.reboot-single for host elastic2053.codfw.wmnet [production]
20:01 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs1014.mgmt.eqiad.wmnet with reboot policy FORCED [production]
19:59 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
19:56 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
19:54 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
19:51 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
19:51 <cmjohnson@cumin1001> END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [production]
19:47 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
19:47 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
19:46 <cmjohnson@cumin1001> START - Cookbook sre.hosts.provision for host wdqs1014.mgmt.eqiad.wmnet with reboot policy FORCED [production]
19:43 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
19:36 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
19:32 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
19:21 <ryankemper@cumin1001> END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.UPGRADE (1 nodes at a time) for ElasticSearch cluster relforge: relforge plugin upgrade - ryankemper@cumin1001 - T309648 [production]
19:21 <ryankemper@cumin1001> START - Cookbook sre.elasticsearch.rolling-operation Operation.UPGRADE (1 nodes at a time) for ElasticSearch cluster relforge: relforge plugin upgrade - ryankemper@cumin1001 - T309648 [production]
19:17 <bking@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host elastic2053.codfw.wmnet with OS bullseye [production]
19:06 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
19:02 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
19:02 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
18:58 <ryankemper@cumin1001> END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.UPGRADE (1 nodes at a time) for ElasticSearch cluster relforge: relforge plugin upgrade - ryankemper@cumin1001 - T309648 [production]