5701-5750 of 10000 results (66ms)
2022-06-09 ยง
20:11 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
20:11 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
20:08 <cmjohnson@cumin1001> START - Cookbook sre.hosts.provision for host wdqs1015.mgmt.eqiad.wmnet with reboot policy FORCED [production]
20:06 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
20:03 <bking@cumin1001> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host elastic2053.codfw.wmnet [production]
20:03 <bking@cumin1001> START - Cookbook sre.hosts.reboot-single for host elastic2053.codfw.wmnet [production]
20:01 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs1014.mgmt.eqiad.wmnet with reboot policy FORCED [production]
19:59 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
19:56 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
19:54 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
19:51 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
19:51 <cmjohnson@cumin1001> END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [production]
19:47 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
19:47 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
19:46 <cmjohnson@cumin1001> START - Cookbook sre.hosts.provision for host wdqs1014.mgmt.eqiad.wmnet with reboot policy FORCED [production]
19:43 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
19:36 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
19:32 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
19:21 <ryankemper@cumin1001> END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.UPGRADE (1 nodes at a time) for ElasticSearch cluster relforge: relforge plugin upgrade - ryankemper@cumin1001 - T309648 [production]
19:21 <ryankemper@cumin1001> START - Cookbook sre.elasticsearch.rolling-operation Operation.UPGRADE (1 nodes at a time) for ElasticSearch cluster relforge: relforge plugin upgrade - ryankemper@cumin1001 - T309648 [production]
19:17 <bking@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host elastic2053.codfw.wmnet with OS bullseye [production]
19:06 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
19:02 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
19:02 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
18:58 <ryankemper@cumin1001> END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.UPGRADE (1 nodes at a time) for ElasticSearch cluster relforge: relforge plugin upgrade - ryankemper@cumin1001 - T309648 [production]
18:57 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
18:56 <dduvall@deploy1002> rebuilt and synchronized wikiversions files: all wikis to 1.39.0-wmf.15 refs T308068 [production]
18:54 <ryankemper@cumin1001> START - Cookbook sre.elasticsearch.rolling-operation Operation.UPGRADE (1 nodes at a time) for ElasticSearch cluster relforge: relforge plugin upgrade - ryankemper@cumin1001 - T309648 [production]
18:53 <ryankemper> T309648 Copied newly built `wmf-elasticsearch-search-plugins` from stretch to bullseye (`root@apt1001:/home/ryankemper# reprepro copy bullseye-wikimedia stretch-wikimedia wmf-elasticsearch-search-plugins`); then ran `apt update` on `relforge*`; new plugin package showing as available now: `6.8.23-3~stretch 1001` [production]
18:52 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
18:46 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
18:46 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
18:39 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
18:35 <dduvall@deploy1002> Synchronized php: group1 wikis to 1.39.0-wmf.15 refs T308068 (duration: 03m 34s) [production]
18:34 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
18:31 <dduvall@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.39.0-wmf.15 refs T308068 [production]
18:28 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
18:28 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
18:26 <dduvall@deploy1002> Finished scap: Backport for [[gerrit:803922]] Truncate failed requests errors to 4kB (duration: 04m 08s) [production]
18:22 <dduvall@deploy1002> Started scap: Backport for [[gerrit:803922]] Truncate failed requests errors to 4kB [production]
18:21 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
18:10 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1064.eqiad.wmnet with OS bullseye [production]
18:04 <dduvall@deploy1002> backport aborted: (duration: 00m 08s) [production]
17:56 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
17:55 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
17:55 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
17:54 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
17:53 <dduvall@deploy1002> rebuilt and synchronized wikiversions files: group0 wikis to 1.39.0-wmf.15 refs T308068 [production]
17:49 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1064.eqiad.wmnet with reason: host reimage [production]
17:48 <bking@cumin1001> START - Cookbook sre.hosts.reimage for host elastic2053.codfw.wmnet with OS bullseye [production]