8501-8550 of 10000 results (87ms)
2022-06-03 §
01:20 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance [production]
01:20 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1121.eqiad.wmnet with reason: Maintenance [production]
01:20 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1121.eqiad.wmnet with reason: Maintenance [production]
01:12 <bking@cumin1001> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: restart to enable S3 plugin - bking@cumin1001 - T309720 [production]
00:36 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host clouddumps1001.wikimedia.org with OS bullseye [production]
2022-06-02 §
23:58 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host clouddumps1001.wikimedia.org with OS bullseye [production]
23:56 <andrew@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host clouddumps1001.wikimedia.org with OS bullseye [production]
23:45 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on clouddumps1001.wikimedia.org with reason: host reimage [production]
23:42 <andrew@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on clouddumps1001.wikimedia.org with reason: host reimage [production]
23:30 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host clouddumps1001.wikimedia.org with OS bullseye [production]
23:27 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
23:27 <tstarling@deploy1002> Synchronized wmf-config/CommonSettings.php: Add db-mainstash g 752807 (duration: 03m 24s) [production]
23:26 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
23:26 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
23:25 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
23:22 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host clouddumps1001.wikimedia.org with OS bullseye [production]
22:53 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host clouddumps1001.wikimedia.org with OS bullseye [production]
22:50 <andrew@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host clouddumps1001.wikimedia.org with OS bullseye [production]
22:43 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host clouddumps1001.wikimedia.org with OS bullseye [production]
22:33 <andrew@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host clouddumps1001.wikimedia.org with OS bullseye [production]
22:31 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host clouddumps1001.wikimedia.org with OS bullseye [production]
22:31 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host clouddumps1001.wikimedia.org with OS bullseye [production]
22:27 <ejegg> updated payments-wiki from 4e9470de to 8c6208c2 [production]
22:23 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1118 (T298560)', diff saved to https://phabricator.wikimedia.org/P29363 and previous config saved to /var/cache/conftool/dbconfig/20220602-222306-ladsgroup.json [production]
22:23 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1118.eqiad.wmnet with reason: Maintenance [production]
22:23 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1118.eqiad.wmnet with reason: Maintenance [production]
22:08 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host clouddumps1001.wikimedia.org with OS bullseye [production]
22:08 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host clouddumps1001.wikimedia.org with OS bullseye [production]
22:08 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host clouddumps1001.wikimedia.org with OS bullseye [production]
22:08 <andrew@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host clouddumps1001.wikimedia.org with OS bullseye [production]
21:54 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host clouddumps1001.wikimedia.org with OS bullseye [production]
21:29 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
21:28 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
21:28 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
21:27 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host clouddumps1001.wikimedia.org with OS bullseye [production]
21:27 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
21:25 <jforrester@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Emergency deploy: [[gerrit:802637|Stop writing to cuc_actor on all wikis (T233004 T309737)]] (duration: 03m 15s) [production]
21:25 <cmjohnson@cumin1001> START - Cookbook sre.hosts.reimage for host backup1009.eqiad.wmnet with OS bullseye [production]
21:15 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on clouddumps1001.wikimedia.org with reason: host reimage [production]
21:11 <andrew@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on clouddumps1001.wikimedia.org with reason: host reimage [production]
20:59 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host clouddumps1001.wikimedia.org with OS bullseye [production]
20:45 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host clouddumps1001.wikimedia.org with OS bullseye [production]
20:26 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host backup1009.mgmt.eqiad.wmnet with reboot policy FORCED [production]
20:16 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host clouddumps1001.wikimedia.org with OS bullseye [production]
20:16 <ryankemper> T306449 Marked `elastic1097` as `Staged` in Netbox (was previously failed, but fixed in https://phabricator.wikimedia.org/T306449#7888260) [production]
20:14 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host clouddumps1001.wikimedia.org with OS bullseye [production]
20:14 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host clouddumps1001.wikimedia.org with OS bullseye [production]
20:14 <andrew@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host clouddumps1001.wikimedia.org with OS bullseye [production]
20:09 <cmjohnson@cumin1001> START - Cookbook sre.hosts.provision for host backup1009.mgmt.eqiad.wmnet with reboot policy FORCED [production]
20:08 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]