3501-3550 of 10000 results (98ms)
2023-12-04 §
07:39 <arnaudb@cumin1001> dbctl commit (dc=all): 'Depooling db1129 (T348183)', diff saved to https://phabricator.wikimedia.org/P54062 and previous config saved to /var/cache/conftool/dbconfig/20231204-073957-arnaudb.json [production]
07:39 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1129.eqiad.wmnet with reason: Maintenance [production]
07:39 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1129.eqiad.wmnet with reason: Maintenance [production]
07:31 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1176.eqiad.wmnet with OS bookworm [production]
07:18 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1176.eqiad.wmnet with reason: host reimage [production]
07:15 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on db1176.eqiad.wmnet with reason: host reimage [production]
07:07 <kart_> Updated MinT to 2023-11-21-115852-production [production]
07:03 <marostegui@cumin1001> START - Cookbook sre.hosts.reimage for host db1176.eqiad.wmnet with OS bookworm [production]
06:57 <marostegui> Failover m5 from db1176 to db1119 - T332155 [production]
06:49 <kartik@deploy2002> helmfile [codfw] DONE helmfile.d/services/machinetranslation: apply [production]
06:47 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2135,2160].codfw.wmnet,db[1119,1176,1217].eqiad.wmnet with reason: m5 master switch T352505 [production]
06:46 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on db[2135,2160].codfw.wmnet,db[1119,1176,1217].eqiad.wmnet with reason: m5 master switch T352505 [production]
06:44 <kartik@deploy2002> helmfile [codfw] START helmfile.d/services/machinetranslation: apply [production]
06:33 <kartik@deploy2002> helmfile [eqiad] DONE helmfile.d/services/machinetranslation: apply [production]
06:28 <kartik@deploy2002> helmfile [eqiad] START helmfile.d/services/machinetranslation: apply [production]
06:14 <kartik@deploy2002> helmfile [staging] DONE helmfile.d/services/machinetranslation: apply [production]
06:11 <kartik@deploy2002> helmfile [staging] START helmfile.d/services/machinetranslation: apply [production]
06:08 <kart_> Updated cxserver to 2023-12-04-055024-production (T270060, T350773, T352620) [production]
06:06 <kartik@deploy2002> helmfile [codfw] DONE helmfile.d/services/cxserver: apply [production]
06:05 <kartik@deploy2002> helmfile [codfw] START helmfile.d/services/cxserver: apply [production]
06:03 <kartik@deploy2002> helmfile [eqiad] DONE helmfile.d/services/cxserver: apply [production]
06:02 <kartik@deploy2002> helmfile [eqiad] START helmfile.d/services/cxserver: apply [production]
05:59 <kartik@deploy2002> helmfile [staging] DONE helmfile.d/services/cxserver: apply [production]
05:58 <kartik@deploy2002> helmfile [staging] START helmfile.d/services/cxserver: apply [production]
04:43 <ryankemper> [WDQS] Clearing `BlazegraphFreeAllocatorsDecreasingRapidly` -> `ryankemper@wdqs1007:~$ sudo systemctl restart wdqs-blazegraph` [production]
00:16 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcontrol1006.eqiad.wmnet [production]
00:09 <andrew@cumin1001> START - Cookbook sre.hosts.reboot-single for host cloudcontrol1006.eqiad.wmnet [production]
2023-12-02 §
01:51 <jclark@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-be1078.eqiad.wmnet with OS bullseye [production]
01:50 <jclark@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-be1079.eqiad.wmnet with OS bullseye [production]
01:50 <jclark@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-be1077.eqiad.wmnet with OS bullseye [production]
01:50 <jclark@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-be1076.eqiad.wmnet with OS bullseye [production]
00:30 <jclark@cumin1001> START - Cookbook sre.hosts.reimage for host ms-be1078.eqiad.wmnet with OS bullseye [production]
00:30 <jclark@cumin1001> START - Cookbook sre.hosts.reimage for host ms-be1079.eqiad.wmnet with OS bullseye [production]
00:30 <jclark@cumin1001> START - Cookbook sre.hosts.reimage for host ms-be1077.eqiad.wmnet with OS bullseye [production]
00:30 <jclark@cumin1001> START - Cookbook sre.hosts.reimage for host ms-be1076.eqiad.wmnet with OS bullseye [production]
00:15 <jclark@cumin1001> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ms-be1076'] [production]
00:15 <jclark@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ms-be1076'] [production]
00:14 <jclark@cumin1001> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ms-be1076'] [production]
00:14 <jclark@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ms-be1076'] [production]
00:14 <jclark@cumin1001> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ms-be1076'] [production]
00:14 <jclark@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ms-be1076'] [production]
00:14 <jclark@cumin1001> END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts ['ms-be1076'] [production]
00:14 <jclark@cumin1001> END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts ['ms-be1076'] [production]
00:14 <jclark@cumin1001> END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts ['ms-be1076'] [production]
00:14 <jclark@cumin1001> END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts ['ms-be1076'] [production]
00:14 <jclark@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ms-be1076'] [production]
00:14 <jclark@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ms-be1076'] [production]
00:13 <jclark@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ms-be1076'] [production]
00:13 <jclark@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ms-be1076'] [production]
00:13 <jclark@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1079.mgmt.eqiad.wmnet with reboot policy FORCED [production]