3901-3950 of 10000 results (91ms)
2023-08-28 §
05:13 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1192.eqiad.wmnet with reason: Maintenance [production]
05:12 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db1178 (re)pooling @ 10%: Maint over', diff saved to https://phabricator.wikimedia.org/P51434 and previous config saved to /var/cache/conftool/dbconfig/20230828-051237-ladsgroup.json [production]
2023-08-27 §
07:28 <elukey> silence rdb1011:6380's Redis alert (ORES-related) for 30 days to avoid spam [production]
2023-08-26 §
13:07 <elukey> silence rdb1011:6378's Redis alert (ORES-related) for 30 days to avoid spam [production]
2023-08-25 §
21:03 <inflatador> bking@cumin1001 shutting off wdqs1005 in preparation for decommission T344198 [production]
21:02 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on wdqs1005.eqiad.wmnet with reason: to be decommissioned soon [production]
21:02 <bking@cumin1001> START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on wdqs1005.eqiad.wmnet with reason: to be decommissioned soon [production]
19:48 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host moss-be2003.codfw.wmnet with OS bullseye [production]
19:39 <bking@cumin1001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
18:55 <jhancock@cumin2002> START - Cookbook sre.hosts.reimage for host moss-be2003.codfw.wmnet with OS bullseye [production]
18:48 <jforrester@deploy1002> helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply [production]
18:47 <jforrester@deploy1002> helmfile [eqiad] START helmfile.d/services/wikifunctions: apply [production]
18:47 <jforrester@deploy1002> helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply [production]
18:46 <jforrester@deploy1002> helmfile [codfw] START helmfile.d/services/wikifunctions: apply [production]
18:45 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host moss-be2003.mgmt.codfw.wmnet with reboot policy FORCED [production]
18:45 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host moss-be2003.mgmt.codfw.wmnet with reboot policy FORCED [production]
18:45 <jforrester@deploy1002> helmfile [staging] DONE helmfile.d/services/wikifunctions: apply [production]
18:45 <jforrester@deploy1002> helmfile [staging] START helmfile.d/services/wikifunctions: apply [production]
18:26 <jforrester@deploy1002> helmfile [staging] DONE helmfile.d/services/wikifunctions: apply [production]
18:25 <jforrester@deploy1002> helmfile [staging] START helmfile.d/services/wikifunctions: apply [production]
18:23 <jforrester@deploy1002> helmfile [staging] DONE helmfile.d/services/wikifunctions: apply [production]
18:23 <jforrester@deploy1002> helmfile [staging] START helmfile.d/services/wikifunctions: apply [production]
18:22 <jforrester@deploy1002> helmfile [staging] DONE helmfile.d/services/wikifunctions: apply [production]
18:22 <jforrester@deploy1002> helmfile [staging] START helmfile.d/services/wikifunctions: apply [production]
18:15 <jforrester@deploy1002> helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply [production]
18:14 <jforrester@deploy1002> helmfile [eqiad] START helmfile.d/services/wikifunctions: apply [production]
18:14 <jforrester@deploy1002> helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply [production]
18:13 <jforrester@deploy1002> helmfile [codfw] START helmfile.d/services/wikifunctions: apply [production]
18:13 <jforrester@deploy1002> helmfile [staging] DONE helmfile.d/services/wikifunctions: apply [production]
18:12 <jforrester@deploy1002> helmfile [staging] START helmfile.d/services/wikifunctions: apply [production]
18:12 <jforrester@deploy1002> helmfile [staging] DONE helmfile.d/services/wikifunctions: apply [production]
18:12 <jforrester@deploy1002> helmfile [staging] START helmfile.d/services/wikifunctions: apply [production]
18:01 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host moss-be2003.mgmt.codfw.wmnet with reboot policy FORCED [production]
18:01 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host moss-be2003.mgmt.codfw.wmnet with reboot policy FORCED [production]
17:47 <jhancock@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['moss-be2003'] [production]
17:46 <jhancock@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['moss-be2003'] [production]
17:46 <jforrester@deploy1002> helmfile [eqiad] DONE helmfile.d/services/wikifunctions: sync [production]
17:45 <bking@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
17:45 <jforrester@deploy1002> helmfile [eqiad] START helmfile.d/services/wikifunctions: sync [production]
17:45 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['moss-be2003'] [production]
17:45 <jforrester@deploy1002> helmfile [codfw] DONE helmfile.d/services/wikifunctions: sync [production]
17:44 <jforrester@deploy1002> helmfile [codfw] START helmfile.d/services/wikifunctions: sync [production]
17:44 <jforrester@deploy1002> helmfile [staging] DONE helmfile.d/services/wikifunctions: sync [production]
17:44 <jforrester@deploy1002> helmfile [staging] START helmfile.d/services/wikifunctions: sync [production]
17:43 <jforrester@deploy1002> helmfile [staging] DONE helmfile.d/services/wikifunctions: apply [production]
17:43 <jforrester@deploy1002> helmfile [staging] START helmfile.d/services/wikifunctions: apply [production]
17:41 <bking@cumin1001> END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) [production]
17:41 <bking@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
17:39 <bking@deploy1002> Finished deploy [wdqs/wdqs@16e3dcf]: push deploy after bullseye reimage T343124 (duration: 00m 19s) [production]
17:39 <bking@deploy1002> Started deploy [wdqs/wdqs@16e3dcf]: push deploy after bullseye reimage T343124 [production]