7251-7300 of 8776 results (25ms)
2023-05-23 §
18:29 <inflatador> bking@cumin1001 rolling restart of codfw wdqs public hosts T337327 [production]
2023-05-22 §
20:11 <bking@cumin1001> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs[2010-2011].codfw.wmnet [production]
20:09 <bking@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs[2010-2011].codfw.wmnet [production]
16:35 <bking@cumin1001> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs2009.codfw.wmnet [production]
16:35 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs2009.codfw.wmnet [production]
15:57 <bking@cumin1001> START - Cookbook sre.hosts.reboot-single for host wdqs2009.codfw.wmnet [production]
15:56 <bking@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs2009.codfw.wmnet [production]
14:32 <bking@deploy1002> helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply [production]
14:31 <bking@deploy1002> helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply [production]
14:10 <bking@deploy1002> helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply [production]
14:10 <bking@deploy1002> helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply [production]
2023-05-19 §
14:17 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on wdqs1014.eqiad.wmnet with reason: firmware update [production]
14:16 <bking@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on wdqs1014.eqiad.wmnet with reason: firmware update [production]
2023-05-18 §
22:59 <bking@cumin1001> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.UPGRADE (3 nodes at a time) for ElasticSearch cluster search_eqiad: eqiad plugin upgrade - bking@cumin1001 - T332355 [production]
20:36 <bking@cumin1001> START - Cookbook sre.elasticsearch.rolling-operation Operation.UPGRADE (3 nodes at a time) for ElasticSearch cluster search_eqiad: eqiad plugin upgrade - bking@cumin1001 - T332355 [production]
20:33 <bking@cumin1001> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.UPGRADE (3 nodes at a time) for ElasticSearch cluster search_codfw: codfw plugin upgrade - bking@cumin1001 - T332355 [production]
18:55 <bking@cumin1001> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.UPGRADE (2 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic elasticsearch and plugin upgrade - bking@cumin1001 - T332355 [production]
18:11 <bking@cumin1001> START - Cookbook sre.elasticsearch.rolling-operation Operation.UPGRADE (3 nodes at a time) for ElasticSearch cluster search_codfw: codfw plugin upgrade - bking@cumin1001 - T332355 [production]
18:09 <bking@cumin1001> START - Cookbook sre.elasticsearch.rolling-operation Operation.UPGRADE (2 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic elasticsearch and plugin upgrade - bking@cumin1001 - T332355 [production]
18:07 <bking@cumin1001> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.UPGRADE (1 nodes at a time) for ElasticSearch cluster relforge: relforge elasticsearch and plugin upgrade - bking@cumin1001 - T274204 [production]
17:59 <bking@cumin1001> START - Cookbook sre.elasticsearch.rolling-operation Operation.UPGRADE (1 nodes at a time) for ElasticSearch cluster relforge: relforge elasticsearch and plugin upgrade - bking@cumin1001 - T274204 [production]
15:57 <inflatador> bking@cumin1001 starting rolling restart of wcqs for java updates T334470 [production]
2023-05-17 §
21:44 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs2012.codfw.wmnet [production]
21:26 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12 days, 0:00:00 on wdqs2012.codfw.wmnet with reason: attempting WDQS stack on bullseye [production]
21:26 <bking@cumin1001> START - Cookbook sre.hosts.downtime for 12 days, 0:00:00 on wdqs2012.codfw.wmnet with reason: attempting WDQS stack on bullseye [production]
20:59 <bking@cumin1001> START - Cookbook sre.hosts.reboot-single for host wdqs2012.codfw.wmnet [production]
20:13 <bking@cumin1001> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs2012.codfw.wmnet [production]
20:12 <bking@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs2012.codfw.wmnet [production]
20:07 <bking@cumin1001> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs2012.codfw.wmnet [production]
19:54 <bking@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs2012.codfw.wmnet [production]
19:41 <inflatador> bking@wdqs2012 depooling to attempt firmware update T331297 [production]
14:10 <bking@deploy1002> helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply [production]
14:09 <bking@deploy1002> helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply [production]
14:09 <bking@deploy1002> helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply [production]
14:08 <bking@deploy1002> helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply [production]
2023-05-16 §
14:55 <bking@deploy1002> helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply [production]
14:47 <bking@deploy1002> helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply [production]
14:36 <bking@deploy1002> helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply [production]
14:26 <bking@deploy1002> helmfile [eqiad] START helmfile.d/services/rdf-streaming-updater: apply [production]
2023-05-15 §
19:50 <bking@cumin1001> END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Banning hosts: elastic[2050-2054,2060,2067-2068,2072,2084-2086]* for row D switch upgrade - bking@cumin1001 - T335042 [production]
19:50 <bking@cumin1001> START - Cookbook sre.elasticsearch.ban Banning hosts: elastic[2050-2054,2060,2067-2068,2072,2084-2086]* for row D switch upgrade - bking@cumin1001 - T335042 [production]
19:49 <bking@cumin1001> END (FAIL) - Cookbook sre.elasticsearch.ban (exit_code=99) Banning hosts: elastic[2050-2054,2060,2067-2068,2072,2084-2086] for row D switch upgrade - bking@cumin1001 - T335042 [production]
19:49 <bking@cumin1001> START - Cookbook sre.elasticsearch.ban Banning hosts: elastic[2050-2054,2060,2067-2068,2072,2084-2086] for row D switch upgrade - bking@cumin1001 - T335042 [production]
19:28 <bking@deploy1002> Finished deploy [wdqs/wdqs@41174d5] (wcqs): deploy 0.3.124 to WCQS (duration: 02m 03s) [production]
19:26 <bking@deploy1002> Started deploy [wdqs/wdqs@41174d5] (wcqs): deploy 0.3.124 to WCQS [production]
19:19 <bking@deploy1002> Finished deploy [wdqs/wdqs@41174d5]: (no justification provided) (duration: 00m 05s) [production]
19:19 <bking@deploy1002> Started deploy [wdqs/wdqs@41174d5]: (no justification provided) [production]
19:18 <bking@deploy1002> Finished deploy [wdqs/wdqs@41174d5]: (no justification provided) (duration: 00m 05s) [production]
19:18 <bking@deploy1002> Started deploy [wdqs/wdqs@41174d5]: (no justification provided) [production]
19:18 <bking@deploy1002> Finished deploy [wdqs/wdqs@41174d5]: (no justification provided) (duration: 05m 46s) [production]