5501-5550 of 10000 results (35ms)
2022-03-23 §
08:10 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on kubernetes1009.eqiad.wmnet with reason: host reimage [production]
07:54 <elukey@cumin1001> START - Cookbook sre.hosts.reimage for host kubernetes1009.eqiad.wmnet with OS bullseye [production]
2022-03-22 §
08:49 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kubernetes1008.eqiad.wmnet with OS bullseye [production]
08:37 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubernetes1008.eqiad.wmnet with reason: host reimage [production]
08:35 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on kubernetes1008.eqiad.wmnet with reason: host reimage [production]
08:19 <elukey@cumin1001> START - Cookbook sre.hosts.reimage for host kubernetes1008.eqiad.wmnet with OS bullseye [production]
07:53 <elukey> restart php-fpm on mw1449 - opcache full after deployment [production]
07:49 <elukey> restart php-fpm on mw1448 - high cpu usage right after yesterday's deployment at 21 UTC [production]
07:47 <elukey> depool mw1448 manually on the node (high cpu usage from php-fpm) [production]
07:21 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kubernetes1007.eqiad.wmnet with OS bullseye [production]
07:08 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubernetes1007.eqiad.wmnet with reason: host reimage [production]
07:06 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on kubernetes1007.eqiad.wmnet with reason: host reimage [production]
06:54 <elukey@cumin1001> START - Cookbook sre.hosts.reimage for host kubernetes1007.eqiad.wmnet with OS bullseye [production]
2022-03-17 §
06:50 <elukey> restart blazegraph on wdqs2004 [production]
06:46 <elukey> kill remaining hanging processes for ppche*lko and accra*ze on an-test-client1001 to allow users offboard (puppet broken) [production]
2022-03-15 §
14:01 <elukey@deploy1002> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . [production]
07:43 <elukey> restart kube-api server on ml-serve-ctrl2002 - 504 responses registered, corresponding to high custom resource definition requests [production]
2022-03-14 §
09:17 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kubernetes2017.codfw.wmnet with OS bullseye [production]
09:04 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubernetes2017.codfw.wmnet with reason: host reimage [production]
09:01 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on kubernetes2017.codfw.wmnet with reason: host reimage [production]
08:46 <elukey@cumin1001> START - Cookbook sre.hosts.reimage for host kubernetes2017.codfw.wmnet with OS bullseye [production]
07:18 <elukey> restart varnishkafka-webrequest on cp6001 to test a metric issue [production]
2022-03-11 §
15:56 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kubernetes2014.codfw.wmnet with OS bullseye [production]
15:44 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubernetes2014.codfw.wmnet with reason: host reimage [production]
15:42 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on kubernetes2014.codfw.wmnet with reason: host reimage [production]
15:27 <elukey@cumin1001> START - Cookbook sre.hosts.reimage for host kubernetes2014.codfw.wmnet with OS bullseye [production]
15:07 <elukey@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host kubernetes2013.codfw.wmnet with OS bullseye [production]
14:54 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubernetes2013.codfw.wmnet with reason: host reimage [production]
14:51 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on kubernetes2013.codfw.wmnet with reason: host reimage [production]
14:35 <elukey@cumin1001> START - Cookbook sre.hosts.reimage for host kubernetes2013.codfw.wmnet with OS bullseye [production]
11:18 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kubernetes2012.codfw.wmnet with OS bullseye [production]
11:05 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubernetes2012.codfw.wmnet with reason: host reimage [production]
11:02 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on kubernetes2012.codfw.wmnet with reason: host reimage [production]
10:59 <elukey@cumin1001> END (PASS) - Cookbook sre.ores.roll-restart-workers (exit_code=0) for ORES codfw cluster: Roll restart of ORES's daemons. [production]
10:46 <elukey@cumin1001> START - Cookbook sre.hosts.reimage for host kubernetes2012.codfw.wmnet with OS bullseye [production]
10:40 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kubernetes2011.codfw.wmnet with OS bullseye [production]
10:39 <elukey@cumin1001> START - Cookbook sre.ores.roll-restart-workers for ORES codfw cluster: Roll restart of ORES's daemons. [production]
10:35 <elukey@cumin1001> END (PASS) - Cookbook sre.ores.roll-restart-workers (exit_code=0) for ORES eqiad cluster: Roll restart of ORES's daemons. [production]
10:28 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubernetes2011.codfw.wmnet with reason: host reimage [production]
10:24 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on kubernetes2011.codfw.wmnet with reason: host reimage [production]
10:16 <elukey@cumin1001> START - Cookbook sre.ores.roll-restart-workers for ORES eqiad cluster: Roll restart of ORES's daemons. [production]
10:09 <elukey@cumin1001> START - Cookbook sre.hosts.reimage for host kubernetes2011.codfw.wmnet with OS bullseye [production]
2022-03-10 §
18:13 <elukey@deploy1002> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. [production]
18:13 <elukey@deploy1002> helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. [production]
10:42 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kubernetes2010.codfw.wmnet with OS bullseye [production]
10:29 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubernetes2010.codfw.wmnet with reason: host reimage [production]
10:26 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on kubernetes2010.codfw.wmnet with reason: host reimage [production]
10:10 <elukey@cumin1001> START - Cookbook sre.hosts.reimage for host kubernetes2010.codfw.wmnet with OS bullseye [production]
09:53 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kubernetes2009.codfw.wmnet with OS bullseye [production]
09:40 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubernetes2009.codfw.wmnet with reason: host reimage [production]