3001-3050 of 10000 results (29ms)
2023-06-29 §
12:48 <elukey@deploy1002> helmfile [ml-serve-codfw] 'sync' command on namespace 'ores-legacy' for release 'main' . [production]
12:46 <elukey@deploy1002> helmfile [ml-serve-eqiad] 'sync' command on namespace 'ores-legacy' for release 'main' . [production]
12:46 <elukey@deploy1002> helmfile [ml-staging-codfw] 'sync' command on namespace 'ores-legacy' for release 'main' . [production]
2023-06-28 §
15:54 <elukey@deploy1002> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . [production]
15:53 <elukey@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . [production]
15:53 <elukey@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . [production]
15:53 <elukey@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . [production]
15:53 <elukey@deploy1002> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . [production]
15:52 <elukey@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . [production]
11:04 <elukey@cumin1001> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:ml-cache-codfw: Roll restart to pick up Java 11 - elukey@cumin1001 [production]
10:47 <elukey@cumin1001> START - Cookbook sre.cassandra.roll-restart for nodes matching A:ml-cache-codfw: Roll restart to pick up Java 11 - elukey@cumin1001 [production]
10:47 <elukey@cumin1001> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:ml-cache-eqiad: Roll restart to pick up Java 11 - elukey@cumin1001 [production]
10:42 <elukey@deploy1002> helmfile [ml-serve-eqiad] 'sync' command on namespace 'ores-legacy' for release 'main' . [production]
10:42 <elukey@deploy1002> helmfile [ml-serve-codfw] 'sync' command on namespace 'ores-legacy' for release 'main' . [production]
10:41 <elukey@deploy1002> helmfile [ml-staging-codfw] 'sync' command on namespace 'ores-legacy' for release 'main' . [production]
10:31 <elukey@deploy1002> helmfile [ml-staging-codfw] 'sync' command on namespace 'ores-legacy' for release 'main' . [production]
10:29 <elukey@cumin1001> START - Cookbook sre.cassandra.roll-restart for nodes matching A:ml-cache-eqiad: Roll restart to pick up Java 11 - elukey@cumin1001 [production]
10:05 <elukey@deploy1002> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. [production]
10:02 <elukey@deploy1002> helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. [production]
10:01 <elukey@deploy1002> helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. [production]
09:57 <elukey@deploy1002> helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. [production]
09:57 <elukey@deploy1002> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. [production]
09:55 <elukey@deploy1002> helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. [production]
2023-06-27 §
14:41 <elukey@cumin1001> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:ml-cache-eqiad: Roll restart to pick up new certs and openjdk version - elukey@cumin1001 [production]
14:23 <elukey@cumin1001> START - Cookbook sre.cassandra.roll-restart for nodes matching A:ml-cache-eqiad: Roll restart to pick up new certs and openjdk version - elukey@cumin1001 [production]
14:21 <elukey@cumin1001> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:ml-cache-codfw: Roll restart to pick up new certs and openjdk version - elukey@cumin1001 [production]
14:04 <elukey@cumin1001> START - Cookbook sre.cassandra.roll-restart for nodes matching A:ml-cache-codfw: Roll restart to pick up new certs and openjdk version - elukey@cumin1001 [production]
13:32 <elukey> expand ml-staging200[12] kubelet partitions - T339231 [production]
12:58 <elukey@deploy1002> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. [production]
12:57 <elukey@deploy1002> helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. [production]
10:48 <elukey@cumin1001> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:ml-cache-eqiad: Roll restart to pick up new certs and openjdk version - elukey@cumin1001 [production]
10:30 <elukey@cumin1001> START - Cookbook sre.cassandra.roll-restart for nodes matching A:ml-cache-eqiad: Roll restart to pick up new certs and openjdk version - elukey@cumin1001 [production]
08:38 <elukey> revoked puppet cert for 'varnishkafka' and cleaned up its cergen's files in puppet private - T337825 [production]
07:15 <elukey> `sudo kill `pgrep -u paramd`` on stat1005 to unblock puppet [production]
2023-06-26 §
14:06 <elukey> move varnishkafka instances in esams to pki [production]
2023-06-23 §
12:40 <elukey> move varnishkafka drmrs instances to pki [production]
08:48 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on ml-cache1001.eqiad.wmnet with reason: Working on pki [production]
08:48 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 0:30:00 on ml-cache1001.eqiad.wmnet with reason: Working on pki [production]
2023-06-22 §
13:17 <elukey> move varnishafka instances in eqiad to PKI [production]
2023-06-21 §
12:51 <elukey> move varnishafka instances in codfw to PKI [production]
2023-06-19 §
15:50 <elukey@cumin1001> END (ERROR) - Cookbook sre.cassandra.roll-restart (exit_code=97) for nodes matching A:ml-cache-codfw: Applying internode-encryption: all - elukey@cumin1001 [production]
15:47 <elukey@cumin1001> START - Cookbook sre.cassandra.roll-restart for nodes matching A:ml-cache-codfw: Applying internode-encryption: all - elukey@cumin1001 [production]
14:04 <elukey> move varnishafka instances in eqsin to PKI [production]
2023-06-16 §
15:58 <elukey@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . [production]
2023-06-15 §
13:28 <elukey@cumin1001> END (PASS) - Cookbook sre.ores.roll-restart-workers (exit_code=0) for ORES codfw cluster: Roll restart of ORES's daemons. [production]
13:08 <elukey@cumin1001> START - Cookbook sre.ores.roll-restart-workers for ORES codfw cluster: Roll restart of ORES's daemons. [production]
13:05 <elukey@cumin1001> END (PASS) - Cookbook sre.ores.roll-restart-workers (exit_code=0) for ORES eqiad cluster: Roll restart of ORES's daemons. [production]
12:45 <elukey@cumin1001> START - Cookbook sre.ores.roll-restart-workers for ORES eqiad cluster: Roll restart of ORES's daemons. [production]
10:58 <elukey@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . [production]
10:54 <elukey@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . [production]