2001-2050 of 10000 results (31ms)
2024-05-03 §
13:26 <elukey> restart karma on alert1001 to verify if probe down alerts shown are stale [production]
2024-05-02 §
15:34 <elukey@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching sessionstore1*: Move to PKI Truststore - elukey@cumin1002 [production]
15:15 <elukey@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching sessionstore1*: Move to PKI Truststore - elukey@cumin1002 [production]
15:12 <elukey@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching sessionstore200[5,6]*: Move to PKI Truststore - elukey@cumin1002 [production]
15:03 <elukey@puppetmaster1001> conftool action : set/pooled=true; selector: dnsdisc=inference,name=codfw [production]
15:00 <elukey@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching sessionstore200[5,6]*: Move to PKI Truststore - elukey@cumin1002 [production]
14:56 <elukey@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching sessionstore2004*: Move to PKI Truststore - elukey@cumin1002 [production]
14:50 <elukey@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching sessionstore2004*: Move to PKI Truststore - elukey@cumin1002 [production]
13:02 <elukey@deploy1002> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . [production]
12:57 <elukey@deploy1002> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . [production]
12:49 <elukey@deploy1002> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . [production]
12:22 <elukey@deploy1002> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'readability' for release 'main' . [production]
12:20 <elukey@deploy1002> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . [production]
12:15 <elukey@deploy1002> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'article-descriptions' for release 'main' . [production]
12:02 <elukey@deploy1002> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . [production]
11:57 <elukey@deploy1002> helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. [production]
11:55 <elukey@deploy1002> helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. [production]
11:54 <elukey@deploy1002> helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. [production]
11:53 <elukey@deploy1002> helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. [production]
11:43 <elukey> depool LiftWing's codfw services from traffic to move all MW API calls to mw-api-int-ro [production]
11:42 <elukey@puppetmaster1001> conftool action : set/pooled=false; selector: dnsdisc=inference,name=codfw [production]
2024-04-30 §
16:16 <elukey@puppetmaster1001> conftool action : set/pooled=true; selector: dnsdisc=inference,name=codfw [production]
16:16 <elukey@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching sessionstore1*.eqiad.wmnet: Move to PKI Truststore - elukey@cumin1002 [production]
15:58 <elukey@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching sessionstore1*.eqiad.wmnet: Move to PKI Truststore - elukey@cumin1002 [production]
15:57 <elukey@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching sessionstore200[5,6].codfw.wmnet: Move to PKI Truststore - elukey@cumin1002 [production]
15:44 <elukey@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching sessionstore200[5,6].codfw.wmnet: Move to PKI Truststore - elukey@cumin1002 [production]
15:40 <elukey@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching sessionstore2004.codfw.wmnet: Move to PKI Truststore - elukey@cumin1002 [production]
15:33 <elukey@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching sessionstore2004.codfw.wmnet: Move to PKI Truststore - elukey@cumin1002 [production]
15:28 <elukey> move Cassandra instances on session store nodes to a new Java Truststore to support PKI - T352647 [production]
15:27 <elukey> depool liftwing codfw for a couple of hours to test eqiad capabilities to handle the traffic [production]
15:27 <elukey@puppetmaster1001> conftool action : set/pooled=false; selector: dnsdisc=inference,name=codfw [production]
2024-04-29 §
15:25 <elukey@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching restbase1028.eqiad.wmnet: Move to PKI TLS certs - elukey@cumin1002 [production]
15:15 <elukey@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching restbase1028.eqiad.wmnet: Move to PKI TLS certs - elukey@cumin1002 [production]
14:54 <elukey@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching restbase20[23-35]*: Roll out PKI TLS certs - elukey@cumin1002 [production]
12:58 <elukey@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching restbase20[23-35]*: Roll out PKI TLS certs - elukey@cumin1002 [production]
12:53 <elukey@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching restbase2022.codfw.wmnet: Move to PKI TLS certs - elukey@cumin1002 [production]
2024-04-26 §
16:20 <elukey@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . [production]
16:19 <elukey@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . [production]
16:18 <elukey@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . [production]
16:18 <elukey@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . [production]
16:17 <elukey@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . [production]
16:17 <elukey@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . [production]
16:16 <elukey@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . [production]
14:48 <elukey@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching restbase2021.codfw.wmnet: Move to PKI TLS certs - elukey@cumin1002 [production]
14:38 <elukey@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching restbase2021.codfw.wmnet: Move to PKI TLS certs - elukey@cumin1002 [production]
13:25 <elukey@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . [production]
12:52 <elukey@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . [production]
12:44 <elukey@deploy1002> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. [production]
12:44 <elukey@deploy1002> helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. [production]
2024-04-24 §
16:43 <elukey@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:restbase-codfw: Deploy new TLS Truststore for PKI - elukey@cumin1002 [production]