5901-5950 of 10000 results (31ms)
2021-12-17 §
15:35 <elukey@deploy1002> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . [production]
15:35 <elukey@deploy1002> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality' for release 'main' . [production]
15:35 <elukey@deploy1002> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . [production]
15:34 <elukey@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . [production]
15:34 <elukey@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality' for release 'main' . [production]
15:33 <elukey@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . [production]
14:52 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main2001.codfw.wmnet with OS buster [production]
14:22 <elukey@cumin1001> START - Cookbook sre.hosts.reimage for host kafka-main2001.codfw.wmnet with OS buster [production]
09:54 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main2002.codfw.wmnet with OS buster [production]
09:23 <elukey@cumin1001> START - Cookbook sre.hosts.reimage for host kafka-main2002.codfw.wmnet with OS buster [production]
2021-12-16 §
15:42 <elukey> shutdown kafka-main2002 for BIOS+NIC firmware upgrades [production]
14:55 <elukey> shutdown kafka-main2001 for BIOS+NIC firmware upgrades [production]
11:08 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main2003.codfw.wmnet with OS buster [production]
10:31 <elukey@cumin1001> START - Cookbook sre.hosts.reimage for host kafka-main2003.codfw.wmnet with OS buster [production]
10:28 <elukey> second attempt to reimage kafka-main2003 to buster [production]
2021-12-15 §
17:47 <elukey> kafka-main2003 up and running (dcops maintenance done) [production]
16:12 <elukey> shutdown kafka-main2003 to allow work for DCops (firmware upgrade) [production]
14:48 <elukey@deploy1002> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . [production]
14:47 <elukey@deploy1002> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality' for release 'main' . [production]
14:47 <elukey@deploy1002> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . [production]
14:46 <elukey@deploy1002> helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. [production]
14:46 <elukey@deploy1002> helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. [production]
14:45 <elukey@deploy1002> helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. [production]
14:44 <elukey@deploy1002> helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. [production]
14:44 <elukey@deploy1002> helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. [production]
14:44 <elukey@deploy1002> helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. [production]
10:27 <elukey@deploy1002> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. [production]
10:27 <elukey@deploy1002> helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. [production]
10:00 <elukey@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . [production]
09:57 <elukey@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . [production]
09:53 <elukey@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality' for release 'main' . [production]
09:43 <elukey@deploy1002> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. [production]
09:42 <elukey@deploy1002> helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. [production]
2021-12-13 §
17:37 <elukey@deploy1002> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. [production]
17:37 <elukey@deploy1002> helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. [production]
17:34 <elukey@deploy1002> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. [production]
17:34 <elukey@deploy1002> helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. [production]
09:31 <elukey@deploy1002> helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. [production]
09:31 <elukey@deploy1002> helmfile [staging-codfw] START helmfile.d/admin 'sync'. [production]
09:31 <elukey@deploy1002> helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. [production]
09:31 <elukey@deploy1002> helmfile [staging-codfw] START helmfile.d/admin 'sync'. [production]
09:29 <elukey@deploy1002> helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. [production]
09:28 <elukey@deploy1002> helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. [production]
09:25 <elukey@deploy1002> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. [production]
09:25 <elukey@deploy1002> helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. [production]
06:51 <elukey> run `apt-get clean` on aphlict1001 to free some space [production]
2021-12-09 §
18:15 <elukey> kafka-main2003 back in service with the old OS (stretch). Re-created a new puppet host key and signed it on the puppet master [production]
17:46 <elukey@cumin1001> END (FAIL) - Cookbook sre.puppet.renew-cert (exit_code=99) for kafka-main2003.codfw.wmnet: Renew puppet certificate - elukey@cumin1001 [production]
17:46 <elukey@cumin1001> START - Cookbook sre.puppet.renew-cert for kafka-main2003.codfw.wmnet: Renew puppet certificate - elukey@cumin1001 [production]
17:40 <elukey@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kafka-main2003.codfw.wmnet with OS buster [production]