3051-3100 of 10000 results (33ms)
2023-06-15 §
10:51 <elukey@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . [production]
10:30 <elukey@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . [production]
10:09 <elukey@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . [production]
10:02 <elukey@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . [production]
09:54 <elukey@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . [production]
09:05 <elukey> move varnishkafka instances in ulsfo to PKI - T337825 [production]
08:52 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1492.eqiad.wmnet with OS buster [production]
08:52 <elukey@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1001" [production]
07:55 <elukey@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1001" [production]
07:34 <elukey@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . [production]
07:27 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1492.eqiad.wmnet with reason: host reimage [production]
07:24 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mw1492.eqiad.wmnet with reason: host reimage [production]
07:11 <elukey@cumin1001> START - Cookbook sre.hosts.reimage for host mw1492.eqiad.wmnet with OS buster [production]
2023-06-14 §
09:54 <elukey@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . [production]
2023-06-13 §
15:00 <elukey> run kafka re-assign partitions for eqiad.change-prop.transcludes.resource-change on kafka-main1001 - T338357 [production]
07:10 <elukey> move varnishkafka instances on cp4037 to PKI TLS certs - T337825 [production]
07:08 <elukey@puppetmaster1001> conftool action : set/pooled=yes; selector: name=cp4037.ulsfo.wmnet [production]
06:56 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on cp4037.ulsfo.wmnet with reason: Working on vk [production]
06:55 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 0:30:00 on cp4037.ulsfo.wmnet with reason: Working on vk [production]
06:55 <elukey@puppetmaster1001> conftool action : set/pooled=no; selector: name=cp4037.ulsfo.wmnet [production]
2023-06-12 §
09:48 <elukey@puppetmaster1001> conftool action : set/pooled=yes; selector: name=cp4037.ulsfo.wmnet [production]
09:26 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on cp4037.ulsfo.wmnet with reason: Working on vk [production]
09:26 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 0:30:00 on cp4037.ulsfo.wmnet with reason: Working on vk [production]
08:56 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on cp4037.ulsfo.wmnet with reason: Working on vk [production]
08:56 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 0:30:00 on cp4037.ulsfo.wmnet with reason: Working on vk [production]
08:30 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on cp4037.ulsfo.wmnet with reason: Working on vk [production]
08:30 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 0:30:00 on cp4037.ulsfo.wmnet with reason: Working on vk [production]
08:30 <elukey@puppetmaster1001> conftool action : set/pooled=no; selector: name=cp4037.ulsfo.wmnet [production]
2023-06-09 §
14:14 <elukey@puppetmaster1001> conftool action : set/pooled=yes; selector: name=cp4037.ulsfo.wmnet [production]
13:29 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on cp4037.ulsfo.wmnet with reason: Working on vk [production]
13:28 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 0:30:00 on cp4037.ulsfo.wmnet with reason: Working on vk [production]
13:25 <elukey@puppetmaster1001> conftool action : set/pooled=no; selector: name=cp4037.ulsfo.wmnet [production]
10:12 <elukey@deploy1002> helmfile [codfw] DONE helmfile.d/services/changeprop: sync [production]
10:12 <elukey@deploy1002> helmfile [codfw] START helmfile.d/services/changeprop: sync [production]
10:09 <elukey@deploy1002> helmfile [eqiad] DONE helmfile.d/services/changeprop: sync [production]
10:08 <elukey@deploy1002> helmfile [eqiad] START helmfile.d/services/changeprop: sync [production]
09:57 <elukey> increase {eqiad,codfw}.change-prop.transcludes.resource-change topic partitions (3->5) on kafka main clusters - T338357 [production]
2023-06-08 §
16:00 <elukey@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . [production]
07:14 <elukey> delete pod kask-production-7dfdfc7cbc-2vw5q in wikikube codfw, since it was scheduled on a non dedicated node [production]
06:10 <elukey> kill remaining processes for `andyrussg` on stat100x nodes to unblock puppet [production]
2023-06-07 §
15:23 <elukey> all varnishkafka instances on caching nodes are getting restarted due to https://gerrit.wikimedia.org/r/c/operations/puppet/+/928087 - T337825 [production]
15:22 <elukey> re-enable puppet on caching nodes [production]
15:10 <elukey> disable puppet on all caching nodes to rollout a varnishakfka change (ref: https://gerrit.wikimedia.org/r/c/operations/puppet/+/928087) [production]
2023-06-06 §
13:41 <elukey@deploy1002> helmfile [staging] DONE helmfile.d/services/changeprop: sync [production]
13:41 <elukey@deploy1002> helmfile [staging] START helmfile.d/services/changeprop: sync [production]
2023-06-05 §
16:08 <elukey@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . [production]
16:06 <elukey@deploy1002> helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. [production]
16:06 <elukey@deploy1002> helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. [production]
16:06 <elukey@deploy1002> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. [production]
16:05 <elukey@deploy1002> helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. [production]