6151-6200 of 10000 results (55ms)
2023-12-18 §
12:23 <kamila@deploy2002> helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. [production]
12:20 <kamila@deploy2002> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. [production]
12:20 <fabfur@cumin1002> START - Cookbook sre.hosts.reimage for host cp4037.ulsfo.wmnet with OS bullseye [production]
12:20 <kamila@deploy2002> helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. [production]
12:19 <fabfur@cumin1002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4037.ulsfo.wmnet with OS bullseye [production]
12:19 <kamila@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
12:18 <kamila@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
12:14 <kamila@deploy2002> helmfile [codfw] DONE helmfile.d/admin 'apply'. [production]
12:13 <kamila@deploy2002> helmfile [codfw] START helmfile.d/admin 'apply'. [production]
12:12 <Emperor> restart swift-proxy and envoyproxy on ms-fe1012 [production]
12:10 <kamila@deploy2002> helmfile [eqiad] DONE helmfile.d/admin 'apply'. [production]
12:09 <kamila@deploy2002> helmfile [eqiad] START helmfile.d/admin 'apply'. [production]
12:04 <kamila@deploy2002> helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. [production]
12:03 <kamila@deploy2002> helmfile [staging-eqiad] START helmfile.d/admin 'apply'. [production]
12:01 <moritzm> installing ncurses security updates [production]
11:59 <kamila@deploy2002> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
11:58 <kamila@deploy2002> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
11:51 <fabfur@cumin1002> START - Cookbook sre.hosts.reimage for host cp4037.ulsfo.wmnet with OS bullseye [production]
11:51 <fabfur@cumin1002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4037.ulsfo.wmnet with OS bullseye [production]
11:41 <elukey@deploy2002> helmfile [staging] DONE helmfile.d/services/recommendation-api: sync [production]
11:41 <elukey@deploy2002> helmfile [staging] START helmfile.d/services/recommendation-api: sync [production]
11:39 <moritzm> installing qemu security updates on bookworm [production]
11:38 <fabfur@cumin1002> START - Cookbook sre.hosts.reimage for host cp4037.ulsfo.wmnet with OS bullseye [production]
11:37 <fabfur@cumin1002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4037.ulsfo.wmnet with OS bullseye [production]
11:36 <fabfur@cumin1002> START - Cookbook sre.hosts.reimage for host cp4037.ulsfo.wmnet with OS bullseye [production]
11:36 <fabfur@cumin1002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4037.ulsfo.wmnet with OS bullseye [production]
10:56 <moritzm> restarting apache/FPM on mw canaries to pick up gnutls update [production]
10:54 <btullis> deploy conda-analytics v 0.0.27 to the hadoop-test-analytics cluster for T345482 [analytics]
10:52 <moritzm> installing gnutls28 security updates [production]
10:47 <fabfur@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4037.ulsfo.wmnet with reason: host reimage [production]
10:44 <fabfur@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on cp4037.ulsfo.wmnet with reason: host reimage [production]
10:39 <moritzm> installing jetty9 security updates [production]
10:29 <volans@cumin1002> END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox [production]
10:29 <volans@cumin1002> START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox [production]
10:17 <fabfur@cumin1002> START - Cookbook sre.hosts.reimage for host cp4037.ulsfo.wmnet with OS bullseye [production]
10:12 <XioNoX> remove VRRP pinning on cr1-eqiad/cr2-eqiad/cr2-codfw [production]
10:09 <moritzm> installing Linux 6.1.67 updates on Bookworm hosts [production]
09:45 <XioNoX> make eqiad-codfw 100G link primary [production]
09:43 <btullis> cleared some space on an-test-worker1001 by deleting old refinery jars from /tmp `btullis@an-test-worker1001:/tmp$ sudo find . -type f -mtime +60 -name *.jar -delete` [analytics]
09:22 <btullis> deploying refinery version 0.02.27 to production refinery jobs with https://gerrit.wikimedia.org/r/c/operations/puppet/+/980923 for T349121 [analytics]
09:10 <vgutierrez> vgutierrez@acmechief1002:~$ sudo -i keyholder arm - T352242 [production]
2023-12-17 §
18:48 <wm-bot> <lucaswerkmeister> deployed 7611d4e980 (l10n updates: ia, krc, sv) [tools.lexeme-forms]
17:39 <Krinkle> Reloading Zuul to deploy https://gerrit.wikimedia.org/r/983553 [releng]
12:59 <elukey> restart kubelet on ml-serve1001 (errors while syncing old containers) [production]
2023-12-16 §
22:01 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0) [tools]
22:01 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors [tools]
20:54 <bd808> Rebuilding all containers to pick up lighttpd config fix and normal package updates (T293552) [tools]
08:14 <dhinus> restarting toolsdb with jemalloc [tools]
05:32 <andrewbogott> restarting mariadb on toolsdb-1 because it's just about to go oom (or possibly just did) [tools]
01:21 <eevans@deploy2002> Finished deploy [cassandra/logstash-logback-encoder@fb10de1]: (no justification provided) (duration: 00m 10s) [production]