2501-2550 of 10000 results (36ms)
2025-02-18 ยง
12:13 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [admin]
12:13 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [admin]
12:13 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [admin]
12:10 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [admin]
12:01 <moritzm> installing openjdk-17 security updates [production]
11:01 <jmm@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ganeti1033.eqiad.wmnet [production]
11:01 <taavi@deploy2002> Finished scap sync-world: Backport for [[gerrit:1120149|wikitech: Unset $wgEnableCreativeCommonsRdf]] (duration: 15m 45s) [production]
11:01 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1033.eqiad.wmnet [production]
10:51 <taavi@deploy2002> taavi: Continuing with sync [production]
10:51 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti1033.eqiad.wmnet [production]
10:51 <taavi@deploy2002> taavi: Backport for [[gerrit:1120149|wikitech: Unset $wgEnableCreativeCommonsRdf]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
10:50 <tappof@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on grafana1002.eqiad.wmnet with reason: expand the root partition and fs on grafana1002 [production]
10:45 <taavi@deploy2002> Started scap sync-world: Backport for [[gerrit:1120149|wikitech: Unset $wgEnableCreativeCommonsRdf]] [production]
10:24 <moritzm> installing libpgjava security updates [production]
10:14 <jmm@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ganeti1033.eqiad.wmnet [production]
10:14 <jayme@deploy1003> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
10:14 <jayme@deploy1003> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
10:10 <jayme@deploy1003> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
10:09 <jayme@deploy1003> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
10:09 <jayme@deploy1003> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
10:09 <jayme@deploy1003> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
10:08 <jayme@deploy1003> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
10:08 <jayme@deploy1003> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
10:07 <jayme@deploy1003> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
10:07 <jayme@deploy1003> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
10:07 <jayme@deploy1003> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
10:07 <jayme@deploy1003> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
09:59 <jayme@deploy1003> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
09:58 <jayme@deploy1003> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
09:58 <jayme@deploy1003> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
09:58 <jayme@deploy1003> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
09:58 <jayme@deploy1003> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
09:58 <jayme@deploy1003> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
09:57 <tappof@dns1004> END - running authdns-update [production]
09:55 <tappof@dns1004> START - running authdns-update [production]
09:53 <elukey@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ganeti1033.eqiad.wmnet [production]
09:47 <elukey@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ganeti1033.eqiad.wmnet [production]
09:42 <tappof> performing grafana failover (grafana2001 is becoming the new active host) T385282 [production]
09:42 <kamila@cumin1002> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1123.eqiad.wmnet [production]
09:42 <kamila@cumin1002> START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1123.eqiad.wmnet [production]
09:35 <elukey@cumin2002> END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts ganeti1033.eqiad.wmnet [production]
09:32 <kamila@cumin1002> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1148-1153].eqiad.wmnet [production]
09:32 <kamila@cumin1002> START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1148-1153].eqiad.wmnet [production]
09:26 <elukey@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ganeti1033.eqiad.wmnet [production]
08:57 <urbanecm@deploy2002> Finished scap sync-world: Backport for [[gerrit:1119778|Rename global variable from the WikimediaIncubator extension]] (duration: 24m 32s) [production]
08:50 <jayme@deploy1003> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
08:50 <jayme@deploy1003> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
08:48 <urbanecm@deploy2002> urbanecm, jhsoby: Continuing with sync [production]
08:39 <urbanecm@deploy2002> urbanecm, jhsoby: Backport for [[gerrit:1119778|Rename global variable from the WikimediaIncubator extension]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
08:33 <urbanecm@deploy2002> Started scap sync-world: Backport for [[gerrit:1119778|Rename global variable from the WikimediaIncubator extension]] [production]