6401-6450 of 10000 results (134ms)
2024-11-14 ยง
13:18 <oblivian@cumin1002> START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "Fix search button height - oblivian@cumin1002" [production]
13:05 <jayme@cumin2002> END (PASS) - Cookbook sre.k8s.reimage-stacked-control-plane (exit_code=0) Reimaging k8s control planes of cluster wikikube-codfw: containerd migration [production]
13:04 <jayme@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-ctrl2003.codfw.wmnet with OS bookworm [production]
12:54 <jmm@cumin2002> END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-eventschemas (exit_code=0) rolling restart_daemons on A:schema-eqiad [production]
12:53 <jmm@cumin2002> START - Cookbook sre.misc-clusters.roll-restart-reboot-eventschemas rolling restart_daemons on A:schema-eqiad [production]
12:53 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti7004.magru.wmnet [production]
12:52 <moritzm> installing apache2 security updates [production]
12:51 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti7004.magru.wmnet [production]
12:51 <dreamyjazz@deploy2002> Finished scap sync-world: Backport for [[gerrit:1090511|Hide IP reveal tools on Special:AbuseLog and Special:GlobalBlockList (T379583)]] (duration: 09m 08s) [production]
12:49 <moritzm> failover ganeti master of magru02 to ganeti7002 [production]
12:46 <dreamyjazz@deploy2002> dreamyjazz: Continuing with sync [production]
12:45 <dreamyjazz@deploy2002> dreamyjazz: Backport for [[gerrit:1090511|Hide IP reveal tools on Special:AbuseLog and Special:GlobalBlockList (T379583)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
12:43 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti7002.magru.wmnet [production]
12:42 <jayme@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-ctrl2003.codfw.wmnet with reason: host reimage [production]
12:41 <dreamyjazz@deploy2002> Started scap sync-world: Backport for [[gerrit:1090511|Hide IP reveal tools on Special:AbuseLog and Special:GlobalBlockList (T379583)]] [production]
12:38 <jayme@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-ctrl2003.codfw.wmnet with reason: host reimage [production]
12:35 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti7002.magru.wmnet [production]
12:29 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti7002.magru.wmnet [production]
12:25 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti7002.magru.wmnet [production]
12:22 <jayme@cumin2002> START - Cookbook sre.hosts.reimage for host wikikube-ctrl2003.codfw.wmnet with OS bookworm [production]
12:19 <jmm@cumin2002> END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-eventschemas (exit_code=0) rolling restart_daemons on A:schema-codfw [production]
12:18 <jmm@cumin2002> START - Cookbook sre.misc-clusters.roll-restart-reboot-eventschemas rolling restart_daemons on A:schema-codfw [production]
12:17 <jayme@cumin2002> START - Cookbook sre.k8s.reimage-stacked-control-plane Reimaging k8s control planes of cluster wikikube-codfw: containerd migration [production]
12:10 <jmm@cumin2002> END (PASS) - Cookbook sre.cdn.roll-restart-reboot-ncredir (exit_code=0) rolling restart_daemons on A:ncredir [production]
12:00 <jmm@cumin2002> START - Cookbook sre.cdn.roll-restart-reboot-ncredir rolling restart_daemons on A:ncredir [production]
11:57 <moritzm> restarting postfix on inbound/outbound servers to pick up openssl updates [production]
11:17 <moritzm> installing openssl security updates [production]
11:08 <jayme@cumin2002> END (PASS) - Cookbook sre.k8s.reimage-stacked-control-plane (exit_code=0) Reimaging k8s control planes of cluster wikikube-codfw: containerd migration [production]
11:08 <jayme@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-ctrl2001.codfw.wmnet with OS bookworm [production]
10:47 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub: sync on production [production]
10:45 <jayme@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-ctrl2001.codfw.wmnet with reason: host reimage [production]
10:44 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub: apply on production [production]
10:42 <jayme@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-ctrl2001.codfw.wmnet with reason: host reimage [production]
10:16 <moritzm> remove ganeti2017 from active ganeti nodes T376594 [production]
10:15 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2017.codfw.wmnet [production]
10:11 <jayme@cumin2002> START - Cookbook sre.hosts.reimage for host wikikube-ctrl2001.codfw.wmnet with OS bookworm [production]
10:07 <jnuche@deploy2002> Finished deploy [releng/jenkins-deploy@34b35a5] (releasing): (no justification provided) (duration: 00m 47s) [production]
10:06 <jayme@cumin2002> START - Cookbook sre.k8s.reimage-stacked-control-plane Reimaging k8s control planes of cluster wikikube-codfw: containerd migration [production]
10:06 <jnuche@deploy2002> Started deploy [releng/jenkins-deploy@34b35a5] (releasing): (no justification provided) [production]
10:03 <jnuche@deploy2002> Finished deploy [releng/jenkins-deploy@34b35a5] (releasing): (no justification provided) (duration: 00m 21s) [production]
10:03 <jnuche@deploy2002> Started deploy [releng/jenkins-deploy@34b35a5] (releasing): (no justification provided) [production]
09:43 <kart_> Done: UTC morning backport window [production]
09:37 <kartik@deploy2002> Finished scap sync-world: Backport for [[gerrit:1090988|Correction to virtual-globaljsonlinks mapping (T374746)]] (duration: 10m 03s) [production]
09:37 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
09:36 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
09:35 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
09:34 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
09:32 <kartik@deploy2002> bvibber, kartik: Continuing with sync [production]
09:31 <kartik@deploy2002> bvibber, kartik: Backport for [[gerrit:1090988|Correction to virtual-globaljsonlinks mapping (T374746)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
09:27 <kartik@deploy2002> Started scap sync-world: Backport for [[gerrit:1090988|Correction to virtual-globaljsonlinks mapping (T374746)]] [production]