1101-1150 of 10000 results (80ms)
2023-01-18 ยง
16:39 <jdrewniak@deploy1002> Finished scap: Backport for [[gerrit:881023|[100%] English Wikipedia uses Vector 2022 skin]] (duration: 09m 27s) [production]
16:31 <jdrewniak@deploy1002> jdrewniak and jdlrobson: Backport for [[gerrit:881023|[100%] English Wikipedia uses Vector 2022 skin]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet [production]
16:29 <jdrewniak@deploy1002> Started scap: Backport for [[gerrit:881023|[100%] English Wikipedia uses Vector 2022 skin]] [production]
16:20 <jdrewniak@deploy1002> Finished scap: Backport for [[gerrit:881022|[75%] English Wikipedia uses Vector 2022 skin (T326892)]] (duration: 09m 24s) [production]
16:13 <jdrewniak@deploy1002> jdrewniak and jdlrobson: Backport for [[gerrit:881022|[75%] English Wikipedia uses Vector 2022 skin (T326892)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet [production]
16:11 <jdrewniak@deploy1002> Started scap: Backport for [[gerrit:881022|[75%] English Wikipedia uses Vector 2022 skin (T326892)]] [production]
16:06 <otto@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/services/flink-app-example: apply [production]
16:06 <otto@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/services/flink-app-example: apply [production]
15:58 <jdrewniak@deploy1002> Finished scap: Backport for [[gerrit:881021|[50%] English Wikipedia uses Vector 2022 skin, adds instrumentation (T326892)]] (duration: 08m 52s) [production]
15:51 <jdrewniak@deploy1002> jdrewniak and jdlrobson: Backport for [[gerrit:881021|[50%] English Wikipedia uses Vector 2022 skin, adds instrumentation (T326892)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet [production]
15:49 <jdrewniak@deploy1002> Started scap: Backport for [[gerrit:881021|[50%] English Wikipedia uses Vector 2022 skin, adds instrumentation (T326892)]] [production]
15:44 <jdrewniak@deploy1002> Finished scap: Backport for [[gerrit:881020|[25%] English Wikipedia uses Vector 2022 skin (T326892)]] (duration: 09m 06s) [production]
15:37 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1052.eqiad.wmnet with OS bullseye [production]
15:37 <elukey@deploy1002> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. [production]
15:37 <elukey@deploy1002> helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. [production]
15:36 <jdrewniak@deploy1002> jdrewniak and jdlrobson: Backport for [[gerrit:881020|[25%] English Wikipedia uses Vector 2022 skin (T326892)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet [production]
15:35 <jdrewniak@deploy1002> Started scap: Backport for [[gerrit:881020|[25%] English Wikipedia uses Vector 2022 skin (T326892)]] [production]
15:31 <urandom> re-enabling Cassandra hinted-handoff for codfw -- T327001 [production]
15:29 <jdrewniak@deploy1002> Finished scap: Backport for [[gerrit:879659|[10%] English Wikipedia uses Vector 2022 skin (T326892)]] (duration: 11m 30s) [production]
15:21 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1052.eqiad.wmnet with reason: host reimage [production]
15:19 <jdrewniak@deploy1002> jdrewniak and jdlrobson: Backport for [[gerrit:879659|[10%] English Wikipedia uses Vector 2022 skin (T326892)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet [production]
15:18 <jiji@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mc1052.eqiad.wmnet with reason: host reimage [production]
15:17 <jdrewniak@deploy1002> Started scap: Backport for [[gerrit:879659|[10%] English Wikipedia uses Vector 2022 skin (T326892)]] [production]
15:14 <lucaswerkmeister-wmde@deploy1002> Finished scap: Backport for [[gerrit:880921|Revert gallery changes in 1.40.0-wmf.18 & .19 (T326990)]] (duration: 09m 11s) [production]
15:12 <bblack> cp2031: rebooting to gather more information (still downtimed + depooled) [production]
15:07 <jiji@cumin1001> START - Cookbook sre.hosts.reimage for host mc1052.eqiad.wmnet with OS bullseye [production]
15:06 <lucaswerkmeister-wmde@deploy1002> lucaswerkmeister-wmde and matmarex: Backport for [[gerrit:880921|Revert gallery changes in 1.40.0-wmf.18 & .19 (T326990)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet [production]
15:05 <lucaswerkmeister-wmde@deploy1002> Started scap: Backport for [[gerrit:880921|Revert gallery changes in 1.40.0-wmf.18 & .19 (T326990)]] [production]
15:04 <lucaswerkmeister-wmde@deploy1002> Finished scap: Backport for [[gerrit:880920|Revert gallery changes in 1.40.0-wmf.18 (T326990)]] (duration: 13m 04s) [production]
15:01 <bblack> cp2031: rebooting to gather more information (still downtimed + depooled) [production]
14:57 <moritzm> uploaded python-jose 3.3.0+dfsg-4~wmf11u1 to apt.wikmedia.org (needed by python-social-auth/Bitu) [production]
14:53 <lucaswerkmeister-wmde@deploy1002> lucaswerkmeister-wmde and matmarex: Backport for [[gerrit:880920|Revert gallery changes in 1.40.0-wmf.18 (T326990)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet [production]
14:51 <lucaswerkmeister-wmde@deploy1002> Started scap: Backport for [[gerrit:880920|Revert gallery changes in 1.40.0-wmf.18 (T326990)]] [production]
14:46 <lucaswerkmeister-wmde@deploy1002> Finished scap: Backport for [[gerrit:881045|Revert "Breaking upgrade: mapdata" (T327151)]] (duration: 10m 33s) [production]
14:37 <lucaswerkmeister-wmde@deploy1002> lucaswerkmeister-wmde and wmde-fisch: Backport for [[gerrit:881045|Revert "Breaking upgrade: mapdata" (T327151)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet [production]
14:35 <lucaswerkmeister-wmde@deploy1002> Started scap: Backport for [[gerrit:881045|Revert "Breaking upgrade: mapdata" (T327151)]] [production]
14:34 <lucaswerkmeister-wmde@deploy1002> Finished scap: Backport for [[gerrit:879946|Write to cul_reason[_plaintext]_id everywhere (T233004)]] (duration: 19m 54s) [production]
14:23 <moritzm> installing mod-wsgi security updates [production]
14:16 <lucaswerkmeister-wmde@deploy1002> lucaswerkmeister-wmde and dreamyjazz: Backport for [[gerrit:879946|Write to cul_reason[_plaintext]_id everywhere (T233004)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet [production]
14:14 <lucaswerkmeister-wmde@deploy1002> Started scap: Backport for [[gerrit:879946|Write to cul_reason[_plaintext]_id everywhere (T233004)]] [production]
13:17 <filippo@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on webperf1004.eqiad.wmnet with reason: decom [production]
13:16 <filippo@cumin1001> START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on webperf1004.eqiad.wmnet with reason: decom [production]
12:20 <jelto@cumin1001> END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) [production]
11:54 <volans> upgraded cumin on cumin1001 to 4.2.0-1+deb11u1 [production]
11:47 <btullis@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 30 days, 0:00:00 on 10 hosts with reason: Still not ready to add these new presto servers to the cluster - btullis [production]
11:47 <btullis@cumin1001> START - Cookbook sre.hosts.downtime for 30 days, 0:00:00 on 10 hosts with reason: Still not ready to add these new presto servers to the cluster - btullis [production]
11:42 <jelto@cumin1001> START - Cookbook sre.gitlab.upgrade [production]
11:27 <jelto@cumin1001> END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) [production]
11:16 <volans@cumin1001> END (PASS) - Cookbook sre.network.cf (exit_code=0) [production]
11:16 <volans@cumin1001> START - Cookbook sre.network.cf [production]