1001-1050 of 10000 results (108ms)
2025-11-10 §
12:56 <fceratto@cumin1003> START - Cookbook sre.mysql.pool db2166 gradually with 4 steps - Migration of db2166.codfw.wmnet completed [production]
12:49 <Amir1> drop database if exists tokiponawiki; drop database if exists tokiponawikibooks; drop database if exists tokiponawikiquote; drop database if exists tokiponawiktionary; (T297297) [production]
12:46 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply [production]
12:46 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [production]
12:44 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
12:44 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
12:43 <Amir1> drop database if exists de_labswikimedia; drop database if exists en_labswikimedia; drop database if exists flaggedrevs_labswikimedia; drop database if exists liquidthreads_labswikimedia; drop database if exists readerfeedback_labswikimedia; (T297297) [production]
12:29 <fceratto@cumin1003> END (PASS) - Cookbook sre.mysql.depool (exit_code=0) db2166 - Upgrading db2166.codfw.wmnet [production]
12:28 <fceratto@cumin1003> START - Cookbook sre.mysql.depool db2166 - Upgrading db2166.codfw.wmnet [production]
12:28 <fceratto@cumin1003> START - Cookbook sre.mysql.major-upgrade [production]
11:10 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1033.eqiad.wmnet [production]
11:09 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1033.eqiad.wmnet [production]
11:08 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1024.eqiad.wmnet [production]
11:04 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1024.eqiad.wmnet [production]
10:04 <slyngshede@dns1004> END - running authdns-update [production]
10:03 <slyngshede@dns1004> START - running authdns-update [production]
10:03 <slyngs> Upgrade CAS (idp.wikimedia.org) to version 7.2.7 [production]
09:59 <dpogorzelski@deploy2002> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. [production]
09:59 <dpogorzelski@deploy2002> helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. [production]
09:59 <dpogorzelski@deploy2002> helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. [production]
09:59 <dpogorzelski@deploy2002> helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. [production]
09:49 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply [production]
09:48 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply [production]
09:23 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
09:22 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
08:48 <dpogorzelski@deploy2002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [production]
08:48 <moritzm> installing Java 8 security updates on Bookworm [production]
08:48 <moritzm> uploaded openjdk-8 8u472-ga-1~deb12u1 to apt.wikimedia.org (forward port of latest Java 8 security updates) [production]
08:21 <dpogorzelski@deploy2002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [production]
08:19 <dpogorzelski@deploy2002> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. [production]
08:19 <dpogorzelski@deploy2002> helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. [production]
08:07 <root@cumin2002> DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Stevemunene out of all services on: 2395 hosts [production]
01:42 <stran@deploy2002> helmfile [codfw] DONE helmfile.d/services/ipoid: apply [production]
01:42 <stran@deploy2002> helmfile [codfw] START helmfile.d/services/ipoid: apply [production]
01:42 <stran@deploy2002> helmfile [eqiad] DONE helmfile.d/services/ipoid: apply [production]
01:41 <stran@deploy2002> helmfile [eqiad] START helmfile.d/services/ipoid: apply [production]
01:38 <stran@deploy2002> helmfile [staging] DONE helmfile.d/services/ipoid: apply [production]
01:36 <stran@deploy2002> helmfile [staging] START helmfile.d/services/ipoid: apply [production]
01:14 <mwpresync@deploy2002> Finished scap build-images: Publishing wmf/next image (duration: 14m 17s) [production]
01:00 <mwpresync@deploy2002> Started scap build-images: Publishing wmf/next image [production]
2025-11-09 §
01:14 <mwpresync@deploy2002> Finished scap build-images: Publishing wmf/next image (duration: 13m 20s) [production]
01:00 <mwpresync@deploy2002> Started scap build-images: Publishing wmf/next image [production]
2025-11-08 §
01:14 <mwpresync@deploy2002> Finished scap build-images: Publishing wmf/next image (duration: 14m 08s) [production]
01:00 <mwpresync@deploy2002> Started scap build-images: Publishing wmf/next image [production]
00:49 <ryankemper@cumin1002> END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.REBOOT (2 nodes at a time) for ElasticSearch cluster search_codfw: codfw cluster reboot (apply updates) - ryankemper@cumin1002 - T390860 [production]
2025-11-07 §
23:27 <andrew@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudlb2003-dev.codfw.wmnet with OS trixie [production]
23:04 <andrew@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudlb2003-dev.codfw.wmnet with reason: host reimage [production]
23:00 <andrew@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudlb2003-dev.codfw.wmnet with reason: host reimage [production]
22:43 <andrew@cumin2002> START - Cookbook sre.hosts.reimage for host cloudlb2003-dev.codfw.wmnet with OS trixie [production]
22:40 <ryankemper@cumin1002> START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (2 nodes at a time) for ElasticSearch cluster search_codfw: codfw cluster reboot (apply updates) - ryankemper@cumin1002 - T390860 [production]