1501-1550 of 10000 results (32ms)
2025-02-21 §
15:19 <jynus> restarting mariadb @ pc2015 [production]
15:15 <jynus@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on pc[2015,2017].codfw.wmnet with reason: processes stuck [production]
14:51 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depool pc5', diff saved to https://phabricator.wikimedia.org/P73503 and previous config saved to /var/cache/conftool/dbconfig/20250221-145110-ladsgroup.json [production]
14:48 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depool pc7', diff saved to https://phabricator.wikimedia.org/P73502 and previous config saved to /var/cache/conftool/dbconfig/20250221-144852-ladsgroup.json [production]
14:21 <dcausse@deploy2002> helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
14:21 <dcausse@deploy2002> helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
14:18 <dcausse@deploy2002> helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
14:18 <dcausse@deploy2002> helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply [production]
13:52 <jynus> set global read_only=false @ db2230 [production]
13:04 <cmooney@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on netflow1002.eqiad.wmnet with reason: keeping gnmic running in debug mode to observe performance change [production]
12:57 <fnegri@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 [tools]
12:52 <fnegri@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 [tools]
12:50 <dcausse@deploy2002> helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
12:50 <dcausse@deploy2002> helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
12:40 <jynus@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on db[2198,2200].codfw.wmnet with reason: Table rebuilding ongoing [production]
12:32 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [admin]
12:29 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.restart_openstack [admin]
11:20 <elukey@deploy2002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . [production]
11:03 <klausman@deploy2002> helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. [production]
11:02 <klausman@deploy2002> helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. [production]
10:59 <klausman@deploy2002> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. [production]
10:58 <klausman@deploy2002> helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. [production]
10:58 <isaranto@deploy2002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revision-models' for release 'main' . [production]
10:56 <isaranto@deploy2002> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revision-models' for release 'main' . [production]
10:39 <jynus@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2198.codfw.wmnet with reason: upgrade and rebuild tables [production]
10:01 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply [production]
10:01 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply [production]
09:21 <elukey@deploy2002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revision-models' for release 'main' . [production]
08:35 <wmbot~dcaro@urcuchillay> END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) [globaleducation]
08:35 <dcaro> reboot peony-web lost network connectivity due to OOM killing things [globaleducation]
08:31 <wmbot~dcaro@urcuchillay> START - Cookbook wmcs.openstack.cloudvirt.vm_console [globaleducation]
08:29 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics: apply [production]
08:28 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics: apply [production]
08:26 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics: apply [production]
08:25 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics: apply [production]
07:54 <jmm@cumin2002> END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1026.eqiad.wmnet to cluster eqiad and group A [production]
07:53 <jmm@cumin2002> START - Cookbook sre.ganeti.addnode for new host ganeti1026.eqiad.wmnet to cluster eqiad and group A [production]
07:51 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1026.eqiad.wmnet [production]
07:43 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti1026.eqiad.wmnet [production]
07:22 <jynus> rebuilding tables for db2200 T385550 [production]
07:21 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti1026.eqiad.wmnet with OS bookworm [production]
07:06 <jynus@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2200.codfw.wmnet with reason: upgrade and rebuild tables [production]
07:02 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1026.eqiad.wmnet with reason: host reimage [production]
06:56 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1026.eqiad.wmnet with reason: host reimage [production]
06:40 <jmm@cumin2002> START - Cookbook sre.hosts.reimage for host ganeti1026.eqiad.wmnet with OS bookworm [production]
2025-02-20 §
22:36 <jdrewniak@deploy2002> Finished scap sync-world: Backport for [[gerrit:1121450|Fix 0 tick not firing for session length mixin, and ensure ticks happen every 30 seconds (T386495)]] (duration: 13m 52s) [production]
22:30 <jdrewniak@deploy2002> jdrewniak: Continuing with sync [production]
22:25 <jdrewniak@deploy2002> jdrewniak: Backport for [[gerrit:1121450|Fix 0 tick not firing for session length mixin, and ensure ticks happen every 30 seconds (T386495)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
22:22 <jdrewniak@deploy2002> Started scap sync-world: Backport for [[gerrit:1121450|Fix 0 tick not firing for session length mixin, and ensure ticks happen every 30 seconds (T386495)]] [production]
22:06 <jhathaway@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on ms-be2088.codfw.wmnet with reason: T381919 [production]