7951-8000 of 10000 results (126ms)
2025-01-21 §
09:52 <dcausse@deploy2002> helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply [production]
09:47 <godog> set udp_localhost-info retention.bytes=100000000000 on kafka-logging - T384233 [production]
09:46 <marostegui@cumin1002> dbctl commit (dc=all): 'db2148 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P72173 and previous config saved to /var/cache/conftool/dbconfig/20250121-094637-root.json [production]
09:45 <marostegui@cumin1002> dbctl commit (dc=all): 'db2207 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P72172 and previous config saved to /var/cache/conftool/dbconfig/20250121-094537-root.json [production]
09:39 <dcausse@deploy2002> helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply [production]
09:39 <dcausse@deploy2002> helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply [production]
09:37 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2024.codfw.wmnet with reason: host reimage [production]
09:33 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2024.codfw.wmnet with reason: host reimage [production]
09:32 <dcausse@deploy2002> helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply [production]
09:32 <dcausse@deploy2002> helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply [production]
09:30 <dcausse@deploy2002> helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply [production]
09:30 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.resource-report (exit_code=0) [production]
09:30 <jmm@cumin2002> START - Cookbook sre.ganeti.resource-report [production]
09:30 <dcausse@deploy2002> helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply [production]
09:30 <dcausse@deploy2002> helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply [production]
09:29 <dcausse@deploy2002> helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply [production]
09:29 <dcausse@deploy2002> helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply [production]
09:10 <jmm@cumin2002> START - Cookbook sre.hosts.reimage for host ganeti2024.codfw.wmnet with OS bookworm [production]
09:04 <dcausse@deploy2002> helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply [production]
09:03 <dcausse@deploy2002> helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply [production]
08:59 <dcausse@deploy2002> helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply [production]
08:58 <dcausse@deploy2002> helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply [production]
08:51 <marostegui@cumin1002> END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db2216.codfw.wmnet onto db2203.codfw.wmnet [production]
08:36 <jmm@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti2024.codfw.wmnet with reason: remove from cluster for reimage [production]
08:02 <kevinbazira@deploy2002> helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'article-models' for release 'main' . [production]
07:59 <kevinbazira@deploy2002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'article-models' for release 'main' . [production]
07:56 <kevinbazira@deploy2002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'article-models' for release 'main' . [production]
07:42 <aqu@deploy2002> Finished deploy [airflow-dags/analytics@868de0c]: 202412 Backfill: Fixes on ExternalTaskMarker experiment (duration: 00m 32s) [production]
07:41 <aqu@deploy2002> Started deploy [airflow-dags/analytics@868de0c]: 202412 Backfill: Fixes on ExternalTaskMarker experiment [production]
07:29 <oblivian@cumin1002> END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "Bugfixes - oblivian@cumin1002" [production]
07:29 <oblivian@cumin1002> END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes - oblivian@cumin1002 [production]
07:28 <oblivian@cumin1002> START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Bugfixes - oblivian@cumin1002 [production]
07:28 <oblivian@cumin1002> START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "Bugfixes - oblivian@cumin1002" [production]
07:02 <marostegui@cumin1002> START - Cookbook sre.mysql.clone of db2216.codfw.wmnet onto db2203.codfw.wmnet [production]
06:58 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2216.codfw.wmnet with reason: rebuilding index [production]
06:56 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2216 T384273', diff saved to https://phabricator.wikimedia.org/P72169 and previous config saved to /var/cache/conftool/dbconfig/20250121-065640-marostegui.json [production]
06:52 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2203.codfw.wmnet with reason: rebuilding index [production]
06:51 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2203 T383690', diff saved to https://phabricator.wikimedia.org/P72168 and previous config saved to /var/cache/conftool/dbconfig/20250121-065114-marostegui.json [production]
06:51 <marostegui@dns1006> END - running authdns-update [production]
06:49 <marostegui@dns1006> START - running authdns-update [production]
06:49 <marostegui@dns1006> START - running authdns-update [production]
05:04 <mwpresync@deploy2002> Pruned MediaWiki: 1.44.0-wmf.8 (duration: 04m 55s) [production]
05:01 <mwpresync@deploy2002> Finished scap sync-world: testwikis to 1.44.0-wmf.13 refs T382364 (duration: 58m 53s) [production]
04:02 <mwpresync@deploy2002> Started scap sync-world: testwikis to 1.44.0-wmf.13 refs T382364 [production]
2025-01-20 §
23:37 <jynus> run OPTIMIZE TABLE recentchanges on db2207 [production]
22:14 <urbanecm> [urbanecm@deploy2002 ~]$ mwscript-k8s -f namespaceDupes.php -- --wiki=uzwiktionary --fix --add-prefix=BROKEN # T362620, logs posted to the task [production]
22:08 <urbanecm> Run mwscript-k8s -f namespaceDupes.php -- --wiki=uzwiktionary --fix # T362620 # logs are at P72163 [production]
22:03 <urbanecm@deploy2002> Finished scap sync-world: Backport for [[gerrit:1079619|Update uzwiktionary project namespace and site name (T362620)]] (duration: 12m 45s) [production]
21:56 <urbanecm@deploy2002> nmw03, urbanecm: Continuing with sync [production]
21:54 <urbanecm@deploy2002> nmw03, urbanecm: Backport for [[gerrit:1079619|Update uzwiktionary project namespace and site name (T362620)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]