51-100 of 10000 results (23ms)
2024-10-07 ยง
13:49 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 2 days, 12:00:00 on db1203.eqiad.wmnet with reason: Maintenance [production]
13:49 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1192 (T367856)', diff saved to https://phabricator.wikimedia.org/P69488 and previous config saved to /var/cache/conftool/dbconfig/20241007-134929-ladsgroup.json [production]
13:37 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti2036.codfw.wmnet [production]
13:37 <vgutierrez> switching to digicert-2024 certificates on esams, eqsin, drmrs and magru [production]
13:36 <Lucas_WMDE> UTC afternoon backport+config window done [production]
13:35 <dreamyjazz@deploy2002> Finished scap sync-world: Backport for [[gerrit:1078406|Update globalblocks 'gb_address' index to allow autoblocks (T376052)]] (duration: 06m 49s) [production]
13:34 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P69487 and previous config saved to /var/cache/conftool/dbconfig/20241007-133422-ladsgroup.json [production]
13:31 <dreamyjazz@deploy2002> dreamyjazz: Continuing with sync [production]
13:30 <dreamyjazz@deploy2002> dreamyjazz: Backport for [[gerrit:1078406|Update globalblocks 'gb_address' index to allow autoblocks (T376052)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
13:28 <dreamyjazz@deploy2002> Started scap sync-world: Backport for [[gerrit:1078406|Update globalblocks 'gb_address' index to allow autoblocks (T376052)]] [production]
13:19 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P69486 and previous config saved to /var/cache/conftool/dbconfig/20241007-131915-ladsgroup.json [production]
13:12 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti2035.codfw.wmnet to cluster codfw and group C [production]
13:11 <jmm@cumin2002> START - Cookbook sre.ganeti.addnode for new host ganeti2035.codfw.wmnet to cluster codfw and group C [production]
13:10 <lucaswerkmeister-wmde@deploy2002> Finished scap sync-world: Backport for [[gerrit:1077800|scandium is being replaced by parsoidtest1001 (T363402)]] (duration: 07m 14s) [production]
13:05 <lucaswerkmeister-wmde@deploy2002> arlolra, lucaswerkmeister-wmde: Continuing with sync [production]
13:05 <lucaswerkmeister-wmde@deploy2002> arlolra, lucaswerkmeister-wmde: Backport for [[gerrit:1077800|scandium is being replaced by parsoidtest1001 (T363402)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
13:04 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1192 (T367856)', diff saved to https://phabricator.wikimedia.org/P69485 and previous config saved to /var/cache/conftool/dbconfig/20241007-130409-ladsgroup.json [production]
13:03 <lucaswerkmeister-wmde@deploy2002> Started scap sync-world: Backport for [[gerrit:1077800|scandium is being replaced by parsoidtest1001 (T363402)]] [production]
13:02 <jmm@cumin2002> END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti2035.codfw.wmnet to cluster codfw and group C [production]
13:02 <jmm@cumin2002> START - Cookbook sre.ganeti.addnode for new host ganeti2035.codfw.wmnet to cluster codfw and group C [production]
13:00 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2035.codfw.wmnet [production]
12:53 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti2035.codfw.wmnet [production]
12:53 <Lucas_WMDE> printf 'https://en.wikipedia.org/static/images/%s\n' 'mobile/copyright/wikimaniawiki-wordmark.svg' 'project-logos/wikimaniawiki-1.5x.png' 'project-logos/wikimaniawiki-2x.png' 'project-logos/wikimaniawiki.png' 'icons/wikimaniawiki.svg' | mwscript-k8s --attach -- purgeList enwiki # T376292 [production]
12:48 <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [admin]
12:48 <aborrero@cloudcumin1001> START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [admin]
12:41 <arturo> delete 2 instances (in shutoff) in preparation for project deletion (T371696) [wmdeanalytics]
12:17 <aborrero@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [admin]
12:06 <aborrero@cloudcumin1001> START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [admin]
12:06 <aborrero@cloudcumin1001> END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [admin]
12:05 <aborrero@cloudcumin1001> START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [admin]
12:03 <jelto@cumin1002> END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: test I946dd0b73b6be2d6b8093f03550f78d76188b92b with dummy upgrade [production]
12:02 <jelto@cumin1002> START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: test I946dd0b73b6be2d6b8093f03550f78d76188b92b with dummy upgrade [production]
11:29 <jayme@deploy1003> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
11:29 <jayme@deploy1003> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
11:28 <aborrero@cloudcumin1001> END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [admin]
11:27 <aborrero@cloudcumin1001> START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [admin]
11:25 <jayme@deploy1003> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
11:25 <jayme@deploy1003> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
11:16 <vgutierrez> uploaded golang-github-mtchavez-jenkins 1.0.0 to apt.wm.o (bookworm-wikimedia) - T376600 [production]
11:04 <arnaudb@cumin1002> dbctl commit (dc=all): 'db1233 (re)pooling @ 100%: T374215', diff saved to https://phabricator.wikimedia.org/P69484 and previous config saved to /var/cache/conftool/dbconfig/20241007-110430-arnaudb.json [production]
10:52 <jayme@cumin1002> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2002.codfw.wmnet [production]
10:50 <Dreamy_Jazz> Started 2 day scan on enwiki for MediaModeration to catchup with monthly request limit - https://wikitech.wikimedia.org/wiki/MediaModeration [production]
10:49 <Dreamy_Jazz> Started MediaModeration scanning script after it crashed for commonswiki - https://wikitech.wikimedia.org/wiki/MediaModeration [production]
10:49 <jayme@cumin1002> START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2002.codfw.wmnet [production]
10:49 <arnaudb@cumin1002> dbctl commit (dc=all): 'db1233 (re)pooling @ 75%: T374215', diff saved to https://phabricator.wikimedia.org/P69483 and previous config saved to /var/cache/conftool/dbconfig/20241007-104925-arnaudb.json [production]
10:47 <jayme@cumin1002> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2001.codfw.wmnet [production]
10:47 <jayme@cumin1002> START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2001.codfw.wmnet [production]
10:34 <arnaudb@cumin1002> dbctl commit (dc=all): 'db1233 (re)pooling @ 50%: T374215', diff saved to https://phabricator.wikimedia.org/P69482 and previous config saved to /var/cache/conftool/dbconfig/20241007-103420-arnaudb.json [production]
10:19 <arnaudb@cumin1002> dbctl commit (dc=all): 'db1233 (re)pooling @ 25%: T374215', diff saved to https://phabricator.wikimedia.org/P69481 and previous config saved to /var/cache/conftool/dbconfig/20241007-101914-arnaudb.json [production]
10:17 <vgutierrez> uploaded golang-github-cloudflare-ipvs 0.10.2 to apt.wm.o (bookworm-wikimedia) - T376600 [production]