401-450 of 10000 results (29ms)
2024-11-04 ยง
13:06 <Dreamy_Jazz> Started MediaModeration scan on all wikis other than s4 (commonswiki + testcommonswiki) - https://wikitech.wikimedia.org/wiki/MediaModeration [production]
13:06 <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 (T362867) [tools]
13:04 <sstefanova@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component builds-api [tools]
13:04 <sstefanova@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api [tools]
13:02 <sstefanova@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component builds-api [tools]
12:55 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1167 (T376905)', diff saved to https://phabricator.wikimedia.org/P70861 and previous config saved to /var/cache/conftool/dbconfig/20241104-125459-ladsgroup.json [production]
12:55 <raymond-ndibe@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 (T362867) [tools]
12:49 <XioNoX> deploy "Add temporary LVS community for liberica test" - T378453 [production]
12:49 <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 (T362867) [tools]
12:47 <raymond-ndibe@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 (T362867) [tools]
12:45 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db1167 (T376905)', diff saved to https://phabricator.wikimedia.org/P70860 and previous config saved to /var/cache/conftool/dbconfig/20241104-124533-ladsgroup.json [production]
12:45 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
12:45 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
12:45 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1167.eqiad.wmnet with reason: Maintenance [production]
12:44 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1167.eqiad.wmnet with reason: Maintenance [production]
12:44 <sstefanova@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [toolsbeta]
12:41 <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 (T362867) [tools]
12:41 <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.27.16 to 1.28.14 (T362867) [tools]
12:40 <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.27.16 to 1.28.14 (T362867) [tools]
12:40 <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-108 from 1.27.16 to 1.28.14 (T362867) [tools]
12:39 <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-108 from 1.27.16 to 1.28.14 (T362867) [tools]
12:39 <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.27.16 to 1.28.14 (T362867) [tools]
12:38 <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.27.16 to 1.28.14 (T362867) [tools]
12:38 <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.27.16 to 1.28.14 (T362867) [tools]
12:38 <sstefanova@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component builds-api [toolsbeta]
12:36 <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.27.16 to 1.28.14 (T362867) [tools]
12:36 <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.27.16 to 1.28.14 (T362867) [tools]
12:35 <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.27.16 to 1.28.14 (T362867) [tools]
12:35 <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.27.16 to 1.28.14 (T362867) [tools]
12:35 <elukey@cumin1002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1052.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART [production]
12:34 <isaranto@deploy2002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . [production]
12:34 <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.27.16 to 1.28.14 (T362867) [tools]
12:34 <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.27.16 to 1.28.14 (T362867) [tools]
12:33 <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.27.16 to 1.28.14 (T362867) [tools]
12:24 <elukey@cumin1002> START - Cookbook sre.hosts.provision for host ganeti1052.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART [production]
12:22 <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.27.16 to 1.28.14 (T362867) [tools]
12:22 <isaranto@deploy2002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . [production]
12:22 <sstefanova@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api [tools]
12:22 <isaranto@deploy2002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . [production]
12:20 <isaranto@deploy2002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . [production]
12:19 <isaranto@deploy2002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . [production]
12:19 <isaranto@deploy2002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . [production]
12:16 <sstefanova@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component envvars-api [tools]
12:13 <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.27.16 to 1.28.14 (T362867) [tools]
12:11 <sstefanova@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api [tools]
12:11 <jmm@cumin2002> END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1039.eqiad.wmnet to cluster eqiad and group B [production]
12:11 <jmm@cumin2002> START - Cookbook sre.ganeti.addnode for new host ganeti1039.eqiad.wmnet to cluster eqiad and group B [production]
12:10 <isaranto@deploy2002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . [production]
12:08 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1039.eqiad.wmnet [production]
12:08 <elukey@cumin1002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1051.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART [production]