4501-4550 of 10000 results (99ms)
2024-06-10 ยง
13:20 <ladsgroup@deploy1002> Finished scap: Backport for [[gerrit:1041044|[huwiki] Add "suppressredirect" user right to editor user group (T366438)]] (duration: 15m 05s) [production]
13:19 <fabfur@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp4047.ulsfo.wmnet [production]
13:18 <taavi@cumin1002> END (FAIL) - Cookbook sre.loadbalancer.restart-pybal (exit_code=99) rolling-restart of pybal on A:lvs-secondary-eqiad or A:lvs-low-traffic-eqiad and A:lvs [production]
13:18 <taavi@cumin1002> START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on A:lvs-secondary-eqiad or A:lvs-low-traffic-eqiad and A:lvs [production]
13:13 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2021.codfw.wmnet [production]
13:13 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2021.codfw.wmnet [production]
13:13 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1018.eqiad.wmnet [production]
13:13 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1018.eqiad.wmnet [production]
13:11 <taavi> restarting eqiad low-traffic LVS for https://gerrit.wikimedia.org/r/c/operations/puppet/+/941459 [production]
13:11 <ladsgroup@deploy1002> ladsgroup and gergesshamon: Continuing with sync [production]
13:10 <fabfur@cumin1002> START - Cookbook sre.hosts.reboot-single for host cp4047.ulsfo.wmnet [production]
13:10 <elukey@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
13:09 <fabfur@cumin1002> conftool action : set/pooled=no; selector: name=cp4047.ulsfo.wmnet [production]
13:09 <fabfur> rebooting cp4047 (T366555) [production]
13:09 <elukey@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
13:08 <ladsgroup@deploy1002> ladsgroup and gergesshamon: Backport for [[gerrit:1041044|[huwiki] Add "suppressredirect" user right to editor user group (T366438)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
13:08 <elukey@deploy1002> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. [production]
13:07 <elukey@deploy1002> helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. [production]
13:07 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti2021.codfw.wmnet [production]
13:07 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti1018.eqiad.wmnet [production]
13:05 <ladsgroup@deploy1002> Started scap: Backport for [[gerrit:1041044|[huwiki] Add "suppressredirect" user right to editor user group (T366438)]] [production]
13:04 <elukey@deploy1002> helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. [production]
13:04 <elukey@deploy1002> helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. [production]
13:03 <elukey@deploy1002> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. [production]
13:03 <elukey@deploy1002> helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. [production]
13:01 <elukey@deploy1002> helmfile [eqiad] DONE helmfile.d/admin 'sync'. [production]
13:01 <elukey@deploy1002> helmfile [eqiad] START helmfile.d/admin 'sync'. [production]
12:58 <elukey@deploy1002> helmfile [eqiad] DONE helmfile.d/admin 'sync'. [production]
12:58 <elukey@deploy1002> helmfile [eqiad] START helmfile.d/admin 'sync'. [production]
12:55 <fabfur> repooling text@drmrs (IPIP encapsulation enabled) (T366466) [production]
12:53 <brouberol@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: sync on staging [production]
12:50 <elukey@deploy1002> helmfile [eqiad] DONE helmfile.d/admin 'sync'. [production]
12:50 <brouberol@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply on staging [production]
12:49 <elukey@deploy1002> helmfile [eqiad] START helmfile.d/admin 'sync'. [production]
12:48 <elukey@deploy1002> helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. [production]
12:46 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1018.eqiad.wmnet [production]
12:46 <elukey@deploy1002> helmfile [staging-codfw] START helmfile.d/admin 'sync'. [production]
12:45 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2021.codfw.wmnet [production]
12:44 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2020.codfw.wmnet [production]
12:44 <elukey@deploy1002> helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. [production]
12:43 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2020.codfw.wmnet [production]
12:43 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1017.eqiad.wmnet [production]
12:43 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1017.eqiad.wmnet [production]
12:43 <elukey@deploy1002> helmfile [staging-eqiad] START helmfile.d/admin 'sync'. [production]
12:41 <elukey@deploy1002> helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'. [production]
12:40 <elukey@deploy1002> helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'. [production]
12:37 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti2020.codfw.wmnet [production]
12:37 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti1017.eqiad.wmnet [production]
12:30 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2020.codfw.wmnet [production]
12:28 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2204 (re)pooling @ 100%: post maintenance repool', diff saved to https://phabricator.wikimedia.org/P64532 and previous config saved to /var/cache/conftool/dbconfig/20240610-122847-arnaudb.json [production]