401-450 of 10000 results (81ms)
2024-06-11 ยง
10:24 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1222.eqiad.wmnet with reason: Maintenance [production]
10:24 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1222.eqiad.wmnet with reason: Maintenance [production]
10:21 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db1206 (T352010)', diff saved to https://phabricator.wikimedia.org/P64607 and previous config saved to /var/cache/conftool/dbconfig/20240611-102125-ladsgroup.json [production]
10:21 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1206.eqiad.wmnet with reason: Maintenance [production]
10:21 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1206.eqiad.wmnet with reason: Maintenance [production]
10:20 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti4006.ulsfo.wmnet [production]
10:18 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1030.eqiad.wmnet [production]
10:16 <jayme@deploy1002> helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply [production]
10:16 <jayme@deploy1002> helmfile [eqiad] START helmfile.d/services/developer-portal: apply [production]
10:16 <jayme@deploy1002> helmfile [codfw] DONE helmfile.d/services/developer-portal: apply [production]
10:16 <fnegri@cumin1002> conftool action : set/pooled=yes; selector: name=clouddb1014.eqiad.wmnet,service=s7 [production]
10:16 <fnegri@cumin1002> conftool action : set/pooled=yes; selector: name=clouddb1014.eqiad.wmnet,service=s2 [production]
10:16 <jayme@deploy1002> helmfile [codfw] START helmfile.d/services/developer-portal: apply [production]
10:15 <jayme@deploy1002> helmfile [staging] DONE helmfile.d/services/developer-portal: apply [production]
10:15 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1029.eqiad.wmnet [production]
10:15 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1029.eqiad.wmnet [production]
10:15 <fnegri@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host clouddb1014.eqiad.wmnet [production]
10:15 <jayme@deploy1002> helmfile [staging] START helmfile.d/services/developer-portal: apply [production]
10:14 <filippo@cumin1002> END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-logging-eqiad [production]
10:14 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1222 (T360332)', diff saved to https://phabricator.wikimedia.org/P64606 and previous config saved to /var/cache/conftool/dbconfig/20240611-101400-arnaudb.json [production]
10:11 <jayme@deploy1002> helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply [production]
10:10 <jayme@deploy1002> helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply [production]
10:10 <jayme@deploy1002> helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply [production]
10:09 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti1029.eqiad.wmnet [production]
10:09 <jayme@deploy1002> helmfile [codfw] START helmfile.d/services/linkrecommendation: apply [production]
10:08 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mx1001.wikimedia.org [production]
10:08 <jayme@deploy1002> helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply [production]
10:08 <jayme@deploy1002> helmfile [staging] START helmfile.d/services/linkrecommendation: apply [production]
10:07 <brouberol@deploy1002> helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. [production]
10:07 <oblivian@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [production]
10:06 <oblivian@deploy1002> helmfile [codfw] START helmfile.d/services/mw-debug: apply [production]
10:06 <brouberol@deploy1002> helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. [production]
10:06 <brouberol@deploy1002> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. [production]
10:06 <brouberol@deploy1002> helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. [production]
10:04 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host mx1001.wikimedia.org [production]
10:04 <brouberol@deploy1002> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. [production]
10:04 <fnegri@cumin1002> START - Cookbook sre.hosts.reboot-single for host clouddb1014.eqiad.wmnet [production]
10:03 <brouberol@deploy1002> helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. [production]
10:02 <brouberol@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
10:02 <brouberol@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
10:02 <brouberol@deploy1002> helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
10:01 <jmm@cumin2002> END (PASS) - Cookbook sre.pki.restart-reboot (exit_code=0) rolling reboot on A:pki [production]
10:01 <brouberol@deploy1002> helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
10:01 <brouberol@deploy1002> helmfile [codfw] DONE helmfile.d/admin 'apply'. [production]
10:00 <brouberol@deploy1002> helmfile [codfw] START helmfile.d/admin 'apply'. [production]
10:00 <sukhe> [end] running authdns-update to send Bolivia (BO) and Paraguay (PY) to magru: T346722 [production]
09:59 <brouberol@deploy1002> helmfile [eqiad] DONE helmfile.d/admin 'apply'. [production]
09:59 <brouberol@deploy1002> helmfile [eqiad] START helmfile.d/admin 'apply'. [production]
09:59 <sukhe> [start] running authdns-update to send Bolivia (BO) and Paraguay (PY) to magru [production]
09:58 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P64605 and previous config saved to /var/cache/conftool/dbconfig/20240611-095853-arnaudb.json [production]