6551-6600 of 10000 results (104ms)
2023-12-13 ยง
10:07 <arnaudb@cumin1001> dbctl commit (dc=all): 'db1229 (re)pooling @ 100%: Post reboot repooling', diff saved to https://phabricator.wikimedia.org/P54374 and previous config saved to /var/cache/conftool/dbconfig/20231213-100708-arnaudb.json [production]
10:06 <arnaudb@cumin1001> dbctl commit (dc=all): 'db1228 (re)pooling @ 100%: Post clone repooling', diff saved to https://phabricator.wikimedia.org/P54373 and previous config saved to /var/cache/conftool/dbconfig/20231213-100651-arnaudb.json [production]
10:05 <arnaudb@cumin1001> dbctl commit (dc=all): 'db1247 (re)pooling @ 100%: Post clone repooling', diff saved to https://phabricator.wikimedia.org/P54372 and previous config saved to /var/cache/conftool/dbconfig/20231213-100555-arnaudb.json [production]
10:00 <moritzm> failover ganeti master in eqsin to ganeti5007 [production]
09:59 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet [production]
09:57 <arnaudb@cumin1001> START - Cookbook sre.hosts.reimage for host db1226.eqiad.wmnet with OS bookworm [production]
09:56 <hashar> Disabled puppet agent on contint1002, contint2002, releases1003 and releases2003 to progressively deploy https://gerrit.wikimedia.org/r/922555 [production]
09:52 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet [production]
09:52 <arnaudb@cumin1001> dbctl commit (dc=all): 'db1229 (re)pooling @ 90%: Post reboot repooling', diff saved to https://phabricator.wikimedia.org/P54371 and previous config saved to /var/cache/conftool/dbconfig/20231213-095203-arnaudb.json [production]
09:51 <arnaudb@cumin1001> dbctl commit (dc=all): 'db1228 (re)pooling @ 90%: Post clone repooling', diff saved to https://phabricator.wikimedia.org/P54370 and previous config saved to /var/cache/conftool/dbconfig/20231213-095146-arnaudb.json [production]
09:50 <arnaudb@cumin1001> dbctl commit (dc=all): 'db1247 (re)pooling @ 90%: Post clone repooling', diff saved to https://phabricator.wikimedia.org/P54369 and previous config saved to /var/cache/conftool/dbconfig/20231213-095049-arnaudb.json [production]
09:49 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5005.eqsin.wmnet [production]
09:42 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5005.eqsin.wmnet [production]
09:41 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet [production]
09:36 <arnaudb@cumin1001> dbctl commit (dc=all): 'db1229 (re)pooling @ 80%: Post reboot repooling', diff saved to https://phabricator.wikimedia.org/P54368 and previous config saved to /var/cache/conftool/dbconfig/20231213-093658-arnaudb.json [production]
09:36 <arnaudb@cumin1001> dbctl commit (dc=all): 'db1228 (re)pooling @ 80%: Post clone repooling', diff saved to https://phabricator.wikimedia.org/P54367 and previous config saved to /var/cache/conftool/dbconfig/20231213-093641-arnaudb.json [production]
09:35 <arnaudb@cumin1001> dbctl commit (dc=all): 'db1247 (re)pooling @ 80%: Post clone repooling', diff saved to https://phabricator.wikimedia.org/P54366 and previous config saved to /var/cache/conftool/dbconfig/20231213-093544-arnaudb.json [production]
09:33 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet [production]
09:30 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet [production]
09:27 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet [production]
09:25 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet [production]
09:25 <klausman@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-staging2001.codfw.wmnet [production]
09:25 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
09:25 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
09:24 <brouberol> increasing pod max requested memory to a higher value than the container max requested memory for dse-k8s-eqiad - T351722 [production]
09:21 <arnaudb@cumin1001> dbctl commit (dc=all): 'db1229 (re)pooling @ 70%: Post reboot repooling', diff saved to https://phabricator.wikimedia.org/P54365 and previous config saved to /var/cache/conftool/dbconfig/20231213-092153-arnaudb.json [production]
09:21 <arnaudb@cumin1001> dbctl commit (dc=all): 'db1228 (re)pooling @ 70%: Post clone repooling', diff saved to https://phabricator.wikimedia.org/P54364 and previous config saved to /var/cache/conftool/dbconfig/20231213-092136-arnaudb.json [production]
09:20 <arnaudb@cumin1001> dbctl commit (dc=all): 'db1247 (re)pooling @ 70%: Post clone repooling', diff saved to https://phabricator.wikimedia.org/P54363 and previous config saved to /var/cache/conftool/dbconfig/20231213-092039-arnaudb.json [production]
09:20 <klausman@cumin1001> START - Cookbook sre.hosts.reboot-single for host ml-staging2001.codfw.wmnet [production]
09:09 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet [production]
09:06 <arnaudb@cumin1001> dbctl commit (dc=all): 'db1229 (re)pooling @ 60%: Post reboot repooling', diff saved to https://phabricator.wikimedia.org/P54362 and previous config saved to /var/cache/conftool/dbconfig/20231213-090648-arnaudb.json [production]
09:06 <arnaudb@cumin1001> dbctl commit (dc=all): 'db1228 (re)pooling @ 60%: Post clone repooling', diff saved to https://phabricator.wikimedia.org/P54361 and previous config saved to /var/cache/conftool/dbconfig/20231213-090631-arnaudb.json [production]
09:05 <arnaudb@cumin1001> dbctl commit (dc=all): 'db1247 (re)pooling @ 60%: Post clone repooling', diff saved to https://phabricator.wikimedia.org/P54360 and previous config saved to /var/cache/conftool/dbconfig/20231213-090534-arnaudb.json [production]
08:55 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 202120 [production]
08:55 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'email' for AS: 202120 [production]
08:51 <arnaudb@cumin1001> dbctl commit (dc=all): 'db1229 (re)pooling @ 50%: Post reboot repooling', diff saved to https://phabricator.wikimedia.org/P54359 and previous config saved to /var/cache/conftool/dbconfig/20231213-085143-arnaudb.json [production]
08:51 <arnaudb@cumin1001> dbctl commit (dc=all): 'db1228 (re)pooling @ 50%: Post clone repooling', diff saved to https://phabricator.wikimedia.org/P54358 and previous config saved to /var/cache/conftool/dbconfig/20231213-085125-arnaudb.json [production]
08:50 <arnaudb@cumin1001> dbctl commit (dc=all): 'db1247 (re)pooling @ 50%: Post clone repooling', diff saved to https://phabricator.wikimedia.org/P54357 and previous config saved to /var/cache/conftool/dbconfig/20231213-085027-arnaudb.json [production]
08:48 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet [production]
08:48 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 3856 [production]
08:48 <XioNoX> delete bgp group Confed_drmrs from cr1-esams - T347892 [production]
08:46 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'email' for AS: 3856 [production]
08:45 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet [production]
08:44 <ayounsi@cumin1001> END (FAIL) - Cookbook sre.network.peering (exit_code=99) with action 'email' for AS: 46997 [production]
08:43 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'email' for AS: 46997 [production]
08:36 <arnaudb@cumin1001> dbctl commit (dc=all): 'db1229 (re)pooling @ 40%: Post reboot repooling', diff saved to https://phabricator.wikimedia.org/P54356 and previous config saved to /var/cache/conftool/dbconfig/20231213-083638-arnaudb.json [production]
08:36 <arnaudb@cumin1001> dbctl commit (dc=all): 'db1228 (re)pooling @ 40%: Post clone repooling', diff saved to https://phabricator.wikimedia.org/P54355 and previous config saved to /var/cache/conftool/dbconfig/20231213-083620-arnaudb.json [production]
08:35 <arnaudb@cumin1001> dbctl commit (dc=all): 'db1247 (re)pooling @ 40%: Post clone repooling', diff saved to https://phabricator.wikimedia.org/P54354 and previous config saved to /var/cache/conftool/dbconfig/20231213-083522-arnaudb.json [production]
08:30 <XioNoX> delete bgp group Confed_esams from cr2-drmrs - T347892 [production]
08:25 <mlitn@deploy2002> Finished scap: Backport for [[gerrit:979113|No custom UW licensing config]] (duration: 09m 43s) [production]