4201-4250 of 10000 results (89ms)
2023-01-30 ยง
16:53 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db2177.codfw.wmnet with reason: Maintenance [production]
16:53 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2156 (T328255)', diff saved to https://phabricator.wikimedia.org/P43512 and previous config saved to /var/cache/conftool/dbconfig/20230130-165348-ladsgroup.json [production]
16:50 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5026.eqsin.wmnet with reason: host reimage [production]
16:48 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3050.esams.wmnet with reason: host reimage [production]
16:44 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on cp5026.eqsin.wmnet with reason: host reimage [production]
16:44 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on cp3050.esams.wmnet with reason: host reimage [production]
16:38 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P43511 and previous config saved to /var/cache/conftool/dbconfig/20230130-163842-ladsgroup.json [production]
16:35 <brett@cumin1001> START - Cookbook sre.hosts.reimage for host cp4043.ulsfo.wmnet with OS bullseye [production]
16:35 <brett@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4043.ulsfo.wmnet with OS bullseye [production]
16:30 <btullis@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1084.eqiad.wmnet [production]
16:25 <brett@cumin1001> START - Cookbook sre.hosts.reimage for host cp4043.ulsfo.wmnet with OS bullseye [production]
16:24 <btullis@cumin1001> START - Cookbook sre.hosts.reboot-single for host an-worker1084.eqiad.wmnet [production]
16:23 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P43510 and previous config saved to /var/cache/conftool/dbconfig/20230130-162336-ladsgroup.json [production]
16:22 <sukhe@puppetmaster1001> conftool action : set/pooled=yes; selector: name=cp3051.esams.wmnet,service=ats-be [production]
16:22 <sukhe@puppetmaster1001> conftool action : set/pooled=yes; selector: name=cp3051.esams.wmnet,service=cdn [production]
16:22 <sukhe@puppetmaster1001> conftool action : set/pooled=yes; selector: name=cp2029.codfw.wmnet,service=ats-be [production]
16:22 <sukhe@puppetmaster1001> conftool action : set/pooled=yes; selector: name=cp2029.codfw.wmnet,service=cdn [production]
16:21 <sukhe@cumin2002> START - Cookbook sre.hosts.reimage for host cp3050.esams.wmnet with OS bullseye [production]
16:17 <sukhe@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp3050.esams.wmnet [production]
16:16 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp2029.codfw.wmnet with OS bullseye [production]
16:13 <marostegui@cumin1001> dbctl commit (dc=all): 'db2110 (re)pooling @ 100%: After switchover', diff saved to https://phabricator.wikimedia.org/P43509 and previous config saved to /var/cache/conftool/dbconfig/20230130-161324-root.json [production]
16:11 <sukhe@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp3050.esams.wmnet [production]
16:10 <sukhe@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp3050.esams.wmnet [production]
16:10 <sukhe@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp3050.esams.wmnet [production]
16:10 <sukhe@puppetmaster1001> conftool action : set/pooled=no; selector: name=cp3050.esams.wmnet,service=ats-be [production]
16:10 <sukhe@puppetmaster1001> conftool action : set/pooled=no; selector: name=cp3050.esams.wmnet,service=cdn [production]
16:08 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2156 (T328255)', diff saved to https://phabricator.wikimedia.org/P43508 and previous config saved to /var/cache/conftool/dbconfig/20230130-160829-ladsgroup.json [production]
16:06 <sukhe@cumin2002> START - Cookbook sre.hosts.reimage for host cp5026.eqsin.wmnet with OS bullseye [production]
16:05 <sukhe@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5026.eqsin.wmnet with OS bullseye [production]
16:03 <sukhe> racreset cp3050.esams.wmnet: firmware cookbook iDRAC upgrade test [production]
16:03 <moritzm> upgrading idp-test to latest Java security update [production]
15:59 <sukhe@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp3050.esams.wmnet [production]
15:59 <sukhe@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp3050.esams.wmnet [production]
15:58 <marostegui@cumin1001> dbctl commit (dc=all): 'db2110 (re)pooling @ 75%: After switchover', diff saved to https://phabricator.wikimedia.org/P43507 and previous config saved to /var/cache/conftool/dbconfig/20230130-155819-root.json [production]
15:58 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2156 (T328255)', diff saved to https://phabricator.wikimedia.org/P43506 and previous config saved to /var/cache/conftool/dbconfig/20230130-155802-ladsgroup.json [production]
15:57 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2094.codfw.wmnet with reason: Maintenance [production]
15:57 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2094.codfw.wmnet with reason: Maintenance [production]
15:57 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2156.codfw.wmnet with reason: Maintenance [production]
15:57 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db2156.codfw.wmnet with reason: Maintenance [production]
15:57 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2149 (T328255)', diff saved to https://phabricator.wikimedia.org/P43505 and previous config saved to /var/cache/conftool/dbconfig/20230130-155747-ladsgroup.json [production]
15:54 <sukhe@cumin2002> START - Cookbook sre.hosts.reimage for host cp5026.eqsin.wmnet with OS bullseye [production]
15:51 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp2029.codfw.wmnet with reason: host reimage [production]
15:48 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on cp2029.codfw.wmnet with reason: host reimage [production]
15:43 <marostegui@cumin1001> dbctl commit (dc=all): 'db2110 (re)pooling @ 50%: After switchover', diff saved to https://phabricator.wikimedia.org/P43504 and previous config saved to /var/cache/conftool/dbconfig/20230130-154314-root.json [production]
15:42 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P43503 and previous config saved to /var/cache/conftool/dbconfig/20230130-154241-ladsgroup.json [production]
15:31 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3051.esams.wmnet with OS bullseye [production]
15:29 <sukhe@cumin2002> START - Cookbook sre.hosts.reimage for host cp2029.codfw.wmnet with OS bullseye [production]
15:28 <marostegui@cumin1001> dbctl commit (dc=all): 'db2110 (re)pooling @ 25%: After switchover', diff saved to https://phabricator.wikimedia.org/P43502 and previous config saved to /var/cache/conftool/dbconfig/20230130-152809-root.json [production]
15:27 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P43501 and previous config saved to /var/cache/conftool/dbconfig/20230130-152734-ladsgroup.json [production]
15:13 <marostegui> Retrospective: Starting s4 codfw failover from db2110 to db2140 - T328022 [production]