1001-1050 of 10000 results (78ms)
2023-01-30 ยง
17:22 <inflatador> bking@build2001 rebuilding docker images for 884351 [production]
17:21 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5026.eqsin.wmnet with OS bullseye [production]
17:19 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P43515 and previous config saved to /var/cache/conftool/dbconfig/20230130-171944-ladsgroup.json [production]
17:12 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3050.esams.wmnet with OS bullseye [production]
17:04 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2177 (T328255)', diff saved to https://phabricator.wikimedia.org/P43514 and previous config saved to /var/cache/conftool/dbconfig/20230130-170437-ladsgroup.json [production]
16:59 <brett@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4043.ulsfo.wmnet with reason: host reimage [production]
16:56 <brett@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cp4043.ulsfo.wmnet with reason: host reimage [production]
16:53 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2177 (T328255)', diff saved to https://phabricator.wikimedia.org/P43513 and previous config saved to /var/cache/conftool/dbconfig/20230130-165359-ladsgroup.json [production]
16:53 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2177.codfw.wmnet with reason: Maintenance [production]
16:53 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db2177.codfw.wmnet with reason: Maintenance [production]
16:53 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2156 (T328255)', diff saved to https://phabricator.wikimedia.org/P43512 and previous config saved to /var/cache/conftool/dbconfig/20230130-165348-ladsgroup.json [production]
16:50 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5026.eqsin.wmnet with reason: host reimage [production]
16:48 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3050.esams.wmnet with reason: host reimage [production]
16:44 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on cp5026.eqsin.wmnet with reason: host reimage [production]
16:44 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on cp3050.esams.wmnet with reason: host reimage [production]
16:38 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P43511 and previous config saved to /var/cache/conftool/dbconfig/20230130-163842-ladsgroup.json [production]
16:35 <brett@cumin1001> START - Cookbook sre.hosts.reimage for host cp4043.ulsfo.wmnet with OS bullseye [production]
16:35 <brett@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4043.ulsfo.wmnet with OS bullseye [production]
16:30 <btullis@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1084.eqiad.wmnet [production]
16:25 <brett@cumin1001> START - Cookbook sre.hosts.reimage for host cp4043.ulsfo.wmnet with OS bullseye [production]
16:24 <btullis@cumin1001> START - Cookbook sre.hosts.reboot-single for host an-worker1084.eqiad.wmnet [production]
16:23 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P43510 and previous config saved to /var/cache/conftool/dbconfig/20230130-162336-ladsgroup.json [production]
16:22 <sukhe@puppetmaster1001> conftool action : set/pooled=yes; selector: name=cp3051.esams.wmnet,service=ats-be [production]
16:22 <sukhe@puppetmaster1001> conftool action : set/pooled=yes; selector: name=cp3051.esams.wmnet,service=cdn [production]
16:22 <sukhe@puppetmaster1001> conftool action : set/pooled=yes; selector: name=cp2029.codfw.wmnet,service=ats-be [production]
16:22 <sukhe@puppetmaster1001> conftool action : set/pooled=yes; selector: name=cp2029.codfw.wmnet,service=cdn [production]
16:21 <sukhe@cumin2002> START - Cookbook sre.hosts.reimage for host cp3050.esams.wmnet with OS bullseye [production]
16:17 <sukhe@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp3050.esams.wmnet [production]
16:16 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp2029.codfw.wmnet with OS bullseye [production]
16:13 <marostegui@cumin1001> dbctl commit (dc=all): 'db2110 (re)pooling @ 100%: After switchover', diff saved to https://phabricator.wikimedia.org/P43509 and previous config saved to /var/cache/conftool/dbconfig/20230130-161324-root.json [production]
16:11 <sukhe@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp3050.esams.wmnet [production]
16:10 <sukhe@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp3050.esams.wmnet [production]
16:10 <sukhe@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp3050.esams.wmnet [production]
16:10 <sukhe@puppetmaster1001> conftool action : set/pooled=no; selector: name=cp3050.esams.wmnet,service=ats-be [production]
16:10 <sukhe@puppetmaster1001> conftool action : set/pooled=no; selector: name=cp3050.esams.wmnet,service=cdn [production]
16:08 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2156 (T328255)', diff saved to https://phabricator.wikimedia.org/P43508 and previous config saved to /var/cache/conftool/dbconfig/20230130-160829-ladsgroup.json [production]
16:06 <sukhe@cumin2002> START - Cookbook sre.hosts.reimage for host cp5026.eqsin.wmnet with OS bullseye [production]
16:05 <sukhe@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5026.eqsin.wmnet with OS bullseye [production]
16:03 <sukhe> racreset cp3050.esams.wmnet: firmware cookbook iDRAC upgrade test [production]
16:03 <moritzm> upgrading idp-test to latest Java security update [production]
15:59 <sukhe@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp3050.esams.wmnet [production]
15:59 <sukhe@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp3050.esams.wmnet [production]
15:58 <marostegui@cumin1001> dbctl commit (dc=all): 'db2110 (re)pooling @ 75%: After switchover', diff saved to https://phabricator.wikimedia.org/P43507 and previous config saved to /var/cache/conftool/dbconfig/20230130-155819-root.json [production]
15:58 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2156 (T328255)', diff saved to https://phabricator.wikimedia.org/P43506 and previous config saved to /var/cache/conftool/dbconfig/20230130-155802-ladsgroup.json [production]
15:57 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2094.codfw.wmnet with reason: Maintenance [production]
15:57 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2094.codfw.wmnet with reason: Maintenance [production]
15:57 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2156.codfw.wmnet with reason: Maintenance [production]
15:57 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db2156.codfw.wmnet with reason: Maintenance [production]
15:57 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2149 (T328255)', diff saved to https://phabricator.wikimedia.org/P43505 and previous config saved to /var/cache/conftool/dbconfig/20230130-155747-ladsgroup.json [production]
15:54 <sukhe@cumin2002> START - Cookbook sre.hosts.reimage for host cp5026.eqsin.wmnet with OS bullseye [production]