3501-3550 of 10000 results (81ms)
2023-01-30 ยง
16:23 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P43510 and previous config saved to /var/cache/conftool/dbconfig/20230130-162336-ladsgroup.json [production]
16:22 <sukhe@puppetmaster1001> conftool action : set/pooled=yes; selector: name=cp3051.esams.wmnet,service=ats-be [production]
16:22 <sukhe@puppetmaster1001> conftool action : set/pooled=yes; selector: name=cp3051.esams.wmnet,service=cdn [production]
16:22 <sukhe@puppetmaster1001> conftool action : set/pooled=yes; selector: name=cp2029.codfw.wmnet,service=ats-be [production]
16:22 <sukhe@puppetmaster1001> conftool action : set/pooled=yes; selector: name=cp2029.codfw.wmnet,service=cdn [production]
16:21 <sukhe@cumin2002> START - Cookbook sre.hosts.reimage for host cp3050.esams.wmnet with OS bullseye [production]
16:17 <sukhe@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp3050.esams.wmnet [production]
16:16 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp2029.codfw.wmnet with OS bullseye [production]
16:13 <marostegui@cumin1001> dbctl commit (dc=all): 'db2110 (re)pooling @ 100%: After switchover', diff saved to https://phabricator.wikimedia.org/P43509 and previous config saved to /var/cache/conftool/dbconfig/20230130-161324-root.json [production]
16:11 <sukhe@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp3050.esams.wmnet [production]
16:10 <sukhe@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp3050.esams.wmnet [production]
16:10 <sukhe@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp3050.esams.wmnet [production]
16:10 <sukhe@puppetmaster1001> conftool action : set/pooled=no; selector: name=cp3050.esams.wmnet,service=ats-be [production]
16:10 <sukhe@puppetmaster1001> conftool action : set/pooled=no; selector: name=cp3050.esams.wmnet,service=cdn [production]
16:08 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2156 (T328255)', diff saved to https://phabricator.wikimedia.org/P43508 and previous config saved to /var/cache/conftool/dbconfig/20230130-160829-ladsgroup.json [production]
16:06 <sukhe@cumin2002> START - Cookbook sre.hosts.reimage for host cp5026.eqsin.wmnet with OS bullseye [production]
16:05 <sukhe@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5026.eqsin.wmnet with OS bullseye [production]
16:03 <sukhe> racreset cp3050.esams.wmnet: firmware cookbook iDRAC upgrade test [production]
16:03 <moritzm> upgrading idp-test to latest Java security update [production]
15:59 <sukhe@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp3050.esams.wmnet [production]
15:59 <sukhe@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp3050.esams.wmnet [production]
15:58 <marostegui@cumin1001> dbctl commit (dc=all): 'db2110 (re)pooling @ 75%: After switchover', diff saved to https://phabricator.wikimedia.org/P43507 and previous config saved to /var/cache/conftool/dbconfig/20230130-155819-root.json [production]
15:58 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2156 (T328255)', diff saved to https://phabricator.wikimedia.org/P43506 and previous config saved to /var/cache/conftool/dbconfig/20230130-155802-ladsgroup.json [production]
15:57 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2094.codfw.wmnet with reason: Maintenance [production]
15:57 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2094.codfw.wmnet with reason: Maintenance [production]
15:57 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2156.codfw.wmnet with reason: Maintenance [production]
15:57 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db2156.codfw.wmnet with reason: Maintenance [production]
15:57 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2149 (T328255)', diff saved to https://phabricator.wikimedia.org/P43505 and previous config saved to /var/cache/conftool/dbconfig/20230130-155747-ladsgroup.json [production]
15:54 <sukhe@cumin2002> START - Cookbook sre.hosts.reimage for host cp5026.eqsin.wmnet with OS bullseye [production]
15:51 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp2029.codfw.wmnet with reason: host reimage [production]
15:48 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on cp2029.codfw.wmnet with reason: host reimage [production]
15:43 <marostegui@cumin1001> dbctl commit (dc=all): 'db2110 (re)pooling @ 50%: After switchover', diff saved to https://phabricator.wikimedia.org/P43504 and previous config saved to /var/cache/conftool/dbconfig/20230130-154314-root.json [production]
15:42 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P43503 and previous config saved to /var/cache/conftool/dbconfig/20230130-154241-ladsgroup.json [production]
15:31 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3051.esams.wmnet with OS bullseye [production]
15:29 <sukhe@cumin2002> START - Cookbook sre.hosts.reimage for host cp2029.codfw.wmnet with OS bullseye [production]
15:28 <marostegui@cumin1001> dbctl commit (dc=all): 'db2110 (re)pooling @ 25%: After switchover', diff saved to https://phabricator.wikimedia.org/P43502 and previous config saved to /var/cache/conftool/dbconfig/20230130-152809-root.json [production]
15:27 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P43501 and previous config saved to /var/cache/conftool/dbconfig/20230130-152734-ladsgroup.json [production]
15:13 <marostegui> Retrospective: Starting s4 codfw failover from db2110 to db2140 - T328022 [production]
15:13 <marostegui@cumin1001> dbctl commit (dc=all): 'db2110 (re)pooling @ 10%: After switchover', diff saved to https://phabricator.wikimedia.org/P43500 and previous config saved to /var/cache/conftool/dbconfig/20230130-151304-root.json [production]
15:12 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2149 (T328255)', diff saved to https://phabricator.wikimedia.org/P43499 and previous config saved to /var/cache/conftool/dbconfig/20230130-151228-ladsgroup.json [production]
15:07 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3051.esams.wmnet with reason: host reimage [production]
15:04 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on cp3051.esams.wmnet with reason: host reimage [production]
15:01 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2149 (T328255)', diff saved to https://phabricator.wikimedia.org/P43498 and previous config saved to /var/cache/conftool/dbconfig/20230130-150132-ladsgroup.json [production]
15:01 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2149.codfw.wmnet with reason: Maintenance [production]
15:01 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db2149.codfw.wmnet with reason: Maintenance [production]
14:58 <marostegui@cumin1001> dbctl commit (dc=all): 'db2110 (re)pooling @ 5%: After switchover', diff saved to https://phabricator.wikimedia.org/P43497 and previous config saved to /var/cache/conftool/dbconfig/20230130-145759-root.json [production]
14:55 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2110 T328022', diff saved to https://phabricator.wikimedia.org/P43496 and previous config saved to /var/cache/conftool/dbconfig/20230130-145508-root.json [production]
14:54 <marostegui@cumin1001> dbctl commit (dc=all): 'Promote db2140 to s4 primary T328022', diff saved to https://phabricator.wikimedia.org/P43495 and previous config saved to /var/cache/conftool/dbconfig/20230130-145421-root.json [production]
14:52 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2139.codfw.wmnet with reason: Maintenance [production]
14:52 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db2139.codfw.wmnet with reason: Maintenance [production]