|
2023-05-08
ยง
|
| 17:27 |
<pt1979@cumin2002> |
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudswift1001.mgmt.eqiad.wmnet with reboot policy FORCED |
[production] |
| 17:27 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1140.eqiad.wmnet with reason: Maintenance |
[production] |
| 17:17 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1214 (T335845)', diff saved to https://phabricator.wikimedia.org/P47917 and previous config saved to /var/cache/conftool/dbconfig/20230508-171720-ladsgroup.json |
[production] |
| 17:16 |
<sukhe@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs2011.codfw.wmnet with OS bullseye |
[production] |
| 17:09 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1214 (T335845)', diff saved to https://phabricator.wikimedia.org/P47916 and previous config saved to /var/cache/conftool/dbconfig/20230508-170902-ladsgroup.json |
[production] |
| 17:08 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1214.eqiad.wmnet with reason: Maintenance |
[production] |
| 17:08 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1214.eqiad.wmnet with reason: Maintenance |
[production] |
| 17:08 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1211 (T335845)', diff saved to https://phabricator.wikimedia.org/P47915 and previous config saved to /var/cache/conftool/dbconfig/20230508-170828-ladsgroup.json |
[production] |
| 17:05 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2181 (T335845)', diff saved to https://phabricator.wikimedia.org/P47914 and previous config saved to /var/cache/conftool/dbconfig/20230508-170542-ladsgroup.json |
[production] |
| 16:58 |
<sukhe@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs2011.codfw.wmnet with reason: host reimage |
[production] |
| 16:55 |
<sukhe@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on lvs2011.codfw.wmnet with reason: host reimage |
[production] |
| 16:53 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1211', diff saved to https://phabricator.wikimedia.org/P47913 and previous config saved to /var/cache/conftool/dbconfig/20230508-165322-ladsgroup.json |
[production] |
| 16:50 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P47912 and previous config saved to /var/cache/conftool/dbconfig/20230508-165036-ladsgroup.json |
[production] |
| 16:46 |
<volans> |
uploaded spicerack_7.0.0 to apt.wikimedia.org bullseye-wikimedia |
[production] |
| 16:39 |
<sukhe@cumin2002> |
START - Cookbook sre.hosts.reimage for host lvs2011.codfw.wmnet with OS bullseye |
[production] |
| 16:39 |
<sukhe@cumin2002> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host lvs2011.codfw.wmnet with OS bullseye |
[production] |
| 16:39 |
<sukhe@cumin2002> |
START - Cookbook sre.hosts.reimage for host lvs2011.codfw.wmnet with OS bullseye |
[production] |
| 16:38 |
<sukhe@cumin2002> |
END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host lvs2011.codfw.wmnet with OS bullseye |
[production] |
| 16:38 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1211', diff saved to https://phabricator.wikimedia.org/P47910 and previous config saved to /var/cache/conftool/dbconfig/20230508-163816-ladsgroup.json |
[production] |
| 16:35 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P47909 and previous config saved to /var/cache/conftool/dbconfig/20230508-163530-ladsgroup.json |
[production] |
| 16:33 |
<otto@deploy1002> |
helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 16:33 |
<otto@deploy1002> |
helmfile [staging-eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 16:32 |
<otto@deploy1002> |
helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 16:32 |
<otto@deploy1002> |
helmfile [staging-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 16:23 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1211 (T335845)', diff saved to https://phabricator.wikimedia.org/P47908 and previous config saved to /var/cache/conftool/dbconfig/20230508-162309-ladsgroup.json |
[production] |
| 16:20 |
<sukhe@cumin2002> |
START - Cookbook sre.hosts.reimage for host lvs2011.codfw.wmnet with OS bullseye |
[production] |
| 16:20 |
<sukhe@cumin2002> |
END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host lvs2011.codfw.wmnet with OS bullseye |
[production] |
| 16:20 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2181 (T335845)', diff saved to https://phabricator.wikimedia.org/P47907 and previous config saved to /var/cache/conftool/dbconfig/20230508-162024-ladsgroup.json |
[production] |
| 16:14 |
<sukhe@cumin2002> |
START - Cookbook sre.hosts.reimage for host lvs2011.codfw.wmnet with OS bullseye |
[production] |
| 16:13 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2181 (T335845)', diff saved to https://phabricator.wikimedia.org/P47906 and previous config saved to /var/cache/conftool/dbconfig/20230508-161313-ladsgroup.json |
[production] |
| 16:13 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2181.codfw.wmnet with reason: Maintenance |
[production] |
| 16:12 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1211 (T335845)', diff saved to https://phabricator.wikimedia.org/P47905 and previous config saved to /var/cache/conftool/dbconfig/20230508-161258-ladsgroup.json |
[production] |
| 16:12 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1211.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:12 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2181.codfw.wmnet with reason: Maintenance |
[production] |
| 16:12 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1211.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:12 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2167:3318 (T335845)', diff saved to https://phabricator.wikimedia.org/P47904 and previous config saved to /var/cache/conftool/dbconfig/20230508-161235-ladsgroup.json |
[production] |
| 16:12 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1209 (T335845)', diff saved to https://phabricator.wikimedia.org/P47903 and previous config saved to /var/cache/conftool/dbconfig/20230508-161234-ladsgroup.json |
[production] |
| 16:11 |
<sukhe@deploy1002> |
Locking from deployment [ALL REPOSITORIES]: LVS reimaging in codfw, blocking deploys T326767 |
[production] |
| 16:02 |
<bking@cumin1001> |
conftool action : set/pooled=false; selector: dnsdisc=wdqs,name=codfw |
[production] |
| 15:57 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2167:3318', diff saved to https://phabricator.wikimedia.org/P47902 and previous config saved to /var/cache/conftool/dbconfig/20230508-155729-ladsgroup.json |
[production] |
| 15:57 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P47901 and previous config saved to /var/cache/conftool/dbconfig/20230508-155728-ladsgroup.json |
[production] |
| 15:47 |
<otto@deploy1002> |
helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 15:46 |
<otto@deploy1002> |
helmfile [staging-eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 15:42 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2167:3318', diff saved to https://phabricator.wikimedia.org/P47900 and previous config saved to /var/cache/conftool/dbconfig/20230508-154222-ladsgroup.json |
[production] |
| 15:42 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P47899 and previous config saved to /var/cache/conftool/dbconfig/20230508-154222-ladsgroup.json |
[production] |
| 15:41 |
<otto@deploy1002> |
helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 15:39 |
<otto@deploy1002> |
helmfile [staging-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 15:32 |
<sukhe> |
ns1: remove dns2001, add dns2004 next-hop [ 208.80.153.48 208.80.153.111 208.80.153.10 ]: T335777 |
[production] |
| 15:27 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2167:3318 (T335845)', diff saved to https://phabricator.wikimedia.org/P47898 and previous config saved to /var/cache/conftool/dbconfig/20230508-152716-ladsgroup.json |
[production] |
| 15:27 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1209 (T335845)', diff saved to https://phabricator.wikimedia.org/P47897 and previous config saved to /var/cache/conftool/dbconfig/20230508-152716-ladsgroup.json |
[production] |