|
2024-02-20
ยง
|
| 15:46 |
<denisse> |
When doing the alert hosts upgrade we encountered some issues that prevented us to properly reimage the hosts to proceed with the upgrade. We're investigating this issue and inform of the new alert hosts upgrade date ASAP. - T333615 |
[production] |
| 15:46 |
<denisse> |
When doing the alert hosts upgrade we encountered some issues that prevented us to properly reimage the hosts to proceed with the upgrade. We're investigating this issue and inform of the new alert hosts upgrade date ASAP. - T333615 |
[production] |
| 15:46 |
<godog> |
re-enable meta-monitoring on wikitech-static.w.o - T333615 |
[production] |
| 15:43 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P57361 and previous config saved to /var/cache/conftool/dbconfig/20240220-154313-marostegui.json |
[production] |
| 15:42 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1233.eqiad.wmnet |
[production] |
| 15:41 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1168.eqiad.wmnet |
[production] |
| 15:41 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1226.eqiad.wmnet |
[production] |
| 15:41 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1210.eqiad.wmnet |
[production] |
| 15:37 |
<arnaudb@cumin1002> |
START - Cookbook sre.mysql.upgrade for db1233.eqiad.wmnet |
[production] |
| 15:37 |
<arnaudb@cumin1002> |
START - Cookbook sre.mysql.upgrade for db1226.eqiad.wmnet |
[production] |
| 15:37 |
<arnaudb@cumin1002> |
START - Cookbook sre.mysql.upgrade for db1210.eqiad.wmnet |
[production] |
| 15:36 |
<arnaudb@cumin1002> |
START - Cookbook sre.mysql.upgrade for db1168.eqiad.wmnet |
[production] |
| 15:35 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1168 db1210 db1226 db1233 depool for T356240', diff saved to https://phabricator.wikimedia.org/P57359 and previous config saved to /var/cache/conftool/dbconfig/20240220-153557-arnaudb.json |
[production] |
| 15:34 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2128 (T357189)', diff saved to https://phabricator.wikimedia.org/P57358 and previous config saved to /var/cache/conftool/dbconfig/20240220-153410-arnaudb.json |
[production] |
| 15:33 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on db[1168,1210,1226,1233].eqiad.wmnet with reason: Silence for reboot T356240 |
[production] |
| 15:33 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 0:30:00 on db[1168,1210,1226,1233].eqiad.wmnet with reason: Silence for reboot T356240 |
[production] |
| 15:32 |
<godog> |
temp disable meta-monitoring on wikitech-static.w.o - T333615 |
[production] |
| 15:30 |
<Emperor> |
import ceph-reef packages to apt1001 T279621 |
[production] |
| 15:30 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Depooling db2128 (T357189)', diff saved to https://phabricator.wikimedia.org/P57357 and previous config saved to /var/cache/conftool/dbconfig/20240220-153000-arnaudb.json |
[production] |
| 15:29 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance |
[production] |
| 15:29 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance |
[production] |
| 15:29 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2128.codfw.wmnet with reason: Maintenance |
[production] |
| 15:29 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2128.codfw.wmnet with reason: Maintenance |
[production] |
| 15:29 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2123 (T357189)', diff saved to https://phabricator.wikimedia.org/P57356 and previous config saved to /var/cache/conftool/dbconfig/20240220-152933-arnaudb.json |
[production] |
| 15:28 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1207 (T355609)', diff saved to https://phabricator.wikimedia.org/P57355 and previous config saved to /var/cache/conftool/dbconfig/20240220-152807-marostegui.json |
[production] |
| 15:25 |
<ayounsi@cumin1002> |
START - Cookbook sre.hosts.reimage for host sretest2005.codfw.wmnet with OS bookworm |
[production] |
| 15:18 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2170 (re)pooling @ 100%: After migration', diff saved to https://phabricator.wikimedia.org/P57354 and previous config saved to /var/cache/conftool/dbconfig/20240220-151812-root.json |
[production] |
| 15:16 |
<dcausse> |
depooled wdqs2009 & wdqs2020 (T355867) |
[production] |
| 15:16 |
<denisse_> |
starting the Alert hosts upgrade to Bookworm - T333615 |
[production] |
| 15:16 |
<denisse_> |
starting the Alert hosts upgrade to Bookworm - T333615 |
[production] |
| 15:14 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2123', diff saved to https://phabricator.wikimedia.org/P57353 and previous config saved to /var/cache/conftool/dbconfig/20240220-151426-arnaudb.json |
[production] |
| 15:13 |
<ayounsi@cumin1002> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2005.codfw.wmnet with OS bookworm |
[production] |
| 15:13 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db[2146,2151].codfw.wmnet |
[production] |
| 14:55 |
<bking@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/rdf-streaming-updater: apply |
[production] |
| 14:55 |
<bking@deploy2002> |
helmfile [codfw] START helmfile.d/services/rdf-streaming-updater: apply |
[production] |
| 14:51 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2194 (re)pooling @ 75%: After rearraging sections T354826', diff saved to https://phabricator.wikimedia.org/P57346 and previous config saved to /var/cache/conftool/dbconfig/20240220-145124-root.json |
[production] |
| 14:50 |
<bking@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/rdf-streaming-updater: apply |
[production] |
| 14:49 |
<sukhe> |
disable puppet on A:cp to merge CR 1004126 |
[production] |
| 14:49 |
<brett@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on cp[2029-2030].codfw.wmnet with reason: T355867 |
[production] |
| 14:49 |
<bking@deploy2002> |
helmfile [eqiad] START helmfile.d/services/rdf-streaming-updater: apply |
[production] |
| 14:49 |
<brett@cumin2002> |
START - Cookbook sre.hosts.downtime for 3:00:00 on cp[2029-2030].codfw.wmnet with reason: T355867 |
[production] |
| 14:48 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1231.eqiad.wmnet |
[production] |
| 14:48 |
<bking@deploy2002> |
helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply |
[production] |
| 14:48 |
<bking@deploy2002> |
helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply |
[production] |
| 14:48 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2170 (re)pooling @ 50%: After migration', diff saved to https://phabricator.wikimedia.org/P57345 and previous config saved to /var/cache/conftool/dbconfig/20240220-144803-root.json |
[production] |
| 14:48 |
<bking@deploy2002> |
helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply |
[production] |
| 14:48 |
<brett@puppetmaster1001> |
conftool action : set/pooled=no; selector: dc=codfw,name=cp20(29|30).codfw.wmnet |
[production] |
| 14:48 |
<bking@deploy2002> |
helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply |
[production] |
| 14:47 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2190 (re)pooling @ 75%: After rearraging sections T354826', diff saved to https://phabricator.wikimedia.org/P57344 and previous config saved to /var/cache/conftool/dbconfig/20240220-144753-root.json |
[production] |
| 14:46 |
<sukhe> |
updating pdns-recursor to 4.8.6-1 on dns* |
[production] |