|
2024-07-16
ยง
|
| 21:18 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 4:00:00 on db2194.codfw.wmnet with reason: Maintenance |
[production] |
| 21:18 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2190 (T367781)', diff saved to https://phabricator.wikimedia.org/P66689 and previous config saved to /var/cache/conftool/dbconfig/20240716-211852-arnaudb.json |
[production] |
| 21:03 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2190', diff saved to https://phabricator.wikimedia.org/P66688 and previous config saved to /var/cache/conftool/dbconfig/20240716-210345-arnaudb.json |
[production] |
| 20:54 |
<urbanecm@deploy1002> |
Finished scap: Backport for [[gerrit:1050083|[July 16th] Enable dark mode for logged out users (tier 1) (T367150)]] (duration: 08m 43s) |
[production] |
| 20:49 |
<urbanecm@deploy1002> |
urbanecm, jdlrobson: Continuing with sync |
[production] |
| 20:48 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2190', diff saved to https://phabricator.wikimedia.org/P66687 and previous config saved to /var/cache/conftool/dbconfig/20240716-204838-arnaudb.json |
[production] |
| 20:48 |
<urbanecm@deploy1002> |
urbanecm, jdlrobson: Backport for [[gerrit:1050083|[July 16th] Enable dark mode for logged out users (tier 1) (T367150)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 20:45 |
<urbanecm@deploy1002> |
Started scap sync-world: Backport for [[gerrit:1050083|[July 16th] Enable dark mode for logged out users (tier 1) (T367150)]] |
[production] |
| 20:39 |
<urbanecm@deploy1002> |
Finished scap: Backport for [[gerrit:1054558|Ensure every test-config has valid defaults]], [[gerrit:1054553|Merge partial config with defaults (T368606)]], [[gerrit:1054554|Merge partial config with defaults (T368606)]] (duration: 09m 55s) |
[production] |
| 20:38 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.reimage for host dbproxy2008.codfw.wmnet with OS bookworm |
[production] |
| 20:34 |
<urbanecm@deploy1002> |
urbanecm, migr: Continuing with sync |
[production] |
| 20:33 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2190 (T367781)', diff saved to https://phabricator.wikimedia.org/P66686 and previous config saved to /var/cache/conftool/dbconfig/20240716-203331-arnaudb.json |
[production] |
| 20:33 |
<urbanecm@deploy1002> |
urbanecm, migr: Backport for [[gerrit:1054558|Ensure every test-config has valid defaults]], [[gerrit:1054553|Merge partial config with defaults (T368606)]], [[gerrit:1054554|Merge partial config with defaults (T368606)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 20:30 |
<pt1979@cumin2002> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host dbproxy2008.codfw.wmnet with OS bookworm |
[production] |
| 20:30 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.reimage for host dbproxy2008.codfw.wmnet with OS bookworm |
[production] |
| 20:29 |
<urbanecm@deploy1002> |
Started scap sync-world: Backport for [[gerrit:1054558|Ensure every test-config has valid defaults]], [[gerrit:1054553|Merge partial config with defaults (T368606)]], [[gerrit:1054554|Merge partial config with defaults (T368606)]] |
[production] |
| 20:27 |
<pt1979@cumin2002> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dbproxy2008.codfw.wmnet with OS bookworm |
[production] |
| 20:14 |
<urbanecm@deploy1002> |
Finished scap: Backport for [[gerrit:1054025|foundationwiki: Restrict `unfuzzy` right to autoconfirmed users (T369979)]] (duration: 09m 31s) |
[production] |
| 20:12 |
<swfrench@cumin2002> |
conftool action : set/pooled=true; selector: dnsdisc=appservers-ro,name=eqiad [reason: Repooling to concentrate clients in eqiad - T367949] |
[production] |
| 20:11 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Depooling db2190 (T367781)', diff saved to https://phabricator.wikimedia.org/P66685 and previous config saved to /var/cache/conftool/dbconfig/20240716-201153-arnaudb.json |
[production] |
| 20:11 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2190.codfw.wmnet with reason: Maintenance |
[production] |
| 20:11 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 4:00:00 on db2190.codfw.wmnet with reason: Maintenance |
[production] |
| 20:11 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2177 (T367781)', diff saved to https://phabricator.wikimedia.org/P66684 and previous config saved to /var/cache/conftool/dbconfig/20240716-201131-arnaudb.json |
[production] |
| 20:09 |
<urbanecm@deploy1002> |
seawolf35gerrit, urbanecm: Continuing with sync |
[production] |
| 20:09 |
<urbanecm@deploy1002> |
seawolf35gerrit, urbanecm: Backport for [[gerrit:1054025|foundationwiki: Restrict `unfuzzy` right to autoconfirmed users (T369979)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 20:05 |
<urbanecm@deploy1002> |
Started scap sync-world: Backport for [[gerrit:1054025|foundationwiki: Restrict `unfuzzy` right to autoconfirmed users (T369979)]] |
[production] |
| 19:56 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P66683 and previous config saved to /var/cache/conftool/dbconfig/20240716-195624-arnaudb.json |
[production] |
| 19:41 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P66682 and previous config saved to /var/cache/conftool/dbconfig/20240716-194117-arnaudb.json |
[production] |
| 19:26 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2177 (T367781)', diff saved to https://phabricator.wikimedia.org/P66681 and previous config saved to /var/cache/conftool/dbconfig/20240716-192610-arnaudb.json |
[production] |
| 19:25 |
<swfrench@cumin2002> |
conftool action : set/pooled=false; selector: dnsdisc=appservers-ro,name=eqiad [reason: Depooling ahead of turndown - T367949] |
[production] |
| 19:24 |
<swfrench-wmf> |
depooling appservers-ro in eqiad, which is not used by remaining analytics workloads - T367949 |
[production] |
| 19:18 |
<cdanis@deploy1002> |
helmfile [eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 19:18 |
<cdanis@deploy1002> |
helmfile [eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 19:17 |
<cdanis@deploy1002> |
helmfile [codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 19:15 |
<cdanis@deploy1002> |
helmfile [codfw] START helmfile.d/admin 'apply'. |
[production] |
| 19:07 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.reimage for host dbproxy2008.codfw.wmnet with OS bookworm |
[production] |
| 19:05 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Depooling db2177 (T367781)', diff saved to https://phabricator.wikimedia.org/P66680 and previous config saved to /var/cache/conftool/dbconfig/20240716-190526-arnaudb.json |
[production] |
| 19:05 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2177.codfw.wmnet with reason: Maintenance |
[production] |
| 19:05 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 4:00:00 on db2177.codfw.wmnet with reason: Maintenance |
[production] |
| 19:05 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2156 (T367781)', diff saved to https://phabricator.wikimedia.org/P66679 and previous config saved to /var/cache/conftool/dbconfig/20240716-190504-arnaudb.json |
[production] |
| 18:56 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db2140 (T367856)', diff saved to https://phabricator.wikimedia.org/P66678 and previous config saved to /var/cache/conftool/dbconfig/20240716-185657-marostegui.json |
[production] |
| 18:56 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2140.codfw.wmnet with reason: Maintenance |
[production] |
| 18:56 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2140.codfw.wmnet with reason: Maintenance |
[production] |
| 18:51 |
<cdanis@deploy1002> |
helmfile [eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 18:50 |
<cdanis@deploy1002> |
helmfile [eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 18:49 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P66677 and previous config saved to /var/cache/conftool/dbconfig/20240716-184956-arnaudb.json |
[production] |
| 18:49 |
<cdanis@deploy1002> |
helmfile [codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 18:49 |
<cdanis@deploy1002> |
helmfile [codfw] START helmfile.d/admin 'apply'. |
[production] |
| 18:45 |
<pt1979@cumin2002> |
END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host dbproxy2007.codfw.wmnet with OS bookworm |
[production] |
| 18:34 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P66675 and previous config saved to /var/cache/conftool/dbconfig/20240716-183449-arnaudb.json |
[production] |