|
2024-07-16
ยง
|
| 20:45 |
<urbanecm@deploy1002> |
Started scap sync-world: Backport for [[gerrit:1050083|[July 16th] Enable dark mode for logged out users (tier 1) (T367150)]] |
[production] |
| 20:39 |
<urbanecm@deploy1002> |
Finished scap: Backport for [[gerrit:1054558|Ensure every test-config has valid defaults]], [[gerrit:1054553|Merge partial config with defaults (T368606)]], [[gerrit:1054554|Merge partial config with defaults (T368606)]] (duration: 09m 55s) |
[production] |
| 20:38 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.reimage for host dbproxy2008.codfw.wmnet with OS bookworm |
[production] |
| 20:34 |
<urbanecm@deploy1002> |
urbanecm, migr: Continuing with sync |
[production] |
| 20:34 |
<ottomata> |
disabled produce_canary_events systemd timer to unblock mw on k8s. airflow should suffice now. T370186 |
[analytics] |
| 20:33 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2190 (T367781)', diff saved to https://phabricator.wikimedia.org/P66686 and previous config saved to /var/cache/conftool/dbconfig/20240716-203331-arnaudb.json |
[production] |
| 20:33 |
<urbanecm@deploy1002> |
urbanecm, migr: Backport for [[gerrit:1054558|Ensure every test-config has valid defaults]], [[gerrit:1054553|Merge partial config with defaults (T368606)]], [[gerrit:1054554|Merge partial config with defaults (T368606)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 20:30 |
<pt1979@cumin2002> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host dbproxy2008.codfw.wmnet with OS bookworm |
[production] |
| 20:30 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.reimage for host dbproxy2008.codfw.wmnet with OS bookworm |
[production] |
| 20:29 |
<urbanecm@deploy1002> |
Started scap sync-world: Backport for [[gerrit:1054558|Ensure every test-config has valid defaults]], [[gerrit:1054553|Merge partial config with defaults (T368606)]], [[gerrit:1054554|Merge partial config with defaults (T368606)]] |
[production] |
| 20:27 |
<pt1979@cumin2002> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dbproxy2008.codfw.wmnet with OS bookworm |
[production] |
| 20:23 |
<wmbot~dcaro@urcuchillay> |
START - Cookbook wmcs.ceph.osd.bootstrap_and_add |
[admin] |
| 20:23 |
<wmbot~dcaro@urcuchillay> |
END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) |
[admin] |
| 20:14 |
<urbanecm@deploy1002> |
Finished scap: Backport for [[gerrit:1054025|foundationwiki: Restrict `unfuzzy` right to autoconfirmed users (T369979)]] (duration: 09m 31s) |
[production] |
| 20:12 |
<swfrench@cumin2002> |
conftool action : set/pooled=true; selector: dnsdisc=appservers-ro,name=eqiad [reason: Repooling to concentrate clients in eqiad - T367949] |
[production] |
| 20:11 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Depooling db2190 (T367781)', diff saved to https://phabricator.wikimedia.org/P66685 and previous config saved to /var/cache/conftool/dbconfig/20240716-201153-arnaudb.json |
[production] |
| 20:11 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2190.codfw.wmnet with reason: Maintenance |
[production] |
| 20:11 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 4:00:00 on db2190.codfw.wmnet with reason: Maintenance |
[production] |
| 20:11 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2177 (T367781)', diff saved to https://phabricator.wikimedia.org/P66684 and previous config saved to /var/cache/conftool/dbconfig/20240716-201131-arnaudb.json |
[production] |
| 20:09 |
<urbanecm@deploy1002> |
seawolf35gerrit, urbanecm: Continuing with sync |
[production] |
| 20:09 |
<urbanecm@deploy1002> |
seawolf35gerrit, urbanecm: Backport for [[gerrit:1054025|foundationwiki: Restrict `unfuzzy` right to autoconfirmed users (T369979)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 20:05 |
<urbanecm@deploy1002> |
Started scap sync-world: Backport for [[gerrit:1054025|foundationwiki: Restrict `unfuzzy` right to autoconfirmed users (T369979)]] |
[production] |
| 19:56 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P66683 and previous config saved to /var/cache/conftool/dbconfig/20240716-195624-arnaudb.json |
[production] |
| 19:53 |
<wmbot~dcaro@urcuchillay> |
START - Cookbook wmcs.ceph.osd.depool_and_destroy |
[admin] |
| 19:52 |
<wmbot~dcaro@urcuchillay> |
END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) |
[admin] |
| 19:41 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P66682 and previous config saved to /var/cache/conftool/dbconfig/20240716-194117-arnaudb.json |
[production] |
| 19:26 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2177 (T367781)', diff saved to https://phabricator.wikimedia.org/P66681 and previous config saved to /var/cache/conftool/dbconfig/20240716-192610-arnaudb.json |
[production] |
| 19:25 |
<swfrench@cumin2002> |
conftool action : set/pooled=false; selector: dnsdisc=appservers-ro,name=eqiad [reason: Depooling ahead of turndown - T367949] |
[production] |
| 19:24 |
<swfrench-wmf> |
depooling appservers-ro in eqiad, which is not used by remaining analytics workloads - T367949 |
[production] |
| 19:18 |
<cdanis@deploy1002> |
helmfile [eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 19:18 |
<cdanis@deploy1002> |
helmfile [eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 19:17 |
<cdanis@deploy1002> |
helmfile [codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 19:15 |
<cdanis@deploy1002> |
helmfile [codfw] START helmfile.d/admin 'apply'. |
[production] |
| 19:07 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.reimage for host dbproxy2008.codfw.wmnet with OS bookworm |
[production] |
| 19:05 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Depooling db2177 (T367781)', diff saved to https://phabricator.wikimedia.org/P66680 and previous config saved to /var/cache/conftool/dbconfig/20240716-190526-arnaudb.json |
[production] |
| 19:05 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2177.codfw.wmnet with reason: Maintenance |
[production] |
| 19:05 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 4:00:00 on db2177.codfw.wmnet with reason: Maintenance |
[production] |
| 19:05 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2156 (T367781)', diff saved to https://phabricator.wikimedia.org/P66679 and previous config saved to /var/cache/conftool/dbconfig/20240716-190504-arnaudb.json |
[production] |
| 18:56 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db2140 (T367856)', diff saved to https://phabricator.wikimedia.org/P66678 and previous config saved to /var/cache/conftool/dbconfig/20240716-185657-marostegui.json |
[production] |
| 18:56 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2140.codfw.wmnet with reason: Maintenance |
[production] |
| 18:56 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2140.codfw.wmnet with reason: Maintenance |
[production] |
| 18:51 |
<cdanis@deploy1002> |
helmfile [eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 18:50 |
<cdanis@deploy1002> |
helmfile [eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 18:49 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P66677 and previous config saved to /var/cache/conftool/dbconfig/20240716-184956-arnaudb.json |
[production] |
| 18:49 |
<cdanis@deploy1002> |
helmfile [codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 18:49 |
<cdanis@deploy1002> |
helmfile [codfw] START helmfile.d/admin 'apply'. |
[production] |
| 18:45 |
<pt1979@cumin2002> |
END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host dbproxy2007.codfw.wmnet with OS bookworm |
[production] |
| 18:34 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P66675 and previous config saved to /var/cache/conftool/dbconfig/20240716-183449-arnaudb.json |
[production] |
| 18:27 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.reimage for host dbproxy2007.codfw.wmnet with OS bookworm |
[production] |
| 18:19 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2156 (T367781)', diff saved to https://phabricator.wikimedia.org/P66674 and previous config saved to /var/cache/conftool/dbconfig/20240716-181942-arnaudb.json |
[production] |