|
2024-05-30
ยง
|
| 13:27 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db1173.eqiad.wmnet with reason: upgrade |
[production] |
| 13:27 |
<logmsgbot> |
@deploy1002 helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 13:27 |
<logmsgbot> |
@deploy1002 helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 13:25 |
<logmsgbot> |
@deploy1002 helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 13:25 |
<logmsgbot> |
@deploy1002 helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 13:25 |
<jiji@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc2044.codfw.wmnet with reason: host reimage |
[production] |
| 13:23 |
<logmsgbot> |
@deploy1002 helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 13:23 |
<logmsgbot> |
@deploy1002 helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 13:22 |
<jiji@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mc2044.codfw.wmnet with reason: host reimage |
[production] |
| 13:20 |
<jiji@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1044.eqiad.wmnet with reason: host reimage |
[production] |
| 13:20 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1173.eqiad.wmnet |
[production] |
| 13:19 |
<dcausse@deploy1002> |
Finished scap: Backport for [[gerrit:1037153|cirrus: Send weighted tags to known clusters]] (duration: 12m 43s) |
[production] |
| 13:17 |
<logmsgbot> |
@deploy1002 helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 13:17 |
<logmsgbot> |
@deploy1002 helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 13:17 |
<jiji@cumin1002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mc1044.eqiad.wmnet with reason: host reimage |
[production] |
| 13:15 |
<logmsgbot> |
@deploy1002 helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 13:15 |
<logmsgbot> |
@deploy1002 helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 13:15 |
<arnaudb@cumin1002> |
START - Cookbook sre.mysql.upgrade for db1173.eqiad.wmnet |
[production] |
| 13:14 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1173.eqiad.wmnet with reason: upgrade |
[production] |
| 13:14 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db1173.eqiad.wmnet with reason: upgrade |
[production] |
| 13:13 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Depool db1173 T356240', diff saved to https://phabricator.wikimedia.org/P63695 and previous config saved to /var/cache/conftool/dbconfig/20240530-131349-arnaudb.json |
[production] |
| 13:12 |
<logmsgbot> |
@deploy1002 helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 13:12 |
<logmsgbot> |
@deploy1002 helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 13:10 |
<dcausse@deploy1002> |
dcausse and ebernhardson: Continuing with sync |
[production] |
| 13:10 |
<logmsgbot> |
@deploy1002 helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 13:10 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db2220 (T366123)', diff saved to https://phabricator.wikimedia.org/P63694 and previous config saved to /var/cache/conftool/dbconfig/20240530-131012-marostegui.json |
[production] |
| 13:10 |
<logmsgbot> |
@deploy1002 helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 13:10 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2220.codfw.wmnet with reason: Maintenance |
[production] |
| 13:09 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db2220.codfw.wmnet with reason: Maintenance |
[production] |
| 13:09 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2208 (T366123)', diff saved to https://phabricator.wikimedia.org/P63693 and previous config saved to /var/cache/conftool/dbconfig/20240530-130946-marostegui.json |
[production] |
| 13:09 |
<dcausse@deploy1002> |
dcausse and ebernhardson: Backport for [[gerrit:1037153|cirrus: Send weighted tags to known clusters]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 13:08 |
<logmsgbot> |
@deploy1002 helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 13:08 |
<logmsgbot> |
@deploy1002 helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 13:06 |
<dcausse@deploy1002> |
Started scap: Backport for [[gerrit:1037153|cirrus: Send weighted tags to known clusters]] |
[production] |
| 13:06 |
<logmsgbot> |
@deploy1002 helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 13:06 |
<logmsgbot> |
@deploy1002 helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 13:04 |
<jiji@cumin1002> |
START - Cookbook sre.hosts.reimage for host mc1044.eqiad.wmnet with OS bookworm |
[production] |
| 13:04 |
<jiji@cumin2002> |
START - Cookbook sre.hosts.reimage for host mc2044.codfw.wmnet with OS bookworm |
[production] |
| 13:01 |
<logmsgbot> |
@deploy1002 helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 13:01 |
<joal@deploy1002> |
Finished deploy [analytics/refinery@ac0b789] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@ac0b789b] (duration: 02m 54s) |
[production] |
| 13:01 |
<logmsgbot> |
@deploy1002 helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 12:58 |
<joal@deploy1002> |
Started deploy [analytics/refinery@ac0b789] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@ac0b789b] |
[production] |
| 12:58 |
<logmsgbot> |
@deploy1002 helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 12:58 |
<logmsgbot> |
@deploy1002 helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 12:55 |
<joal@deploy1002> |
Finished deploy [analytics/refinery@ac0b789] (thin): Regular analytics weekly train THIN [analytics/refinery@ac0b789b] (duration: 04m 27s) |
[production] |
| 12:54 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P63692 and previous config saved to /var/cache/conftool/dbconfig/20240530-125438-marostegui.json |
[production] |
| 12:53 |
<cdanis@deploy1002> |
helmfile [eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 12:53 |
<cdanis@deploy1002> |
helmfile [eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 12:52 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db2155 (T364069)', diff saved to https://phabricator.wikimedia.org/P63691 and previous config saved to /var/cache/conftool/dbconfig/20240530-125232-marostegui.json |
[production] |
| 12:52 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance |
[production] |