|
2024-04-15
ยง
|
| 15:55 |
<brett@cumin2002> |
START - Cookbook sre.hosts.reimage for host ncredir1001.eqiad.wmnet with OS bullseye |
[production] |
| 15:53 |
<logmsgbot> |
@deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 15:53 |
<logmsgbot> |
@deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 15:51 |
<logmsgbot> |
@deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 15:51 |
<logmsgbot> |
@deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 15:50 |
<eevans@deploy1002> |
helmfile [staging] DONE helmfile.d/services/sessionstore: apply |
[production] |
| 15:48 |
<logmsgbot> |
@deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 15:48 |
<logmsgbot> |
@deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 15:45 |
<logmsgbot> |
@deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 15:45 |
<logmsgbot> |
@deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 15:44 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1202 (T356166)', diff saved to https://phabricator.wikimedia.org/P60520 and previous config saved to /var/cache/conftool/dbconfig/20240415-154422-marostegui.json |
[production] |
| 15:40 |
<eevans@deploy1002> |
helmfile [staging] START helmfile.d/services/sessionstore: apply |
[production] |
| 15:21 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2111 (re)pooling @ 100%: post reimage repool', diff saved to https://phabricator.wikimedia.org/P60519 and previous config saved to /var/cache/conftool/dbconfig/20240415-152132-arnaudb.json |
[production] |
| 15:19 |
<pfischer@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 15:18 |
<pfischer@deploy1002> |
helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 15:15 |
<jclark@cumin1002> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dbprov1006.eqiad.wmnet with OS bullseye |
[production] |
| 15:11 |
<bking@cumin2002> |
END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: T361647 - bking@cumin2002 |
[production] |
| 15:06 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2111 (re)pooling @ 75%: post reimage repool', diff saved to https://phabricator.wikimedia.org/P60518 and previous config saved to /var/cache/conftool/dbconfig/20240415-150626-arnaudb.json |
[production] |
| 15:04 |
<dancy@deploy1002> |
Installation of scap version "4.76.0" completed for 340 hosts |
[production] |
| 15:04 |
<Daimona> |
Running query for T362365#9710047 |
[production] |
| 15:03 |
<dancy@deploy1002> |
Installing scap version "4.76.0" for 340 hosts |
[production] |
| 15:03 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db1202 (T356166)', diff saved to https://phabricator.wikimedia.org/P60517 and previous config saved to /var/cache/conftool/dbconfig/20240415-150257-marostegui.json |
[production] |
| 15:03 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1202.eqiad.wmnet with reason: Maintenance |
[production] |
| 15:02 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1202.eqiad.wmnet with reason: Maintenance |
[production] |
| 15:02 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1194 (T356166)', diff saved to https://phabricator.wikimedia.org/P60516 and previous config saved to /var/cache/conftool/dbconfig/20240415-150235-marostegui.json |
[production] |
| 14:52 |
<Dreamy_Jazz> |
Afternoon backport window done |
[production] |
| 14:51 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2111 (re)pooling @ 50%: post reimage repool', diff saved to https://phabricator.wikimedia.org/P60515 and previous config saved to /var/cache/conftool/dbconfig/20240415-145120-arnaudb.json |
[production] |
| 14:50 |
<dreamyjazz@deploy1002> |
Finished scap: Backport for [[gerrit:1019821|Define 'useYear' as true for temp user serial mapping config (T349506)]] (duration: 16m 16s) |
[production] |
| 14:48 |
<bking@cumin2002> |
START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: T361647 - bking@cumin2002 |
[production] |
| 14:47 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P60514 and previous config saved to /var/cache/conftool/dbconfig/20240415-144725-marostegui.json |
[production] |
| 14:41 |
<jynus> |
fixed grants for db2098 |
[production] |
| 14:37 |
<dreamyjazz@deploy1002> |
dreamyjazz: Continuing with sync |
[production] |
| 14:36 |
<dreamyjazz@deploy1002> |
dreamyjazz: Backport for [[gerrit:1019821|Define 'useYear' as true for temp user serial mapping config (T349506)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 14:36 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2111 (re)pooling @ 25%: post reimage repool', diff saved to https://phabricator.wikimedia.org/P60513 and previous config saved to /var/cache/conftool/dbconfig/20240415-143614-arnaudb.json |
[production] |
| 14:34 |
<dreamyjazz@deploy1002> |
Started scap: Backport for [[gerrit:1019821|Define 'useYear' as true for temp user serial mapping config (T349506)]] |
[production] |
| 14:32 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P60512 and previous config saved to /var/cache/conftool/dbconfig/20240415-143217-marostegui.json |
[production] |
| 14:31 |
<urbanecm@deploy1002> |
Finished scap: Backport for [[gerrit:1014526|Add wgAutoCreateTempUser configuration for production (T349506 T337090)]], [[gerrit:1019694|Change mul deployment on beta to limited version (T356169)]] (duration: 30m 12s) |
[production] |
| 14:31 |
<stevemunene@deploy1002> |
helmfile [staging] DONE helmfile.d/services/datahub: sync on main |
[production] |
| 14:30 |
<vgutierrez@cumin1002> |
END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P{cp4051.ulsfo.wmnet,cp5030.eqsin.wmnet,cp5032.eqsin.wmnet} and A:cp |
[production] |
| 14:27 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2111.codfw.wmnet with OS bookworm |
[production] |
| 14:23 |
<stevemunene@deploy1002> |
helmfile [staging] START helmfile.d/services/datahub: apply on main |
[production] |
| 14:21 |
<vgutierrez@cumin1002> |
START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P{cp4051.ulsfo.wmnet,cp5030.eqsin.wmnet,cp5032.eqsin.wmnet} and A:cp |
[production] |
| 14:18 |
<urbanecm@deploy1002> |
urbanecm and dreamyjazz and arthurtaylor: Continuing with sync |
[production] |
| 14:17 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1194 (T356166)', diff saved to https://phabricator.wikimedia.org/P60511 and previous config saved to /var/cache/conftool/dbconfig/20240415-141710-marostegui.json |
[production] |
| 14:16 |
<elukey> |
move cassandra instances on cassandra-dev to pki - T352647 |
[production] |
| 14:14 |
<urbanecm@deploy1002> |
urbanecm and dreamyjazz and arthurtaylor: Backport for [[gerrit:1014526|Add wgAutoCreateTempUser configuration for production (T349506 T337090)]], [[gerrit:1019694|Change mul deployment on beta to limited version (T356169)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 14:13 |
<vgutierrez> |
uploaded tcp-mss-clamper 0.4+deb11u2 to bullseye-wikimedia (apt.wm.o) |
[production] |
| 14:09 |
<bking@cumin2002> |
END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: T361647 - bking@cumin2002 |
[production] |
| 14:06 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2111.codfw.wmnet with reason: host reimage |
[production] |
| 14:04 |
<bking@cumin2002> |
START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: T361647 - bking@cumin2002 |
[production] |