|
2024-05-24
§
|
| 07:40 |
<dcausse@deploy1002> |
Finished deploy [airflow-dags/search@8f0b4a1]: search: fix import_ttl dag (duration: 00m 19s) |
[production] |
| 07:40 |
<dcausse@deploy1002> |
Started deploy [airflow-dags/search@8f0b4a1]: search: fix import_ttl dag |
[production] |
| 07:26 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db1173 (T364299)', diff saved to https://phabricator.wikimedia.org/P63065 and previous config saved to /var/cache/conftool/dbconfig/20240524-072639-marostegui.json |
[production] |
| 07:26 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1173.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:26 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1173.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:26 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1168 (T364299)', diff saved to https://phabricator.wikimedia.org/P63064 and previous config saved to /var/cache/conftool/dbconfig/20240524-072616-marostegui.json |
[production] |
| 07:13 |
<logmsgbot> |
@deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 07:13 |
<logmsgbot> |
@deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 07:11 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P63063 and previous config saved to /var/cache/conftool/dbconfig/20240524-071108-marostegui.json |
[production] |
| 06:56 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P63062 and previous config saved to /var/cache/conftool/dbconfig/20240524-065600-marostegui.json |
[production] |
| 06:40 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1168 (T364299)', diff saved to https://phabricator.wikimedia.org/P63061 and previous config saved to /var/cache/conftool/dbconfig/20240524-064053-marostegui.json |
[production] |
| 06:18 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2122 (re)pooling @ 100%: After reimage', diff saved to https://phabricator.wikimedia.org/P63060 and previous config saved to /var/cache/conftool/dbconfig/20240524-061812-root.json |
[production] |
| 06:03 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2122 (re)pooling @ 75%: After reimage', diff saved to https://phabricator.wikimedia.org/P63059 and previous config saved to /var/cache/conftool/dbconfig/20240524-060305-root.json |
[production] |
| 05:56 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db1168 (T364299)', diff saved to https://phabricator.wikimedia.org/P63058 and previous config saved to /var/cache/conftool/dbconfig/20240524-055616-marostegui.json |
[production] |
| 05:56 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance |
[production] |
| 05:55 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance |
[production] |
| 05:55 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1165 (T364299)', diff saved to https://phabricator.wikimedia.org/P63057 and previous config saved to /var/cache/conftool/dbconfig/20240524-055553-marostegui.json |
[production] |
| 05:48 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2122 (re)pooling @ 50%: After reimage', diff saved to https://phabricator.wikimedia.org/P63056 and previous config saved to /var/cache/conftool/dbconfig/20240524-054759-root.json |
[production] |
| 05:40 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P63055 and previous config saved to /var/cache/conftool/dbconfig/20240524-054045-marostegui.json |
[production] |
| 05:32 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2122 (re)pooling @ 25%: After reimage', diff saved to https://phabricator.wikimedia.org/P63054 and previous config saved to /var/cache/conftool/dbconfig/20240524-053250-root.json |
[production] |
| 05:25 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P63053 and previous config saved to /var/cache/conftool/dbconfig/20240524-052537-marostegui.json |
[production] |
| 05:23 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2122.codfw.wmnet with OS bookworm |
[production] |
| 05:17 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2122 (re)pooling @ 10%: After reimage', diff saved to https://phabricator.wikimedia.org/P63052 and previous config saved to /var/cache/conftool/dbconfig/20240524-051744-root.json |
[production] |
| 05:10 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1165 (T364299)', diff saved to https://phabricator.wikimedia.org/P63051 and previous config saved to /var/cache/conftool/dbconfig/20240524-051028-marostegui.json |
[production] |
| 04:59 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2122.codfw.wmnet with reason: host reimage |
[production] |
| 04:56 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db2122.codfw.wmnet with reason: host reimage |
[production] |
| 04:36 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.reimage for host db2122.codfw.wmnet with OS bookworm |
[production] |
| 04:34 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool db2122', diff saved to https://phabricator.wikimedia.org/P63050 and previous config saved to /var/cache/conftool/dbconfig/20240524-043441-root.json |
[production] |
| 04:23 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db1165 (T364299)', diff saved to https://phabricator.wikimedia.org/P63049 and previous config saved to /var/cache/conftool/dbconfig/20240524-042358-marostegui.json |
[production] |
| 04:23 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance |
[production] |
| 04:23 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance |
[production] |
| 04:23 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance |
[production] |
| 04:23 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance |
[production] |
| 02:58 |
<logmsgbot> |
@deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 02:58 |
<logmsgbot> |
@deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 00:43 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2217 (T364299)', diff saved to https://phabricator.wikimedia.org/P63048 and previous config saved to /var/cache/conftool/dbconfig/20240524-004342-marostegui.json |
[production] |
| 00:28 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2217', diff saved to https://phabricator.wikimedia.org/P63047 and previous config saved to /var/cache/conftool/dbconfig/20240524-002834-marostegui.json |
[production] |
| 00:13 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2217', diff saved to https://phabricator.wikimedia.org/P63046 and previous config saved to /var/cache/conftool/dbconfig/20240524-001326-marostegui.json |
[production] |
|
2024-05-23
§
|
| 23:58 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2217 (T364299)', diff saved to https://phabricator.wikimedia.org/P63045 and previous config saved to /var/cache/conftool/dbconfig/20240523-235817-marostegui.json |
[production] |
| 23:30 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'db1238 (re)pooling @ 100%: Maint over', diff saved to https://phabricator.wikimedia.org/P63044 and previous config saved to /var/cache/conftool/dbconfig/20240523-233017-ladsgroup.json |
[production] |
| 23:24 |
<zabe@deploy1002> |
Finished scap: Backport for [[gerrit:1032586|Deploy configuration for wrapping B type passwords with encrypted Argon2 (T112359)]] (duration: 16m 00s) |
[production] |
| 23:15 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'db1238 (re)pooling @ 75%: Maint over', diff saved to https://phabricator.wikimedia.org/P63043 and previous config saved to /var/cache/conftool/dbconfig/20240523-231511-ladsgroup.json |
[production] |
| 23:13 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db2217 (T364299)', diff saved to https://phabricator.wikimedia.org/P63042 and previous config saved to /var/cache/conftool/dbconfig/20240523-231302-marostegui.json |
[production] |
| 23:12 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2217.codfw.wmnet with reason: Maintenance |
[production] |
| 23:12 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db2217.codfw.wmnet with reason: Maintenance |
[production] |
| 23:12 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2214 (T364299)', diff saved to https://phabricator.wikimedia.org/P63041 and previous config saved to /var/cache/conftool/dbconfig/20240523-231238-marostegui.json |
[production] |
| 23:11 |
<zabe@deploy1002> |
zabe: Continuing with sync |
[production] |
| 23:10 |
<zabe@deploy1002> |
zabe: Backport for [[gerrit:1032586|Deploy configuration for wrapping B type passwords with encrypted Argon2 (T112359)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 23:08 |
<zabe@deploy1002> |
Started scap: Backport for [[gerrit:1032586|Deploy configuration for wrapping B type passwords with encrypted Argon2 (T112359)]] |
[production] |
| 23:00 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'db1238 (re)pooling @ 25%: Maint over', diff saved to https://phabricator.wikimedia.org/P63040 and previous config saved to /var/cache/conftool/dbconfig/20240523-230005-ladsgroup.json |
[production] |