|
2022-08-25
ยง
|
| 18:34 |
<otto@deploy1002> |
helmfile [codfw] START helmfile.d/services/eventgate-analytics-external: sync |
[production] |
| 18:34 |
<otto@deploy1002> |
helmfile [staging] DONE helmfile.d/services/eventgate-analytics-external: sync |
[production] |
| 18:33 |
<otto@deploy1002> |
helmfile [staging] START helmfile.d/services/eventgate-analytics-external: sync |
[production] |
| 18:33 |
<ottomata> |
rolling restart of eventgate-analytics-external to pick up retroactive schema change for android schemas in T316047 |
[production] |
| 18:27 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P33141 and previous config saved to /var/cache/conftool/dbconfig/20220825-182727-ladsgroup.json |
[production] |
| 18:19 |
<dancy@deploy1002> |
rebuilt and synchronized wikiversions files: (no justification provided) |
[production] |
| 18:18 |
<bmansurov@deploy1002> |
Finished deploy [airflow-dags/research@5712187]: (no justification provided) (duration: 00m 09s) |
[production] |
| 18:18 |
<bmansurov@deploy1002> |
Started deploy [airflow-dags/research@5712187]: (no justification provided) |
[production] |
| 18:13 |
<dancy@deploy1002> |
Installation of scap version "4.15.0" completed for 557 hosts |
[production] |
| 18:12 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P33140 and previous config saved to /var/cache/conftool/dbconfig/20220825-181221-ladsgroup.json |
[production] |
| 18:11 |
<dancy@deploy1002> |
Installing scap version "4.15.0" for 557 hosts |
[production] |
| 18:11 |
<dancy@deploy1002> |
install-world aborted: (duration: 00m 02s) |
[production] |
| 17:57 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1131 (T316186)', diff saved to https://phabricator.wikimedia.org/P33139 and previous config saved to /var/cache/conftool/dbconfig/20220825-175715-ladsgroup.json |
[production] |
| 17:49 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1131 (T316186)', diff saved to https://phabricator.wikimedia.org/P33138 and previous config saved to /var/cache/conftool/dbconfig/20220825-174946-ladsgroup.json |
[production] |
| 17:49 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1131.eqiad.wmnet with reason: Maintenance |
[production] |
| 17:49 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1131.eqiad.wmnet with reason: Maintenance |
[production] |
| 17:48 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2115 (T312160)', diff saved to https://phabricator.wikimedia.org/P33137 and previous config saved to /var/cache/conftool/dbconfig/20220825-174826-ladsgroup.json |
[production] |
| 17:48 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2115.codfw.wmnet with reason: Maintenance |
[production] |
| 17:47 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2115.codfw.wmnet with reason: Maintenance |
[production] |
| 17:39 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1140.eqiad.wmnet with reason: Maintenance |
[production] |
| 17:38 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1140.eqiad.wmnet with reason: Maintenance |
[production] |
| 17:37 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2169:3317 (T316186)', diff saved to https://phabricator.wikimedia.org/P33136 and previous config saved to /var/cache/conftool/dbconfig/20220825-173731-ladsgroup.json |
[production] |
| 17:22 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2169:3317', diff saved to https://phabricator.wikimedia.org/P33135 and previous config saved to /var/cache/conftool/dbconfig/20220825-172225-ladsgroup.json |
[production] |
| 17:10 |
<bd808@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply |
[production] |
| 17:10 |
<bd808@deploy1002> |
helmfile [eqiad] START helmfile.d/services/developer-portal: apply |
[production] |
| 17:10 |
<bd808@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/developer-portal: apply |
[production] |
| 17:09 |
<bd808@deploy1002> |
helmfile [codfw] START helmfile.d/services/developer-portal: apply |
[production] |
| 17:09 |
<bd808@deploy1002> |
helmfile [staging] DONE helmfile.d/services/developer-portal: apply |
[production] |
| 17:08 |
<bd808@deploy1002> |
helmfile [staging] START helmfile.d/services/developer-portal: apply |
[production] |
| 17:07 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2169:3317', diff saved to https://phabricator.wikimedia.org/P33133 and previous config saved to /var/cache/conftool/dbconfig/20220825-170719-ladsgroup.json |
[production] |
| 16:52 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2169:3317 (T316186)', diff saved to https://phabricator.wikimedia.org/P33132 and previous config saved to /var/cache/conftool/dbconfig/20220825-165213-ladsgroup.json |
[production] |
| 16:45 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2169:3316 (T316186)', diff saved to https://phabricator.wikimedia.org/P33131 and previous config saved to /var/cache/conftool/dbconfig/20220825-164556-ladsgroup.json |
[production] |
| 16:40 |
<urandom> |
shutting down ms-be2067.codfw.wmnet for backplane replacement -- T314049 |
[production] |
| 16:37 |
<eevans@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ms-be2067.codfw.wmnet with reason: backplane replacement |
[production] |
| 16:37 |
<eevans@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on ms-be2067.codfw.wmnet with reason: backplane replacement |
[production] |
| 16:30 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2169:3316', diff saved to https://phabricator.wikimedia.org/P33130 and previous config saved to /var/cache/conftool/dbconfig/20220825-163050-ladsgroup.json |
[production] |
| 16:15 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2169:3316', diff saved to https://phabricator.wikimedia.org/P33129 and previous config saved to /var/cache/conftool/dbconfig/20220825-161544-ladsgroup.json |
[production] |
| 16:07 |
<bmansurov@deploy1002> |
Finished deploy [airflow-dags/research@5712187]: (no justification provided) (duration: 00m 09s) |
[production] |
| 16:07 |
<bmansurov@deploy1002> |
Started deploy [airflow-dags/research@5712187]: (no justification provided) |
[production] |
| 16:02 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:02 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:02 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1120 (T312160)', diff saved to https://phabricator.wikimedia.org/P33128 and previous config saved to /var/cache/conftool/dbconfig/20220825-160250-ladsgroup.json |
[production] |
| 16:00 |
<bmansurov@deploy1002> |
Finished deploy [airflow-dags/research@5712187]: (no justification provided) (duration: 00m 09s) |
[production] |
| 16:00 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2169:3316 (T316186)', diff saved to https://phabricator.wikimedia.org/P33127 and previous config saved to /var/cache/conftool/dbconfig/20220825-160036-ladsgroup.json |
[production] |
| 16:00 |
<bmansurov@deploy1002> |
Started deploy [airflow-dags/research@5712187]: (no justification provided) |
[production] |
| 15:55 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2169:3317 (T316186)', diff saved to https://phabricator.wikimedia.org/P33126 and previous config saved to /var/cache/conftool/dbconfig/20220825-155529-ladsgroup.json |
[production] |
| 15:55 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2169:3316 (T316186)', diff saved to https://phabricator.wikimedia.org/P33125 and previous config saved to /var/cache/conftool/dbconfig/20220825-155506-ladsgroup.json |
[production] |
| 15:55 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2169.codfw.wmnet with reason: Maintenance |
[production] |
| 15:54 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2169.codfw.wmnet with reason: Maintenance |
[production] |
| 15:54 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P33124 and previous config saved to /var/cache/conftool/dbconfig/20220825-155401-ladsgroup.json |
[production] |