|
2022-09-16
§
|
| 08:21 |
<elukey@deploy1002> |
helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. |
[production] |
| 08:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1189 (re)pooling @ 5%: After being recloned', diff saved to https://phabricator.wikimedia.org/P34814 and previous config saved to /var/cache/conftool/dbconfig/20220916-082110-root.json |
[production] |
| 08:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1198 (re)pooling @ 5%: Repooling after cloning db1189', diff saved to https://phabricator.wikimedia.org/P34813 and previous config saved to /var/cache/conftool/dbconfig/20220916-081557-root.json |
[production] |
| 08:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1189 (re)pooling @ 3%: After being recloned', diff saved to https://phabricator.wikimedia.org/P34812 and previous config saved to /var/cache/conftool/dbconfig/20220916-080605-root.json |
[production] |
| 08:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1198 (re)pooling @ 3%: Repooling after cloning db1189', diff saved to https://phabricator.wikimedia.org/P34811 and previous config saved to /var/cache/conftool/dbconfig/20220916-080052-root.json |
[production] |
| 07:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1189 (re)pooling @ 1%: After being recloned', diff saved to https://phabricator.wikimedia.org/P34810 and previous config saved to /var/cache/conftool/dbconfig/20220916-075100-root.json |
[production] |
| 07:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1198 (re)pooling @ 1%: Repooling after cloning db1189', diff saved to https://phabricator.wikimedia.org/P34809 and previous config saved to /var/cache/conftool/dbconfig/20220916-074548-root.json |
[production] |
| 07:42 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1168 (re)pooling @ 100%: After upgrade', diff saved to https://phabricator.wikimedia.org/P34808 and previous config saved to /var/cache/conftool/dbconfig/20220916-074251-root.json |
[production] |
| 07:29 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2180', diff saved to https://phabricator.wikimedia.org/P34807 and previous config saved to /var/cache/conftool/dbconfig/20220916-072958-root.json |
[production] |
| 07:27 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1168 (re)pooling @ 75%: After upgrade', diff saved to https://phabricator.wikimedia.org/P34806 and previous config saved to /var/cache/conftool/dbconfig/20220916-072746-root.json |
[production] |
| 07:26 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 07:25 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 07:25 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 07:21 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 07:12 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1168 (re)pooling @ 50%: After upgrade', diff saved to https://phabricator.wikimedia.org/P34805 and previous config saved to /var/cache/conftool/dbconfig/20220916-071241-root.json |
[production] |
| 06:57 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1168 (re)pooling @ 25%: After upgrade', diff saved to https://phabricator.wikimedia.org/P34804 and previous config saved to /var/cache/conftool/dbconfig/20220916-065737-root.json |
[production] |
| 06:42 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1168 (re)pooling @ 10%: After upgrade', diff saved to https://phabricator.wikimedia.org/P34803 and previous config saved to /var/cache/conftool/dbconfig/20220916-064232-root.json |
[production] |
| 06:27 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1168 (re)pooling @ 5%: After upgrade', diff saved to https://phabricator.wikimedia.org/P34802 and previous config saved to /var/cache/conftool/dbconfig/20220916-062727-root.json |
[production] |
| 06:12 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1168 (re)pooling @ 3%: After upgrade', diff saved to https://phabricator.wikimedia.org/P34801 and previous config saved to /var/cache/conftool/dbconfig/20220916-061222-root.json |
[production] |
| 05:57 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1168 (re)pooling @ 1%: After upgrade', diff saved to https://phabricator.wikimedia.org/P34800 and previous config saved to /var/cache/conftool/dbconfig/20220916-055717-root.json |
[production] |
| 05:55 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1168', diff saved to https://phabricator.wikimedia.org/P34799 and previous config saved to /var/cache/conftool/dbconfig/20220916-055542-root.json |
[production] |
| 05:54 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1168 (re)pooling @ 1%: After upgrade', diff saved to https://phabricator.wikimedia.org/P34798 and previous config saved to /var/cache/conftool/dbconfig/20220916-055424-root.json |
[production] |
| 05:51 |
<marostegui> |
Install 10.6 on db1168 T301879 |
[production] |
| 05:50 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1168', diff saved to https://phabricator.wikimedia.org/P34797 and previous config saved to /var/cache/conftool/dbconfig/20220916-055031-root.json |
[production] |
| 05:44 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1198', diff saved to https://phabricator.wikimedia.org/P34795 and previous config saved to /var/cache/conftool/dbconfig/20220916-054438-root.json |
[production] |
| 01:57 |
<bmansurov@deploy1002> |
Finished deploy [airflow-dags/research@b9be20d]: (no justification provided) (duration: 00m 09s) |
[production] |
| 01:57 |
<bmansurov@deploy1002> |
Started deploy [airflow-dags/research@b9be20d]: (no justification provided) |
[production] |
| 01:54 |
<bmansurov@deploy1002> |
Finished deploy [airflow-dags/research@b9be20d]: (no justification provided) (duration: 00m 10s) |
[production] |
| 01:54 |
<bmansurov@deploy1002> |
Started deploy [airflow-dags/research@b9be20d]: (no justification provided) |
[production] |
| 00:14 |
<bmansurov@deploy1002> |
Finished deploy [airflow-dags/research@b9be20d]: (no justification provided) (duration: 00m 17s) |
[production] |
| 00:14 |
<bmansurov@deploy1002> |
Started deploy [airflow-dags/research@b9be20d]: (no justification provided) |
[production] |
|
2022-09-15
§
|
| 23:51 |
<mutante> |
gerrit1001 - disabled puppet - gerrit:832411 |
[production] |
| 22:01 |
<bking@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on wcqs2001.codfw.wmnet with reason: T316236 |
[production] |
| 22:01 |
<bking@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on wcqs2001.codfw.wmnet with reason: T316236 |
[production] |
| 21:30 |
<ebernhardson> |
depool wcqs2001 for T316236 |
[production] |
| 20:25 |
<thcipriani@deploy1002> |
Finished scap: Backport for [[gerrit:832526|Increase coverage of Research Incentive Survey on idwiki (T316466)]] (duration: 07m 06s) |
[production] |
| 20:18 |
<thcipriani@deploy1002> |
thcipriani and dani: Backport for [[gerrit:832526|Increase coverage of Research Incentive Survey on idwiki (T316466)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet |
[production] |
| 20:18 |
<thcipriani@deploy1002> |
Started scap: Backport for [[gerrit:832526|Increase coverage of Research Incentive Survey on idwiki (T316466)]] |
[production] |
| 20:15 |
<thcipriani@deploy1002> |
Finished scap: Backport for [[gerrit:832323|Revert "cirrus: Handle transition to elasticsearch 7.10" (T308676)]] (duration: 07m 39s) |
[production] |
| 20:08 |
<thcipriani@deploy1002> |
thcipriani and dcausse: Backport for [[gerrit:832323|Revert "cirrus: Handle transition to elasticsearch 7.10" (T308676)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet |
[production] |
| 20:07 |
<thcipriani@deploy1002> |
Started scap: Backport for [[gerrit:832323|Revert "cirrus: Handle transition to elasticsearch 7.10" (T308676)]] |
[production] |
| 19:26 |
<ebernhardson> |
pool'd wdqs2001, some blockers before reload can start T316236 |
[production] |
| 18:45 |
<dancy@deploy1002> |
rebuilt and synchronized wikiversions files: group2 wikis to 1.40.0-wmf.1 refs T314190 |
[production] |
| 18:39 |
<dancy@deploy1002> |
Finished scap: Backport for [[gerrit:832547|Use more permissive match for TOC_PLACEHOLDER in parser output (T317857)]] (duration: 09m 53s) |
[production] |
| 18:38 |
<cwhite> |
restart thanos-compact (thanos-fe2001) and swift_ring_manager (thanos-fe1001) |
[production] |
| 18:29 |
<dancy@deploy1002> |
dancy and cscott: Backport for [[gerrit:832547|Use more permissive match for TOC_PLACEHOLDER in parser output (T317857)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet |
[production] |
| 18:29 |
<dancy@deploy1002> |
Started scap: Backport for [[gerrit:832547|Use more permissive match for TOC_PLACEHOLDER in parser output (T317857)]] |
[production] |
| 18:17 |
<cwhite@cumin2002> |
END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) thanos-fe2003.codfw.wmnet on all recursors |
[production] |
| 18:17 |
<cwhite@cumin2002> |
START - Cookbook sre.dns.wipe-cache thanos-fe2003.codfw.wmnet on all recursors |
[production] |
| 18:17 |
<cwhite@cumin2002> |
END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) thanos-fe2002.codfw.wmnet on all recursors |
[production] |