|
2022-06-06
§
|
| 08:58 |
<mvernon@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1049.eqiad.wmnet with reason: host reimage |
[production] |
| 08:46 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P29433 and previous config saved to /var/cache/conftool/dbconfig/20220606-084658-ladsgroup.json |
[production] |
| 08:41 |
<mvernon@cumin2002> |
START - Cookbook sre.hosts.reimage for host ms-be1049.eqiad.wmnet with OS bullseye |
[production] |
| 08:39 |
<mbsantos> |
maintenance: trigger full planet re-import for maps codfw |
[production] |
| 08:31 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T298560)', diff saved to https://phabricator.wikimedia.org/P29432 and previous config saved to /var/cache/conftool/dbconfig/20220606-083153-ladsgroup.json |
[production] |
| 08:25 |
<mvernon@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1048.eqiad.wmnet with OS bullseye |
[production] |
| 08:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1128 (re)pooling @ 100%: After onsite maintenance', diff saved to https://phabricator.wikimedia.org/P29431 and previous config saved to /var/cache/conftool/dbconfig/20220606-081647-root.json |
[production] |
| 08:11 |
<mvernon@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1048.eqiad.wmnet with reason: host reimage |
[production] |
| 08:08 |
<mvernon@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1048.eqiad.wmnet with reason: host reimage |
[production] |
| 08:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1128 (re)pooling @ 75%: After onsite maintenance', diff saved to https://phabricator.wikimedia.org/P29430 and previous config saved to /var/cache/conftool/dbconfig/20220606-080142-root.json |
[production] |
| 07:51 |
<mvernon@cumin2002> |
START - Cookbook sre.hosts.reimage for host ms-be1048.eqiad.wmnet with OS bullseye |
[production] |
| 07:48 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 07:46 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1128 (re)pooling @ 60%: After onsite maintenance', diff saved to https://phabricator.wikimedia.org/P29429 and previous config saved to /var/cache/conftool/dbconfig/20220606-074638-root.json |
[production] |
| 07:44 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 07:44 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 07:41 |
<marostegui@deploy1002> |
Synchronized wmf-config/ProductionServices.php: Promote pc1014 to pc1 master T309612 (duration: 02m 53s) |
[production] |
| 07:39 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 07:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1128 (re)pooling @ 50%: After onsite maintenance', diff saved to https://phabricator.wikimedia.org/P29428 and previous config saved to /var/cache/conftool/dbconfig/20220606-073134-root.json |
[production] |
| 07:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1128 (re)pooling @ 40%: After onsite maintenance', diff saved to https://phabricator.wikimedia.org/P29427 and previous config saved to /var/cache/conftool/dbconfig/20220606-071630-root.json |
[production] |
| 07:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1128 (re)pooling @ 25%: After onsite maintenance', diff saved to https://phabricator.wikimedia.org/P29426 and previous config saved to /var/cache/conftool/dbconfig/20220606-070126-root.json |
[production] |
| 06:46 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1128 (re)pooling @ 20%: After onsite maintenance', diff saved to https://phabricator.wikimedia.org/P29425 and previous config saved to /var/cache/conftool/dbconfig/20220606-064622-root.json |
[production] |
| 06:38 |
<marostegui> |
Migrate pc1014 to mariadb 10.6.8 T309612 |
[production] |
| 06:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1128 (re)pooling @ 10%: After onsite maintenance', diff saved to https://phabricator.wikimedia.org/P29424 and previous config saved to /var/cache/conftool/dbconfig/20220606-063118-root.json |
[production] |
| 06:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1128 (re)pooling @ 5%: After onsite maintenance', diff saved to https://phabricator.wikimedia.org/P29423 and previous config saved to /var/cache/conftool/dbconfig/20220606-061614-root.json |
[production] |
| 06:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1128 (re)pooling @ 2%: After onsite maintenance', diff saved to https://phabricator.wikimedia.org/P29422 and previous config saved to /var/cache/conftool/dbconfig/20220606-060110-root.json |
[production] |
| 05:25 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully pool db1137 in x1 to with 10.6.8 T309679 ', diff saved to https://phabricator.wikimedia.org/P29421 and previous config saved to /var/cache/conftool/dbconfig/20220606-052546-marostegui.json |
[production] |
| 05:12 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Give more weight to db1137 in x1 to test 10.6.8 T309679 ', diff saved to https://phabricator.wikimedia.org/P29420 and previous config saved to /var/cache/conftool/dbconfig/20220606-051205-marostegui.json |
[production] |
| 05:07 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Pool db1128 on s1 with small weight after DIMM replacement T309303', diff saved to https://phabricator.wikimedia.org/P29419 and previous config saved to /var/cache/conftool/dbconfig/20220606-050707-root.json |
[production] |
| 05:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Add db1128 to dbctl T309303', diff saved to https://phabricator.wikimedia.org/P29418 and previous config saved to /var/cache/conftool/dbconfig/20220606-050616-marostegui.json |
[production] |
| 04:53 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 04:49 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 04:49 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 04:45 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 04:40 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 04:36 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 04:36 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 04:33 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 04:31 |
<tstarling@deploy1002> |
Synchronized wmf-config/db-production.php: enable SSL for cross-DC master connections (duration: 03m 10s) |
[production] |
|
2022-06-05
§
|
| 22:21 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1144:3314 (T298560)', diff saved to https://phabricator.wikimedia.org/P29417 and previous config saved to /var/cache/conftool/dbconfig/20220605-222110-ladsgroup.json |
[production] |
| 22:21 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1144.eqiad.wmnet with reason: Maintenance |
[production] |
| 22:21 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1144.eqiad.wmnet with reason: Maintenance |
[production] |
| 22:21 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1138 (T298560)', diff saved to https://phabricator.wikimedia.org/P29416 and previous config saved to /var/cache/conftool/dbconfig/20220605-222102-ladsgroup.json |
[production] |
| 22:05 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P29415 and previous config saved to /var/cache/conftool/dbconfig/20220605-220557-ladsgroup.json |
[production] |
| 21:50 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P29414 and previous config saved to /var/cache/conftool/dbconfig/20220605-215052-ladsgroup.json |
[production] |
| 21:35 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1138 (T298560)', diff saved to https://phabricator.wikimedia.org/P29413 and previous config saved to /var/cache/conftool/dbconfig/20220605-213547-ladsgroup.json |
[production] |
| 12:53 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1140.eqiad.wmnet with reason: Maintenance |
[production] |
| 12:53 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1140.eqiad.wmnet with reason: Maintenance |
[production] |
| 12:53 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1135 (T298560)', diff saved to https://phabricator.wikimedia.org/P29412 and previous config saved to /var/cache/conftool/dbconfig/20220605-125302-ladsgroup.json |
[production] |
| 12:37 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P29411 and previous config saved to /var/cache/conftool/dbconfig/20220605-123757-ladsgroup.json |
[production] |
| 12:27 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1138 (T298560)', diff saved to https://phabricator.wikimedia.org/P29410 and previous config saved to /var/cache/conftool/dbconfig/20220605-122702-ladsgroup.json |
[production] |