|
2024-02-19
ยง
|
| 14:22 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1201 (T357189)', diff saved to https://phabricator.wikimedia.org/P57075 and previous config saved to /var/cache/conftool/dbconfig/20240219-142238-arnaudb.json |
[production] |
| 14:20 |
<akosiaris@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply |
[production] |
| 14:20 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2119', diff saved to https://phabricator.wikimedia.org/P57074 and previous config saved to /var/cache/conftool/dbconfig/20240219-142044-ladsgroup.json |
[production] |
| 14:19 |
<akosiaris@deploy2002> |
helmfile [codfw] START helmfile.d/services/wikifunctions: apply |
[production] |
| 14:19 |
<akosiaris@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply |
[production] |
| 14:19 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Depooling db1201 (T357189)', diff saved to https://phabricator.wikimedia.org/P57073 and previous config saved to /var/cache/conftool/dbconfig/20240219-141919-arnaudb.json |
[production] |
| 14:19 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1201.eqiad.wmnet with reason: Maintenance |
[production] |
| 14:19 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1201.eqiad.wmnet with reason: Maintenance |
[production] |
| 14:18 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1187 (T357189)', diff saved to https://phabricator.wikimedia.org/P57072 and previous config saved to /var/cache/conftool/dbconfig/20240219-141858-arnaudb.json |
[production] |
| 14:18 |
<akosiaris@deploy2002> |
helmfile [eqiad] START helmfile.d/services/wikifunctions: apply |
[production] |
| 14:18 |
<akosiaris@deploy2002> |
helmfile [staging] DONE helmfile.d/services/wikifunctions: apply |
[production] |
| 14:18 |
<akosiaris@deploy2002> |
helmfile [staging] START helmfile.d/services/wikifunctions: apply |
[production] |
| 14:16 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2169:3317 (re)pooling @ 75%: Cloning to db2194 done', diff saved to https://phabricator.wikimedia.org/P57071 and previous config saved to /var/cache/conftool/dbconfig/20240219-141640-arnaudb.json |
[production] |
| 14:14 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2170 (re)pooling @ 50%: After rearraging sections T354826', diff saved to https://phabricator.wikimedia.org/P57070 and previous config saved to /var/cache/conftool/dbconfig/20240219-141412-root.json |
[production] |
| 14:12 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es1020 (re)pooling @ 10%: After migration to 10.6', diff saved to https://phabricator.wikimedia.org/P57069 and previous config saved to /var/cache/conftool/dbconfig/20240219-141227-root.json |
[production] |
| 14:05 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2119 (T352010)', diff saved to https://phabricator.wikimedia.org/P57068 and previous config saved to /var/cache/conftool/dbconfig/20240219-140538-ladsgroup.json |
[production] |
| 14:03 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P57067 and previous config saved to /var/cache/conftool/dbconfig/20240219-140351-arnaudb.json |
[production] |
| 14:01 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2169:3317 (re)pooling @ 50%: Cloning to db2194 done', diff saved to https://phabricator.wikimedia.org/P57066 and previous config saved to /var/cache/conftool/dbconfig/20240219-140135-arnaudb.json |
[production] |
| 13:59 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2170 (re)pooling @ 25%: After rearraging sections T354826', diff saved to https://phabricator.wikimedia.org/P57065 and previous config saved to /var/cache/conftool/dbconfig/20240219-135907-root.json |
[production] |
| 13:58 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on dbproxy1027.eqiad.wmnet with reason: Silence for reboot T356240 |
[production] |
| 13:58 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 0:30:00 on dbproxy1027.eqiad.wmnet with reason: Silence for reboot T356240 |
[production] |
| 13:58 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on dbproxy1026.eqiad.wmnet with reason: Silence for reboot T356240 |
[production] |
| 13:58 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 0:30:00 on dbproxy1026.eqiad.wmnet with reason: Silence for reboot T356240 |
[production] |
| 13:58 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on dbproxy1024.eqiad.wmnet with reason: Silence for reboot T356240 |
[production] |
| 13:58 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 0:30:00 on dbproxy1024.eqiad.wmnet with reason: Silence for reboot T356240 |
[production] |
| 13:57 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on dbproxy1023.eqiad.wmnet with reason: Silence for reboot T356240 |
[production] |
| 13:57 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 0:30:00 on dbproxy1023.eqiad.wmnet with reason: Silence for reboot T356240 |
[production] |
| 13:57 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es1020 (re)pooling @ 5%: After migration to 10.6', diff saved to https://phabricator.wikimedia.org/P57064 and previous config saved to /var/cache/conftool/dbconfig/20240219-135722-root.json |
[production] |
| 13:48 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P57063 and previous config saved to /var/cache/conftool/dbconfig/20240219-134845-arnaudb.json |
[production] |
| 13:48 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool es1020', diff saved to https://phabricator.wikimedia.org/P57062 and previous config saved to /var/cache/conftool/dbconfig/20240219-134804-root.json |
[production] |
| 13:46 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2169:3317 (re)pooling @ 40%: Cloning to db2194 done', diff saved to https://phabricator.wikimedia.org/P57061 and previous config saved to /var/cache/conftool/dbconfig/20240219-134630-arnaudb.json |
[production] |
| 13:45 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Promote es1021 to es4 primary ', diff saved to https://phabricator.wikimedia.org/P57060 and previous config saved to /var/cache/conftool/dbconfig/20240219-134551-root.json |
[production] |
| 13:44 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2170 (re)pooling @ 10%: After rearraging sections T354826', diff saved to https://phabricator.wikimedia.org/P57059 and previous config saved to /var/cache/conftool/dbconfig/20240219-134402-root.json |
[production] |
| 13:43 |
<marostegui> |
Starting es4 eqiad failover from es1020 to es1021 - T357904 |
[production] |
| 13:42 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Change weight of es1021', diff saved to https://phabricator.wikimedia.org/P57058 and previous config saved to /var/cache/conftool/dbconfig/20240219-134205-root.json |
[production] |
| 13:40 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: es4 switchover T357904 |
[production] |
| 13:40 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 6 hosts with reason: es4 switchover T357904 |
[production] |
| 13:38 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on dbproxy1021.eqiad.wmnet with reason: Silence for reboot T356240 |
[production] |
| 13:37 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 0:30:00 on dbproxy1021.eqiad.wmnet with reason: Silence for reboot T356240 |
[production] |
| 13:36 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on dbproxy1020.eqiad.wmnet with reason: Silence for reboot T356240 |
[production] |
| 13:35 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 0:30:00 on dbproxy1020.eqiad.wmnet with reason: Silence for reboot T356240 |
[production] |
| 13:33 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1187 (T357189)', diff saved to https://phabricator.wikimedia.org/P57057 and previous config saved to /var/cache/conftool/dbconfig/20240219-133339-arnaudb.json |
[production] |
| 13:32 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool es1021', diff saved to https://phabricator.wikimedia.org/P57056 and previous config saved to /var/cache/conftool/dbconfig/20240219-133245-root.json |
[production] |
| 13:31 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2169:3317 (re)pooling @ 30%: Cloning to db2194 done', diff saved to https://phabricator.wikimedia.org/P57055 and previous config saved to /var/cache/conftool/dbconfig/20240219-133125-arnaudb.json |
[production] |
| 13:30 |
<moritzm> |
installing runc security updates on buster |
[production] |
| 13:30 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Depooling db1187 (T357189)', diff saved to https://phabricator.wikimedia.org/P57054 and previous config saved to /var/cache/conftool/dbconfig/20240219-133019-arnaudb.json |
[production] |
| 13:30 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1187.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:30 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1187.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:29 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1180 (T357189)', diff saved to https://phabricator.wikimedia.org/P57053 and previous config saved to /var/cache/conftool/dbconfig/20240219-132958-arnaudb.json |
[production] |
| 13:28 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2170 (re)pooling @ 5%: After rearraging sections T354826', diff saved to https://phabricator.wikimedia.org/P57052 and previous config saved to /var/cache/conftool/dbconfig/20240219-132858-root.json |
[production] |