|
2024-06-11
ยง
|
| 07:09 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1233 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P64593 and previous config saved to /var/cache/conftool/dbconfig/20240611-070958-root.json |
[production] |
| 07:05 |
<arnaudb@deploy1002> |
Finished scap: Backport for [[gerrit:1041401|Revert "dbconfig: temporary disable writes on es6"]] (duration: 11m 36s) |
[production] |
| 07:02 |
<moritzm> |
failover ganeti master in codfw to ganeti2020 |
[production] |
| 06:57 |
<arnaudb@deploy1002> |
arnaudb: Continuing with sync |
[production] |
| 06:56 |
<arnaudb@deploy1002> |
arnaudb: Backport for [[gerrit:1041401|Revert "dbconfig: temporary disable writes on es6"]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 06:54 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1233 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P64592 and previous config saved to /var/cache/conftool/dbconfig/20240611-065453-root.json |
[production] |
| 06:54 |
<arnaudb@deploy1002> |
Started scap: Backport for [[gerrit:1041401|Revert "dbconfig: temporary disable writes on es6"]] |
[production] |
| 06:40 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'mimic weight', diff saved to https://phabricator.wikimedia.org/P64591 and previous config saved to /var/cache/conftool/dbconfig/20240611-064041-arnaudb.json |
[production] |
| 06:40 |
<oblivian@deploy1002> |
Unlocked for deployment [ALL REPOSITORIES]: incident in progress, blocking deploys --joe (duration: 15m 33s) |
[production] |
| 06:39 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1233 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P64590 and previous config saved to /var/cache/conftool/dbconfig/20240611-063947-root.json |
[production] |
| 06:39 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'mimic weight', diff saved to https://phabricator.wikimedia.org/P64589 and previous config saved to /var/cache/conftool/dbconfig/20240611-063903-arnaudb.json |
[production] |
| 06:31 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Promote es1037 to es6 primary T367055', diff saved to https://phabricator.wikimedia.org/P64588 and previous config saved to /var/cache/conftool/dbconfig/20240611-063109-arnaudb.json |
[production] |
| 06:30 |
<oblivian@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 06:30 |
<arnaudb> |
Starting es6 eqiad failover from es1038 to es1037 - T367055 |
[production] |
| 06:24 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1233 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P64587 and previous config saved to /var/cache/conftool/dbconfig/20240611-062441-root.json |
[production] |
| 06:24 |
<oblivian@deploy1002> |
Locking from deployment [ALL REPOSITORIES]: incident in progress, blocking deploys --joe |
[production] |
| 06:23 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Set es1037 with weight 0 T367055', diff saved to https://phabricator.wikimedia.org/P64586 and previous config saved to /var/cache/conftool/dbconfig/20240611-062353-arnaudb.json |
[production] |
| 06:23 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: Primary switchover es6 T367055 |
[production] |
| 06:23 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 6 hosts with reason: Primary switchover es6 T367055 |
[production] |
| 06:19 |
<oblivian@deploy1002> |
helmfile [codfw] START helmfile.d/services/mw-debug: apply |
[production] |
| 06:14 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2140 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P64585 and previous config saved to /var/cache/conftool/dbconfig/20240611-061413-root.json |
[production] |
| 06:12 |
<oblivian@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 06:11 |
<oblivian@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mw-debug: apply |
[production] |
| 06:09 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1233 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P64584 and previous config saved to /var/cache/conftool/dbconfig/20240611-060935-root.json |
[production] |
| 06:09 |
<oblivian@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 06:07 |
<oblivian@deploy1002> |
helmfile [codfw] START helmfile.d/services/mw-debug: apply |
[production] |
| 06:07 |
<arnaudb@deploy1002> |
Finished scap: Backport for [[gerrit:1041107|dbconfig: temporary disable writes on es6 (T367055)]] (duration: 15m 42s) |
[production] |
| 05:59 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2140 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P64583 and previous config saved to /var/cache/conftool/dbconfig/20240611-055907-root.json |
[production] |
| 05:58 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1233.eqiad.wmnet with reason: maintenance |
[production] |
| 05:58 |
<arnaudb@deploy1002> |
arnaudb: Continuing with sync |
[production] |
| 05:58 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1233.eqiad.wmnet with reason: maintenance |
[production] |
| 05:58 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'depool db1233', diff saved to https://phabricator.wikimedia.org/P64582 and previous config saved to /var/cache/conftool/dbconfig/20240611-055816-arnaudb.json |
[production] |
| 05:56 |
<arnaudb@deploy1002> |
arnaudb: Backport for [[gerrit:1041107|dbconfig: temporary disable writes on es6 (T367055)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 05:51 |
<arnaudb@deploy1002> |
Started scap: Backport for [[gerrit:1041107|dbconfig: temporary disable writes on es6 (T367055)]] |
[production] |
| 05:44 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2140 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P64581 and previous config saved to /var/cache/conftool/dbconfig/20240611-054401-root.json |
[production] |
| 05:28 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2140 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P64580 and previous config saved to /var/cache/conftool/dbconfig/20240611-052856-root.json |
[production] |
| 05:24 |
<marostegui> |
dbmaint eqiad s3 deploy schema change on db1223 T364069 |
[production] |
| 05:22 |
<marostegui> |
dbmaint eqiad s3 deploy schema change on db1223 T364299 |
[production] |
| 05:21 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1223.eqiad.wmnet with reason: Long schema change |
[production] |
| 05:21 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1223.eqiad.wmnet with reason: Long schema change |
[production] |
| 05:21 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool db1223 T367140', diff saved to https://phabricator.wikimedia.org/P64579 and previous config saved to /var/cache/conftool/dbconfig/20240611-052101-root.json |
[production] |
| 05:20 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Promote db1157 to s3 primary and set section read-write T367140', diff saved to https://phabricator.wikimedia.org/P64578 and previous config saved to /var/cache/conftool/dbconfig/20240611-052000-root.json |
[production] |
| 05:19 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Set s3 eqiad as read-only for maintenance - T367140', diff saved to https://phabricator.wikimedia.org/P64577 and previous config saved to /var/cache/conftool/dbconfig/20240611-051941-root.json |
[production] |
| 05:19 |
<marostegui> |
Starting s3 eqiad failover from db1223 to db1157 - T367140 |
[production] |
| 05:13 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2140 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P64576 and previous config saved to /var/cache/conftool/dbconfig/20240611-051351-root.json |
[production] |
| 05:04 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Primary switchover s3 T367140 |
[production] |
| 05:03 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Set db1157 with weight 0 T367140', diff saved to https://phabricator.wikimedia.org/P64575 and previous config saved to /var/cache/conftool/dbconfig/20240611-050351-root.json |
[production] |
| 05:03 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 24 hosts with reason: Primary switchover s3 T367140 |
[production] |
| 04:58 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2140 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P64574 and previous config saved to /var/cache/conftool/dbconfig/20240611-045845-root.json |
[production] |
| 04:57 |
<marostegui> |
dbmaint eqiad s2 deploy schema change on db1222 T364299 |
[production] |