|
2024-07-11
ยง
|
| 07:45 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Promote db2205 to s3 primary T369691', diff saved to https://phabricator.wikimedia.org/P66275 and previous config saved to /var/cache/conftool/dbconfig/20240711-074534-marostegui.json |
[production] |
| 07:45 |
<marostegui> |
Starting s3 codfw failover from db2127 to db2205 - T369691 |
[production] |
| 07:42 |
<dcausse@deploy1002> |
dcausse: Continuing with sync |
[production] |
| 07:41 |
<dcausse@deploy1002> |
dcausse: Backport for [[gerrit:1053533|Fix pool counter metric]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 07:37 |
<dcausse@deploy1002> |
Started scap sync-world: Backport for [[gerrit:1053533|Fix pool counter metric]] |
[production] |
| 07:31 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 25 hosts with reason: Primary switchover s3 T369691 |
[production] |
| 07:31 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Set db2205 with weight 0 T369691', diff saved to https://phabricator.wikimedia.org/P66274 and previous config saved to /var/cache/conftool/dbconfig/20240711-073101-root.json |
[production] |
| 07:30 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1150.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:30 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 25 hosts with reason: Primary switchover s3 T369691 |
[production] |
| 07:30 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1150.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:28 |
<jgiannelos@deploy1002> |
Finished scap: Backport for [[gerrit:1053006|Linter: trigger parsoid parses on template changes (T361013)]] (duration: 14m 25s) |
[production] |
| 07:23 |
<jgiannelos@deploy1002> |
daniel, jgiannelos: Continuing with sync |
[production] |
| 07:17 |
<jgiannelos@deploy1002> |
daniel, jgiannelos: Backport for [[gerrit:1053006|Linter: trigger parsoid parses on template changes (T361013)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 07:14 |
<jgiannelos@deploy1002> |
Started scap sync-world: Backport for [[gerrit:1053006|Linter: trigger parsoid parses on template changes (T361013)]] |
[production] |
| 07:12 |
<kartik@deploy1002> |
Finished scap: Backport for [[gerrit:1053429|Enable MinT for Wikipedia readers MVP on a second group of pilot wikis (T367067)]] (duration: 09m 32s) |
[production] |
| 07:07 |
<kartik@deploy1002> |
kartik: Continuing with sync |
[production] |
| 07:05 |
<kartik@deploy1002> |
kartik: Backport for [[gerrit:1053429|Enable MinT for Wikipedia readers MVP on a second group of pilot wikis (T367067)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 07:02 |
<kartik@deploy1002> |
Started scap sync-world: Backport for [[gerrit:1053429|Enable MinT for Wikipedia readers MVP on a second group of pilot wikis (T367067)]] |
[production] |
| 07:00 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1163 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P66273 and previous config saved to /var/cache/conftool/dbconfig/20240711-070004-root.json |
[production] |
| 06:55 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2200.codfw.wmnet with reason: Maintenance |
[production] |
| 06:55 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 4:00:00 on db2200.codfw.wmnet with reason: Maintenance |
[production] |
| 06:55 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2198.codfw.wmnet with reason: Maintenance |
[production] |
| 06:55 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 4:00:00 on db2198.codfw.wmnet with reason: Maintenance |
[production] |
| 06:55 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2195 (T367781)', diff saved to https://phabricator.wikimedia.org/P66272 and previous config saved to /var/cache/conftool/dbconfig/20240711-065508-arnaudb.json |
[production] |
| 06:54 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2209 (T367856)', diff saved to https://phabricator.wikimedia.org/P66271 and previous config saved to /var/cache/conftool/dbconfig/20240711-065432-marostegui.json |
[production] |
| 06:29 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1163 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P66267 and previous config saved to /var/cache/conftool/dbconfig/20240711-062953-root.json |
[production] |
| 06:24 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P66266 and previous config saved to /var/cache/conftool/dbconfig/20240711-062454-arnaudb.json |
[production] |
| 06:24 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2209', diff saved to https://phabricator.wikimedia.org/P66265 and previous config saved to /var/cache/conftool/dbconfig/20240711-062417-marostegui.json |
[production] |
| 06:14 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1163 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P66264 and previous config saved to /var/cache/conftool/dbconfig/20240711-061447-root.json |
[production] |
| 06:09 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2195 (T367781)', diff saved to https://phabricator.wikimedia.org/P66263 and previous config saved to /var/cache/conftool/dbconfig/20240711-060947-arnaudb.json |
[production] |
| 06:09 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2209 (T367856)', diff saved to https://phabricator.wikimedia.org/P66262 and previous config saved to /var/cache/conftool/dbconfig/20240711-060910-marostegui.json |
[production] |
| 06:07 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Depooling db2195 (T367781)', diff saved to https://phabricator.wikimedia.org/P66261 and previous config saved to /var/cache/conftool/dbconfig/20240711-060736-arnaudb.json |
[production] |
| 06:07 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2195.codfw.wmnet with reason: Maintenance |
[production] |
| 06:07 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 4:00:00 on db2195.codfw.wmnet with reason: Maintenance |
[production] |
| 06:07 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2181 (T367781)', diff saved to https://phabricator.wikimedia.org/P66260 and previous config saved to /var/cache/conftool/dbconfig/20240711-060714-arnaudb.json |
[production] |
| 05:59 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1163 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P66259 and previous config saved to /var/cache/conftool/dbconfig/20240711-055942-root.json |
[production] |
| 05:52 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P66258 and previous config saved to /var/cache/conftool/dbconfig/20240711-055206-arnaudb.json |
[production] |
| 05:44 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1163 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P66257 and previous config saved to /var/cache/conftool/dbconfig/20240711-054436-root.json |
[production] |
| 05:36 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P66256 and previous config saved to /var/cache/conftool/dbconfig/20240711-053659-arnaudb.json |
[production] |
| 05:29 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1163 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P66255 and previous config saved to /var/cache/conftool/dbconfig/20240711-052931-root.json |
[production] |
| 05:27 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool db1163 T369514', diff saved to https://phabricator.wikimedia.org/P66254 and previous config saved to /var/cache/conftool/dbconfig/20240711-052702-root.json |
[production] |
| 05:25 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Promote db1184 to s1 primary and set section read-write T369514', diff saved to https://phabricator.wikimedia.org/P66253 and previous config saved to /var/cache/conftool/dbconfig/20240711-052540-root.json |
[production] |
| 05:25 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Set s1 eqiad as read-only for maintenance - T369514', diff saved to https://phabricator.wikimedia.org/P66252 and previous config saved to /var/cache/conftool/dbconfig/20240711-052507-root.json |
[production] |
| 05:24 |
<marostegui> |
Starting s1 eqiad failover from db1163 to db1184 - T369514 |
[production] |
| 05:21 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2181 (T367781)', diff saved to https://phabricator.wikimedia.org/P66251 and previous config saved to /var/cache/conftool/dbconfig/20240711-052151-arnaudb.json |
[production] |
| 05:19 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Depooling db2181 (T367781)', diff saved to https://phabricator.wikimedia.org/P66250 and previous config saved to /var/cache/conftool/dbconfig/20240711-051941-arnaudb.json |
[production] |
| 05:19 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2181.codfw.wmnet with reason: Maintenance |
[production] |
| 05:19 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 4:00:00 on db2181.codfw.wmnet with reason: Maintenance |
[production] |
| 05:19 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2167 (T367781)', diff saved to https://phabricator.wikimedia.org/P66249 and previous config saved to /var/cache/conftool/dbconfig/20240711-051920-arnaudb.json |
[production] |
| 05:04 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P66248 and previous config saved to /var/cache/conftool/dbconfig/20240711-050413-arnaudb.json |
[production] |