|
2025-02-07
ยง
|
| 11:16 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2145 (re)pooling @ 100%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73389 and previous config saved to /var/cache/conftool/dbconfig/20250207-111619-root.json |
[production] |
| 11:14 |
<jmm@cumin2002> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ganeti1033.eqiad.wmnet |
[production] |
| 11:01 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2145 (re)pooling @ 75%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73388 and previous config saved to /var/cache/conftool/dbconfig/20250207-110114-root.json |
[production] |
| 10:48 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1174 (re)pooling @ 100%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73387 and previous config saved to /var/cache/conftool/dbconfig/20250207-104818-root.json |
[production] |
| 10:46 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2145 (re)pooling @ 50%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73386 and previous config saved to /var/cache/conftool/dbconfig/20250207-104609-root.json |
[production] |
| 10:37 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1234 (re)pooling @ 100%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73385 and previous config saved to /var/cache/conftool/dbconfig/20250207-103710-root.json |
[production] |
| 10:33 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1174 (re)pooling @ 75%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73384 and previous config saved to /var/cache/conftool/dbconfig/20250207-103312-root.json |
[production] |
| 10:31 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2145 (re)pooling @ 25%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73383 and previous config saved to /var/cache/conftool/dbconfig/20250207-103104-root.json |
[production] |
| 10:30 |
<fnegri@cumin1002> |
END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for clouddb1014.eqiad.wmnet |
[production] |
| 10:30 |
<fnegri@cumin1002> |
START - Cookbook sre.hosts.remove-downtime for clouddb1014.eqiad.wmnet |
[production] |
| 10:24 |
<fnegri@cumin1002> |
conftool action : set/pooled=yes; selector: name=clouddb1014.eqiad.wmnet,service=s7 |
[production] |
| 10:24 |
<fnegri@cumin1002> |
conftool action : set/pooled=yes; selector: name=clouddb1014.eqiad.wmnet,service=s2 |
[production] |
| 10:22 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1234 (re)pooling @ 75%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73382 and previous config saved to /var/cache/conftool/dbconfig/20250207-102205-root.json |
[production] |
| 10:18 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1174 (re)pooling @ 50%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73381 and previous config saved to /var/cache/conftool/dbconfig/20250207-101807-root.json |
[production] |
| 10:15 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2145 (re)pooling @ 10%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73380 and previous config saved to /var/cache/conftool/dbconfig/20250207-101559-root.json |
[production] |
| 10:08 |
<fnegri@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on clouddb1014.eqiad.wmnet with reason: Rebooting clouddb1014 T384946 |
[production] |
| 10:07 |
<fnegri@cumin1002> |
conftool action : set/pooled=no; selector: name=clouddb1014.eqiad.wmnet,service=s2 |
[production] |
| 10:07 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1234 (re)pooling @ 50%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73379 and previous config saved to /var/cache/conftool/dbconfig/20250207-100700-root.json |
[production] |
| 10:07 |
<fnegri@cumin1002> |
conftool action : set/pooled=no; selector: name=clouddb1014.eqiad.wmnet,service=s7 |
[production] |
| 10:03 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1174 (re)pooling @ 25%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73378 and previous config saved to /var/cache/conftool/dbconfig/20250207-100302-root.json |
[production] |
| 09:51 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1234 (re)pooling @ 25%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73377 and previous config saved to /var/cache/conftool/dbconfig/20250207-095154-root.json |
[production] |
| 09:47 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1174 (re)pooling @ 10%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73376 and previous config saved to /var/cache/conftool/dbconfig/20250207-094756-root.json |
[production] |
| 09:36 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1234 (re)pooling @ 10%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P73375 and previous config saved to /var/cache/conftool/dbconfig/20250207-093649-root.json |
[production] |
| 09:15 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db2206 (T384592)', diff saved to https://phabricator.wikimedia.org/P73374 and previous config saved to /var/cache/conftool/dbconfig/20250207-091459-marostegui.json |
[production] |
| 09:14 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2206.codfw.wmnet with reason: Maintenance |
[production] |
| 08:06 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es1030 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P73373 and previous config saved to /var/cache/conftool/dbconfig/20250207-080638-root.json |
[production] |
| 08:02 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es1027 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P73372 and previous config saved to /var/cache/conftool/dbconfig/20250207-080218-root.json |
[production] |
| 07:51 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es1030 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P73371 and previous config saved to /var/cache/conftool/dbconfig/20250207-075132-root.json |
[production] |
| 07:47 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es1027 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P73370 and previous config saved to /var/cache/conftool/dbconfig/20250207-074712-root.json |
[production] |
| 07:36 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es1030 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P73369 and previous config saved to /var/cache/conftool/dbconfig/20250207-073627-root.json |
[production] |
| 07:32 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es1027 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P73368 and previous config saved to /var/cache/conftool/dbconfig/20250207-073207-root.json |
[production] |
| 07:21 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es1030 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P73367 and previous config saved to /var/cache/conftool/dbconfig/20250207-072122-root.json |
[production] |
| 07:17 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es1027 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P73366 and previous config saved to /var/cache/conftool/dbconfig/20250207-071702-root.json |
[production] |
| 07:13 |
<kevinbazira@deploy2002> |
helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'article-models' for release 'main' . |
[production] |
| 07:12 |
<kevinbazira@deploy2002> |
helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'article-models' for release 'main' . |
[production] |
| 07:08 |
<kevinbazira@deploy2002> |
helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'article-models' for release 'main' . |
[production] |
| 07:06 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es1030 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P73365 and previous config saved to /var/cache/conftool/dbconfig/20250207-070617-root.json |
[production] |
| 07:06 |
<root@cumin1002> |
END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for es1030.eqiad.wmnet |
[production] |
| 07:01 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es1027 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P73364 and previous config saved to /var/cache/conftool/dbconfig/20250207-070156-root.json |
[production] |
| 07:01 |
<root@cumin1002> |
END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for es1027.eqiad.wmnet |
[production] |
| 06:57 |
<root@cumin1002> |
START - Cookbook sre.mysql.upgrade for es1030.eqiad.wmnet |
[production] |
| 06:57 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool es1030', diff saved to https://phabricator.wikimedia.org/P73363 and previous config saved to /var/cache/conftool/dbconfig/20250207-065730-marostegui.json |
[production] |
| 06:57 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Promote es1026 to es2 master', diff saved to https://phabricator.wikimedia.org/P73362 and previous config saved to /var/cache/conftool/dbconfig/20250207-065700-root.json |
[production] |
| 06:56 |
<root@cumin1002> |
START - Cookbook sre.mysql.upgrade for es1027.eqiad.wmnet |
[production] |
| 06:56 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool es1027', diff saved to https://phabricator.wikimedia.org/P73361 and previous config saved to /var/cache/conftool/dbconfig/20250207-065600-marostegui.json |
[production] |
| 06:55 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Promote es1029 to es1 master', diff saved to https://phabricator.wikimedia.org/P73360 and previous config saved to /var/cache/conftool/dbconfig/20250207-065546-root.json |
[production] |
| 06:36 |
<root@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1234.eqiad.wmnet with reason: Index rebuild |
[production] |
| 06:36 |
<root@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2145.codfw.wmnet with reason: Index rebuild |
[production] |
| 06:36 |
<root@cumin1002> |
END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2145.codfw.wmnet |
[production] |
| 06:35 |
<root@cumin1002> |
END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1234.eqiad.wmnet |
[production] |