|
2025-06-23
ยง
|
| 08:38 |
<mvernon@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host thanos-be2007.codfw.wmnet |
[production] |
| 08:36 |
<mvernon@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-be2006.codfw.wmnet |
[production] |
| 08:31 |
<mvernon@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host thanos-be2006.codfw.wmnet |
[production] |
| 08:26 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1221 (T396130)', diff saved to https://phabricator.wikimedia.org/P78592 and previous config saved to /var/cache/conftool/dbconfig/20250623-082600-marostegui.json |
[production] |
| 08:19 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db1221 (T396130)', diff saved to https://phabricator.wikimedia.org/P78591 and previous config saved to /var/cache/conftool/dbconfig/20250623-081920-marostegui.json |
[production] |
| 08:19 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1015,1019].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:18 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1221.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:18 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1199 (T396130)', diff saved to https://phabricator.wikimedia.org/P78590 and previous config saved to /var/cache/conftool/dbconfig/20250623-081839-marostegui.json |
[production] |
| 08:15 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db[1217,1228].eqiad.wmnet with reason: Maintenance |
[production] |
| 08:09 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2160.codfw.wmnet with reason: Maintenance |
[production] |
| 08:09 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2233.codfw.wmnet with reason: Maintenance |
[production] |
| 08:03 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P78588 and previous config saved to /var/cache/conftool/dbconfig/20250623-080332-marostegui.json |
[production] |
| 07:48 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P78587 and previous config saved to /var/cache/conftool/dbconfig/20250623-074824-marostegui.json |
[production] |
| 07:42 |
<brouberol@deploy1003> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply |
[production] |
| 07:41 |
<brouberol@deploy1003> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply |
[production] |
| 07:37 |
<hashar@deploy1003> |
Finished scap sync-world: Backport for [[gerrit:1161622|ApiQueryZFunctionReference: Return an actual empty array instead of [false] (T396978)]], [[gerrit:1154121|captureSpeedtest: Drop PHP 7 check, no longer needed]], [[gerrit:1156351|diffConfig: Add a quick list of affected wikis to the end of the output]] (duration: 41m 07s) |
[production] |
| 07:33 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1199 (T396130)', diff saved to https://phabricator.wikimedia.org/P78586 and previous config saved to /var/cache/conftool/dbconfig/20250623-073316-marostegui.json |
[production] |
| 07:31 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es2039 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P78585 and previous config saved to /var/cache/conftool/dbconfig/20250623-073145-root.json |
[production] |
| 07:31 |
<stevemunene@cumin1002> |
END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) for Zookeeper A:zookeeper-analytics cluster: Roll restart of jvm daemons. |
[production] |
| 07:25 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db1199 (T396130)', diff saved to https://phabricator.wikimedia.org/P78584 and previous config saved to /var/cache/conftool/dbconfig/20250623-072542-marostegui.json |
[production] |
| 07:25 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1199.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:25 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1190 (T396130)', diff saved to https://phabricator.wikimedia.org/P78583 and previous config saved to /var/cache/conftool/dbconfig/20250623-072519-marostegui.json |
[production] |
| 07:25 |
<stevemunene@cumin1002> |
START - Cookbook sre.zookeeper.roll-restart-zookeeper for Zookeeper A:zookeeper-analytics cluster: Roll restart of jvm daemons. |
[production] |
| 07:24 |
<hashar@deploy1003> |
hashar, jforrester: Continuing with sync |
[production] |
| 07:20 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1228.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:18 |
<hashar@deploy1003> |
hashar, jforrester: Backport for [[gerrit:1161622|ApiQueryZFunctionReference: Return an actual empty array instead of [false] (T396978)]], [[gerrit:1154121|captureSpeedtest: Drop PHP 7 check, no longer needed]], [[gerrit:1156351|diffConfig: Add a quick list of affected wikis to the end of the output]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be ver |
[production] |
| 07:16 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es2039 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P78582 and previous config saved to /var/cache/conftool/dbconfig/20250623-071639-root.json |
[production] |
| 07:16 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es2035 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P78581 and previous config saved to /var/cache/conftool/dbconfig/20250623-071618-root.json |
[production] |
| 07:10 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P78580 and previous config saved to /var/cache/conftool/dbconfig/20250623-071011-marostegui.json |
[production] |
| 07:06 |
<marostegui> |
Failover m5 from db1228 to db1164 - T397413 |
[production] |
| 07:01 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es2039 (re)pooling @ 60%: Repooling', diff saved to https://phabricator.wikimedia.org/P78579 and previous config saved to /var/cache/conftool/dbconfig/20250623-070134-root.json |
[production] |
| 07:01 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es2035 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P78578 and previous config saved to /var/cache/conftool/dbconfig/20250623-070112-root.json |
[production] |
| 06:58 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2235].codfw.wmnet,db[1164,1217,1228].eqiad.wmnet with reason: m5 master switch T397413 |
[production] |
| 06:56 |
<hashar@deploy1003> |
Started scap sync-world: Backport for [[gerrit:1161622|ApiQueryZFunctionReference: Return an actual empty array instead of [false] (T396978)]], [[gerrit:1154121|captureSpeedtest: Drop PHP 7 check, no longer needed]], [[gerrit:1156351|diffConfig: Add a quick list of affected wikis to the end of the output]] |
[production] |
| 06:55 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P78577 and previous config saved to /var/cache/conftool/dbconfig/20250623-065503-marostegui.json |
[production] |
| 06:48 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2235].codfw.wmnet,db[1164,1217,1228].eqiad.wmnet with reason: m5 master switch T397413 |
[production] |
| 06:46 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es2039 (re)pooling @ 40%: Repooling', diff saved to https://phabricator.wikimedia.org/P78576 and previous config saved to /var/cache/conftool/dbconfig/20250623-064628-root.json |
[production] |
| 06:46 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es2035 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P78575 and previous config saved to /var/cache/conftool/dbconfig/20250623-064606-root.json |
[production] |
| 06:43 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2215 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P78574 and previous config saved to /var/cache/conftool/dbconfig/20250623-064358-root.json |
[production] |
| 06:39 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1190 (T396130)', diff saved to https://phabricator.wikimedia.org/P78573 and previous config saved to /var/cache/conftool/dbconfig/20250623-063956-marostegui.json |
[production] |
| 06:32 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db1190 (T396130)', diff saved to https://phabricator.wikimedia.org/P78572 and previous config saved to /var/cache/conftool/dbconfig/20250623-063217-marostegui.json |
[production] |
| 06:32 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1190.eqiad.wmnet with reason: Maintenance |
[production] |
| 06:31 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1160 (T396130)', diff saved to https://phabricator.wikimedia.org/P78571 and previous config saved to /var/cache/conftool/dbconfig/20250623-063155-marostegui.json |
[production] |
| 06:31 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es2039 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P78570 and previous config saved to /var/cache/conftool/dbconfig/20250623-063123-root.json |
[production] |
| 06:31 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es2035 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P78569 and previous config saved to /var/cache/conftool/dbconfig/20250623-063100-root.json |
[production] |
| 06:30 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool es2039 T397599', diff saved to https://phabricator.wikimedia.org/P78568 and previous config saved to /var/cache/conftool/dbconfig/20250623-063050-marostegui.json |
[production] |
| 06:29 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Promote es2038 to es7 primary and set section read-write T397599', diff saved to https://phabricator.wikimedia.org/P78567 and previous config saved to /var/cache/conftool/dbconfig/20250623-062949-marostegui.json |
[production] |
| 06:29 |
<marostegui> |
Starting es7 codfw failover from es2039 to es2038 - T397599 |
[production] |
| 06:28 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2215 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P78566 and previous config saved to /var/cache/conftool/dbconfig/20250623-062852-root.json |
[production] |
| 06:24 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Set es2038 with weight 0 T397599', diff saved to https://phabricator.wikimedia.org/P78565 and previous config saved to /var/cache/conftool/dbconfig/20250623-062420-root.json |
[production] |