|
2025-06-23
ยง
|
| 10:05 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db1241 (T396130)', diff saved to https://phabricator.wikimedia.org/P78617 and previous config saved to /var/cache/conftool/dbconfig/20250623-100516-marostegui.json |
[production] |
| 10:05 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1241.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:04 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1238 (T396130)', diff saved to https://phabricator.wikimedia.org/P78616 and previous config saved to /var/cache/conftool/dbconfig/20250623-100453-marostegui.json |
[production] |
| 10:03 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1220 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P78615 and previous config saved to /var/cache/conftool/dbconfig/20250623-100342-root.json |
[production] |
| 10:00 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1222 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P78614 and previous config saved to /var/cache/conftool/dbconfig/20250623-100024-root.json |
[production] |
| 09:56 |
<ladsgroup@cumin1002> |
START - Cookbook sre.mysql.pool db2155 gradually with 4 steps - Work done |
[production] |
| 09:49 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1238', diff saved to https://phabricator.wikimedia.org/P78612 and previous config saved to /var/cache/conftool/dbconfig/20250623-094945-marostegui.json |
[production] |
| 09:48 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1220 (re)pooling @ 60%: Repooling', diff saved to https://phabricator.wikimedia.org/P78611 and previous config saved to /var/cache/conftool/dbconfig/20250623-094837-root.json |
[production] |
| 09:48 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1224 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P78610 and previous config saved to /var/cache/conftool/dbconfig/20250623-094822-root.json |
[production] |
| 09:48 |
<ladsgroup@cumin1002> |
END (PASS) - Cookbook sre.mysql.pool (exit_code=0) db1255 gradually with 4 steps - Work done |
[production] |
| 09:48 |
<ladsgroup@cumin1002> |
START - Cookbook sre.mysql.pool db1255 gradually with 4 steps - Work done |
[production] |
| 09:45 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1222 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P78609 and previous config saved to /var/cache/conftool/dbconfig/20250623-094519-root.json |
[production] |
| 09:34 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1238', diff saved to https://phabricator.wikimedia.org/P78608 and previous config saved to /var/cache/conftool/dbconfig/20250623-093438-marostegui.json |
[production] |
| 09:33 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1220 (re)pooling @ 40%: Repooling', diff saved to https://phabricator.wikimedia.org/P78607 and previous config saved to /var/cache/conftool/dbconfig/20250623-093331-root.json |
[production] |
| 09:33 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1224 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P78606 and previous config saved to /var/cache/conftool/dbconfig/20250623-093317-root.json |
[production] |
| 09:22 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1222.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:22 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool db1222', diff saved to https://phabricator.wikimedia.org/P78605 and previous config saved to /var/cache/conftool/dbconfig/20250623-092230-root.json |
[production] |
| 09:19 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1238 (T396130)', diff saved to https://phabricator.wikimedia.org/P78604 and previous config saved to /var/cache/conftool/dbconfig/20250623-091930-marostegui.json |
[production] |
| 09:18 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1220 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P78603 and previous config saved to /var/cache/conftool/dbconfig/20250623-091825-root.json |
[production] |
| 09:18 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1224 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P78602 and previous config saved to /var/cache/conftool/dbconfig/20250623-091811-root.json |
[production] |
| 09:11 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db1238 (T396130)', diff saved to https://phabricator.wikimedia.org/P78601 and previous config saved to /var/cache/conftool/dbconfig/20250623-091146-marostegui.json |
[production] |
| 09:11 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1238.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:11 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1221 (T396130)', diff saved to https://phabricator.wikimedia.org/P78600 and previous config saved to /var/cache/conftool/dbconfig/20250623-091123-marostegui.json |
[production] |
| 09:08 |
<taavi> |
restrict logging in to tools-sgebastion-10 (aka login-buster) T397459 |
[tools] |
| 09:07 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1220.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:06 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool db1220', diff saved to https://phabricator.wikimedia.org/P78599 and previous config saved to /var/cache/conftool/dbconfig/20250623-090619-root.json |
[production] |
| 09:03 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1224 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P78598 and previous config saved to /var/cache/conftool/dbconfig/20250623-090305-root.json |
[production] |
| 08:56 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P78597 and previous config saved to /var/cache/conftool/dbconfig/20250623-085616-marostegui.json |
[production] |
| 08:48 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1224 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P78596 and previous config saved to /var/cache/conftool/dbconfig/20250623-084800-root.json |
[production] |
| 08:44 |
<mvernon@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-be2007.codfw.wmnet |
[production] |
| 08:41 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P78595 and previous config saved to /var/cache/conftool/dbconfig/20250623-084108-marostegui.json |
[production] |
| 08:39 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool db1224', diff saved to https://phabricator.wikimedia.org/P78594 and previous config saved to /var/cache/conftool/dbconfig/20250623-083954-root.json |
[production] |
| 08:39 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1224.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:38 |
<mvernon@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host thanos-be2007.codfw.wmnet |
[production] |
| 08:36 |
<mvernon@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thanos-be2006.codfw.wmnet |
[production] |
| 08:31 |
<mvernon@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host thanos-be2006.codfw.wmnet |
[production] |
| 08:26 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1221 (T396130)', diff saved to https://phabricator.wikimedia.org/P78592 and previous config saved to /var/cache/conftool/dbconfig/20250623-082600-marostegui.json |
[production] |
| 08:20 |
<btullis> |
applying views fix to an-redacteddb1001 for T397508 |
[analytics] |
| 08:19 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db1221 (T396130)', diff saved to https://phabricator.wikimedia.org/P78591 and previous config saved to /var/cache/conftool/dbconfig/20250623-081920-marostegui.json |
[production] |
| 08:19 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1015,1019].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:18 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1221.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:18 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1199 (T396130)', diff saved to https://phabricator.wikimedia.org/P78590 and previous config saved to /var/cache/conftool/dbconfig/20250623-081839-marostegui.json |
[production] |
| 08:15 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db[1217,1228].eqiad.wmnet with reason: Maintenance |
[production] |
| 08:09 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2160.codfw.wmnet with reason: Maintenance |
[production] |
| 08:09 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2233.codfw.wmnet with reason: Maintenance |
[production] |
| 08:03 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P78588 and previous config saved to /var/cache/conftool/dbconfig/20250623-080332-marostegui.json |
[production] |
| 07:48 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P78587 and previous config saved to /var/cache/conftool/dbconfig/20250623-074824-marostegui.json |
[production] |
| 07:43 |
<stevemunene> |
restart 'hadoop-hdfs-zkfc.service' on an-master1003 T374922 |
[analytics] |
| 07:42 |
<brouberol@deploy1003> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply |
[production] |
| 07:41 |
<brouberol@deploy1003> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply |
[production] |