|
2026-04-17
§
|
| 14:42 |
<fceratto@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1233 (T419961)', diff saved to https://phabricator.wikimedia.org/P91080 and previous config saved to /var/cache/conftool/dbconfig/20260417-144247-fceratto.json |
[production] |
| 14:36 |
<dpogorzelski@deploy1003> |
helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'edit-check' for release 'main' . |
[production] |
| 14:36 |
<dpogorzelski@deploy1003> |
helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. |
[production] |
| 14:34 |
<fceratto@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2182 (T419635)', diff saved to https://phabricator.wikimedia.org/P91079 and previous config saved to /var/cache/conftool/dbconfig/20260417-143416-fceratto.json |
[production] |
| 14:33 |
<dpogorzelski@deploy1003> |
helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. |
[production] |
| 14:33 |
<dpogorzelski@deploy1003> |
helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. |
[production] |
| 14:33 |
<dpogorzelski@deploy1003> |
helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. |
[production] |
| 14:32 |
<dpogorzelski@deploy1003> |
helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. |
[production] |
| 14:32 |
<dpogorzelski@deploy1003> |
helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. |
[production] |
| 14:32 |
<fceratto@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P91078 and previous config saved to /var/cache/conftool/dbconfig/20260417-143238-fceratto.json |
[production] |
| 14:32 |
<fceratto@cumin1003> |
dbctl commit (dc=all): 'Depooling db2182 (T419635)', diff saved to https://phabricator.wikimedia.org/P91077 and previous config saved to /var/cache/conftool/dbconfig/20260417-143204-fceratto.json |
[production] |
| 14:31 |
<fceratto@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2182.codfw.wmnet with reason: Maintenance |
[production] |
| 14:31 |
<fceratto@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2168 (T419635)', diff saved to https://phabricator.wikimedia.org/P91076 and previous config saved to /var/cache/conftool/dbconfig/20260417-143139-fceratto.json |
[production] |
| 14:22 |
<fceratto@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P91075 and previous config saved to /var/cache/conftool/dbconfig/20260417-142230-fceratto.json |
[production] |
| 14:21 |
<fceratto@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2168', diff saved to https://phabricator.wikimedia.org/P91074 and previous config saved to /var/cache/conftool/dbconfig/20260417-142130-fceratto.json |
[production] |
| 14:18 |
<dpogorzelski@deploy1003> |
helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. |
[production] |
| 14:16 |
<dpogorzelski@deploy1003> |
helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. |
[production] |
| 14:16 |
<dpogorzelski@deploy1003> |
helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. |
[production] |
| 14:14 |
<dpogorzelski@deploy1003> |
helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. |
[production] |
| 14:13 |
<dpogorzelski@deploy1003> |
helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. |
[production] |
| 14:12 |
<dpogorzelski@deploy1003> |
helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. |
[production] |
| 14:12 |
<dpogorzelski@deploy1003> |
helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. |
[production] |
| 14:12 |
<fceratto@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1233 (T419961)', diff saved to https://phabricator.wikimedia.org/P91073 and previous config saved to /var/cache/conftool/dbconfig/20260417-141222-fceratto.json |
[production] |
| 14:12 |
<dpogorzelski@deploy1003> |
helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. |
[production] |
| 14:11 |
<dpogorzelski@deploy1003> |
helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. |
[production] |
| 14:11 |
<fceratto@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2168', diff saved to https://phabricator.wikimedia.org/P91072 and previous config saved to /var/cache/conftool/dbconfig/20260417-141123-fceratto.json |
[production] |
| 14:10 |
<dpogorzelski@deploy1003> |
helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. |
[production] |
| 14:09 |
<urandom> |
decommissioning Cassandra, aqs1011 [a,b] — T412830 |
[production] |
| 14:06 |
<fabfur@cumin1003> |
END (PASS) - Cookbook sre.cdn.roll-restart-varnish (exit_code=0) rolling restart of Varnish on 1 hosts matching query P{cp3073.*} |
[production] |
| 14:06 |
<eevans@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 30 days, 0:00:00 on aqs1011.eqiad.wmnet with reason: Bootstrapping — T412830 |
[production] |
| 14:05 |
<fabfur@cumin1003> |
START - Cookbook sre.cdn.roll-restart-varnish rolling restart of Varnish on 1 hosts matching query P{cp3073.*} |
[production] |
| 14:04 |
<fceratto@cumin1003> |
dbctl commit (dc=all): 'Depooling db1233 (T419961)', diff saved to https://phabricator.wikimedia.org/P91071 and previous config saved to /var/cache/conftool/dbconfig/20260417-140454-fceratto.json |
[production] |
| 14:04 |
<fceratto@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1233.eqiad.wmnet with reason: Maintenance |
[production] |
| 14:04 |
<fabfur@cumin1003> |
END (PASS) - Cookbook sre.cdn.roll-restart-varnish (exit_code=0) rolling restart of Varnish on 1 hosts matching query P{cp3072.*} |
[production] |
| 14:04 |
<fceratto@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1229 (T419961)', diff saved to https://phabricator.wikimedia.org/P91070 and previous config saved to /var/cache/conftool/dbconfig/20260417-140424-fceratto.json |
[production] |
| 14:04 |
<dpogorzelski@deploy1003> |
helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. |
[production] |
| 14:03 |
<dpogorzelski@deploy1003> |
helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. |
[production] |
| 14:03 |
<fabfur@cumin1003> |
START - Cookbook sre.cdn.roll-restart-varnish rolling restart of Varnish on 1 hosts matching query P{cp3072.*} |
[production] |
| 14:02 |
<fabfur@cumin1003> |
END (PASS) - Cookbook sre.cdn.roll-restart-varnish (exit_code=0) rolling restart of Varnish on 1 hosts matching query P{cp3070.*} |
[production] |
| 14:01 |
<dpogorzelski@deploy1003> |
helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. |
[production] |
| 14:01 |
<fceratto@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2168 (T419635)', diff saved to https://phabricator.wikimedia.org/P91069 and previous config saved to /var/cache/conftool/dbconfig/20260417-140115-fceratto.json |
[production] |
| 14:01 |
<fabfur@cumin1003> |
START - Cookbook sre.cdn.roll-restart-varnish rolling restart of Varnish on 1 hosts matching query P{cp3070.*} |
[production] |
| 14:00 |
<fabfur@cumin1003> |
END (PASS) - Cookbook sre.cdn.roll-restart-varnish (exit_code=0) rolling restart of Varnish on 1 hosts matching query P{cp3069.*} |
[production] |
| 14:00 |
<fabfur> |
restart varnish on cp3069, cp3070, cp3072, cp3073 to clear alerts |
[production] |
| 14:00 |
<fceratto@cumin1003> |
dbctl commit (dc=all): 'Depooling db2168 (T419635)', diff saved to https://phabricator.wikimedia.org/P91068 and previous config saved to /var/cache/conftool/dbconfig/20260417-140003-fceratto.json |
[production] |
| 13:59 |
<fceratto@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2168.codfw.wmnet with reason: Maintenance |
[production] |
| 13:59 |
<dpogorzelski@deploy1003> |
helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. |
[production] |
| 13:59 |
<fceratto@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2159 (T419635)', diff saved to https://phabricator.wikimedia.org/P91067 and previous config saved to /var/cache/conftool/dbconfig/20260417-135938-fceratto.json |
[production] |
| 13:58 |
<fabfur@cumin1003> |
START - Cookbook sre.cdn.roll-restart-varnish rolling restart of Varnish on 1 hosts matching query P{cp3069.*} |
[production] |
| 13:57 |
<fabfur@cumin1003> |
END (PASS) - Cookbook sre.cdn.roll-restart-varnish (exit_code=0) rolling restart of Varnish on 1 hosts matching query P{cp3066.*} |
[production] |