|
2026-01-08
ยง
|
| 19:07 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1207.eqiad.wmnet with reason: Maintenance |
[production] |
| 19:07 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1200 (T413525)', diff saved to https://phabricator.wikimedia.org/P86883 and previous config saved to /var/cache/conftool/dbconfig/20260108-190702-marostegui.json |
[production] |
| 18:57 |
<eileen> |
civicrm upgraded from b454e4ba to 2689332e |
[production] |
| 18:56 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P86882 and previous config saved to /var/cache/conftool/dbconfig/20260108-185654-marostegui.json |
[production] |
| 18:46 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P86881 and previous config saved to /var/cache/conftool/dbconfig/20260108-184645-marostegui.json |
[production] |
| 18:36 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1200 (T413525)', diff saved to https://phabricator.wikimedia.org/P86880 and previous config saved to /var/cache/conftool/dbconfig/20260108-183637-marostegui.json |
[production] |
| 18:26 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db1200 (T413525)', diff saved to https://phabricator.wikimedia.org/P86879 and previous config saved to /var/cache/conftool/dbconfig/20260108-182641-marostegui.json |
[production] |
| 18:26 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1200.eqiad.wmnet with reason: Maintenance |
[production] |
| 18:26 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1185 (T413525)', diff saved to https://phabricator.wikimedia.org/P86878 and previous config saved to /var/cache/conftool/dbconfig/20260108-182627-marostegui.json |
[production] |
| 18:16 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P86877 and previous config saved to /var/cache/conftool/dbconfig/20260108-181619-marostegui.json |
[production] |
| 18:15 |
<bd808@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/developer-portal: apply |
[production] |
| 18:14 |
<bd808@deploy2002> |
helmfile [codfw] START helmfile.d/services/developer-portal: apply |
[production] |
| 18:12 |
<bd808@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply |
[production] |
| 18:12 |
<bd808@deploy2002> |
helmfile [eqiad] START helmfile.d/services/developer-portal: apply |
[production] |
| 18:11 |
<bd808@deploy2002> |
helmfile [staging] DONE helmfile.d/services/developer-portal: apply |
[production] |
| 18:11 |
<bd808@deploy2002> |
helmfile [staging] START helmfile.d/services/developer-portal: apply |
[production] |
| 18:06 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P86876 and previous config saved to /var/cache/conftool/dbconfig/20260108-180611-marostegui.json |
[production] |
| 17:56 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1185 (T413525)', diff saved to https://phabricator.wikimedia.org/P86875 and previous config saved to /var/cache/conftool/dbconfig/20260108-175602-marostegui.json |
[production] |
| 17:50 |
<ladsgroup@deploy2002> |
Finished scap sync-world: Backport for [[gerrit:1224071|Only generate 120,250 thumbs (temporary) (T408062 T412971)]] (duration: 17m 34s) |
[production] |
| 17:46 |
<ladsgroup@deploy2002> |
mvernon, ladsgroup: Continuing with sync |
[production] |
| 17:46 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db1185 (T413525)', diff saved to https://phabricator.wikimedia.org/P86874 and previous config saved to /var/cache/conftool/dbconfig/20260108-174606-marostegui.json |
[production] |
| 17:45 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1185.eqiad.wmnet with reason: Maintenance |
[production] |
| 17:45 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1161 (T413525)', diff saved to https://phabricator.wikimedia.org/P86873 and previous config saved to /var/cache/conftool/dbconfig/20260108-174542-marostegui.json |
[production] |
| 17:35 |
<ladsgroup@deploy2002> |
mvernon, ladsgroup: Backport for [[gerrit:1224071|Only generate 120,250 thumbs (temporary) (T408062 T412971)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. |
[production] |
| 17:35 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P86872 and previous config saved to /var/cache/conftool/dbconfig/20260108-173534-marostegui.json |
[production] |
| 17:32 |
<ladsgroup@deploy2002> |
Started scap sync-world: Backport for [[gerrit:1224071|Only generate 120,250 thumbs (temporary) (T408062 T412971)]] |
[production] |
| 17:32 |
<andrew@cumin2002> |
START - Cookbook sre.hosts.reimage for host cloudbackup2003.codfw.wmnet with OS trixie |
[production] |
| 17:31 |
<andrew@cumin2002> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudbackup2003.codfw.wmnet with OS trixie |
[production] |
| 17:31 |
<jynus> |
restarted wmf_auto_restart_prometheus-mysqld-exporter.service @ db2231 |
[production] |
| 17:25 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db2155 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P86871 and previous config saved to /var/cache/conftool/dbconfig/20260108-172551-marostegui.json |
[production] |
| 17:25 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2155.codfw.wmnet with reason: Maintenance |
[production] |
| 17:25 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2147 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P86870 and previous config saved to /var/cache/conftool/dbconfig/20260108-172526-marostegui.json |
[production] |
| 17:25 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P86869 and previous config saved to /var/cache/conftool/dbconfig/20260108-172526-marostegui.json |
[production] |
| 17:21 |
<oblivian@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/thumbor: sync |
[production] |
| 17:21 |
<oblivian@deploy2002> |
helmfile [eqiad] START helmfile.d/services/thumbor: sync |
[production] |
| 17:15 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P86867 and previous config saved to /var/cache/conftool/dbconfig/20260108-171517-marostegui.json |
[production] |
| 17:15 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1161 (T413525)', diff saved to https://phabricator.wikimedia.org/P86868 and previous config saved to /var/cache/conftool/dbconfig/20260108-171517-marostegui.json |
[production] |
| 17:05 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P86866 and previous config saved to /var/cache/conftool/dbconfig/20260108-170509-marostegui.json |
[production] |
| 17:02 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db1161 (T413525)', diff saved to https://phabricator.wikimedia.org/P86865 and previous config saved to /var/cache/conftool/dbconfig/20260108-170241-marostegui.json |
[production] |
| 17:02 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance |
[production] |
| 17:02 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance |
[production] |
| 17:01 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1159 (T413525)', diff saved to https://phabricator.wikimedia.org/P86864 and previous config saved to /var/cache/conftool/dbconfig/20260108-170156-marostegui.json |
[production] |
| 16:55 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2147 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P86863 and previous config saved to /var/cache/conftool/dbconfig/20260108-165501-marostegui.json |
[production] |
| 16:51 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P86862 and previous config saved to /var/cache/conftool/dbconfig/20260108-165148-marostegui.json |
[production] |
| 16:41 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P86861 and previous config saved to /var/cache/conftool/dbconfig/20260108-164140-marostegui.json |
[production] |
| 16:31 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1159 (T413525)', diff saved to https://phabricator.wikimedia.org/P86860 and previous config saved to /var/cache/conftool/dbconfig/20260108-163131-marostegui.json |
[production] |
| 16:18 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db1159 (T413525)', diff saved to https://phabricator.wikimedia.org/P86858 and previous config saved to /var/cache/conftool/dbconfig/20260108-161848-marostegui.json |
[production] |
| 16:18 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1159.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:14 |
<elukey@cumin1003> |
END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wdqs1029.eqiad.wmnet with OS trixie |
[production] |
| 16:10 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2229 (T413525)', diff saved to https://phabricator.wikimedia.org/P86857 and previous config saved to /var/cache/conftool/dbconfig/20260108-161038-marostegui.json |
[production] |