|
2023-01-11
ยง
|
| 19:19 |
<otto@deploy1002> |
helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 19:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1206 (re)pooling @ 5%: After being recloned', diff saved to https://phabricator.wikimedia.org/P43009 and previous config saved to /var/cache/conftool/dbconfig/20230111-191616-root.json |
[production] |
| 19:04 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2130', diff saved to https://phabricator.wikimedia.org/P43008 and previous config saved to /var/cache/conftool/dbconfig/20230111-190453-marostegui.json |
[production] |
| 19:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1206 (re)pooling @ 1%: After being recloned', diff saved to https://phabricator.wikimedia.org/P43007 and previous config saved to /var/cache/conftool/dbconfig/20230111-190111-root.json |
[production] |
| 18:57 |
<marostegui> |
dbmaint deploy schema change with replication on s3 eqiad T321391 |
[production] |
| 18:52 |
<brett> |
Removing legacy vips from dns servers - T239993 |
[production] |
| 18:49 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2130 (T321391)', diff saved to https://phabricator.wikimedia.org/P43006 and previous config saved to /var/cache/conftool/dbconfig/20230111-184946-marostegui.json |
[production] |
| 18:47 |
<marostegui> |
dbmaint deploy schema change with replication on s2 eqiad T321391 |
[production] |
| 18:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db2130 (T321391)', diff saved to https://phabricator.wikimedia.org/P43005 and previous config saved to /var/cache/conftool/dbconfig/20230111-184723-marostegui.json |
[production] |
| 18:47 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2130.codfw.wmnet with reason: Maintenance |
[production] |
| 18:47 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db2130.codfw.wmnet with reason: Maintenance |
[production] |
| 18:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2116 (T321391)', diff saved to https://phabricator.wikimedia.org/P43004 and previous config saved to /var/cache/conftool/dbconfig/20230111-184701-marostegui.json |
[production] |
| 18:40 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1106 (re)pooling @ 100%: After cloning db1206', diff saved to https://phabricator.wikimedia.org/P43003 and previous config saved to /var/cache/conftool/dbconfig/20230111-184051-root.json |
[production] |
| 18:36 |
<ebernhardson@deploy1002> |
Finished deploy [wikimedia/discovery/analytics@5a19b9d]: drop-snapshots: Accept snapshot= partition from any level (duration: 02m 33s) |
[production] |
| 18:33 |
<ebernhardson@deploy1002> |
Started deploy [wikimedia/discovery/analytics@5a19b9d]: drop-snapshots: Accept snapshot= partition from any level |
[production] |
| 18:33 |
<jayme@deploy1002> |
helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 18:32 |
<jayme@deploy1002> |
helmfile [staging-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 18:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2116', diff saved to https://phabricator.wikimedia.org/P43002 and previous config saved to /var/cache/conftool/dbconfig/20230111-183155-marostegui.json |
[production] |
| 18:30 |
<jayme@deploy1002> |
helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 18:30 |
<jayme@deploy1002> |
helmfile [staging-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 18:28 |
<bblack> |
repool eqsin edge DC |
[production] |
| 18:25 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1106 (re)pooling @ 75%: After cloning db1206', diff saved to https://phabricator.wikimedia.org/P43001 and previous config saved to /var/cache/conftool/dbconfig/20230111-182546-root.json |
[production] |
| 18:22 |
<btullis@cumin1001> |
END (PASS) - Cookbook sre.wikireplicas.add-wiki (exit_code=0) |
[production] |
| 18:22 |
<btullis@cumin1001> |
Added views for new wiki: blkwiki T310872 |
[production] |
| 18:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2116', diff saved to https://phabricator.wikimedia.org/P43000 and previous config saved to /var/cache/conftool/dbconfig/20230111-181648-marostegui.json |
[production] |
| 18:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1106 (re)pooling @ 50%: After cloning db1206', diff saved to https://phabricator.wikimedia.org/P42999 and previous config saved to /var/cache/conftool/dbconfig/20230111-181041-root.json |
[production] |
| 18:09 |
<hnowlan@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/thumbor: sync |
[production] |
| 18:09 |
<jayme@deploy1002> |
helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 18:08 |
<jayme@deploy1002> |
helmfile [staging-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 18:08 |
<jayme@deploy1002> |
helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 18:07 |
<jayme@deploy1002> |
helmfile [staging-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 18:02 |
<jayme@deploy1002> |
helmfile [staging-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 18:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2116 (T321391)', diff saved to https://phabricator.wikimedia.org/P42998 and previous config saved to /var/cache/conftool/dbconfig/20230111-180142-marostegui.json |
[production] |
| 18:01 |
<hnowlan@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/thumbor: apply |
[production] |
| 17:59 |
<hnowlan@deploy1002> |
helmfile [eqiad] START helmfile.d/services/thumbor: sync |
[production] |
| 17:59 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db2116 (T321391)', diff saved to https://phabricator.wikimedia.org/P42997 and previous config saved to /var/cache/conftool/dbconfig/20230111-175919-marostegui.json |
[production] |
| 17:59 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2116.codfw.wmnet with reason: Maintenance |
[production] |
| 17:59 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db2116.codfw.wmnet with reason: Maintenance |
[production] |
| 17:58 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2112 (T321391)', diff saved to https://phabricator.wikimedia.org/P42996 and previous config saved to /var/cache/conftool/dbconfig/20230111-175857-marostegui.json |
[production] |
| 17:58 |
<btullis@cumin1001> |
START - Cookbook sre.wikireplicas.add-wiki |
[production] |
| 17:56 |
<btullis@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on an-worker[1080,1084].eqiad.wmnet with reason: Shutting down to enable RAID battery replacement |
[production] |
| 17:55 |
<btullis@cumin1001> |
START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on an-worker[1080,1084].eqiad.wmnet with reason: Shutting down to enable RAID battery replacement |
[production] |
| 17:55 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1106 (re)pooling @ 25%: After cloning db1206', diff saved to https://phabricator.wikimedia.org/P42995 and previous config saved to /var/cache/conftool/dbconfig/20230111-175536-root.json |
[production] |
| 17:50 |
<hnowlan@deploy1002> |
helmfile [codfw] START helmfile.d/services/thumbor: apply |
[production] |
| 17:50 |
<hnowlan@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/thumbor: apply |
[production] |
| 17:43 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2112', diff saved to https://phabricator.wikimedia.org/P42994 and previous config saved to /var/cache/conftool/dbconfig/20230111-174351-marostegui.json |
[production] |
| 17:40 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1106 (re)pooling @ 10%: After cloning db1206', diff saved to https://phabricator.wikimedia.org/P42993 and previous config saved to /var/cache/conftool/dbconfig/20230111-174031-root.json |
[production] |
| 17:40 |
<hnowlan@deploy1002> |
helmfile [codfw] START helmfile.d/services/thumbor: apply |
[production] |
| 17:39 |
<hnowlan@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/thumbor: apply |
[production] |
| 17:29 |
<hnowlan@deploy1002> |
helmfile [eqiad] START helmfile.d/services/thumbor: apply |
[production] |