|
2023-09-11
§
|
| 08:03 |
<kharlan@deploy1002> |
Started scap: Backport for [[gerrit:955732|[beta] ReportIncident: Enable on kowiki beta (T339275)]], [[gerrit:955735|[beta] Enable ReportIncident for configured beta wikis (T339275)]], [[gerrit:955821|ReportIncident: Set default help page (T343382)]] |
[production] |
| 08:03 |
<filippo@deploy1002> |
helmfile [codfw] START helmfile.d/services/citoid: apply |
[production] |
| 08:02 |
<filippo@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/citoid: apply |
[production] |
| 08:02 |
<filippo@deploy1002> |
helmfile [eqiad] START helmfile.d/services/citoid: apply |
[production] |
| 08:02 |
<filippo@deploy1002> |
helmfile [staging] DONE helmfile.d/services/citoid: apply |
[production] |
| 08:01 |
<filippo@deploy1002> |
helmfile [staging] START helmfile.d/services/citoid: apply |
[production] |
| 08:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1128 (re)pooling @ 25%: Repooling after being recloned T345509', diff saved to https://phabricator.wikimedia.org/P52390 and previous config saved to /var/cache/conftool/dbconfig/20230911-080133-root.json |
[production] |
| 08:00 |
<filippo@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/citoid: apply |
[production] |
| 08:00 |
<filippo@deploy1002> |
helmfile [eqiad] START helmfile.d/services/citoid: apply |
[production] |
| 08:00 |
<kharlan@deploy1002> |
Finished scap: Backport for [[gerrit:953999|ReportIncident: Default deployment to false (T339275)]] (duration: 11m 15s) |
[production] |
| 08:00 |
<filippo@deploy1002> |
helmfile [staging] DONE helmfile.d/services/citoid: apply |
[production] |
| 08:00 |
<filippo@deploy1002> |
helmfile [staging] START helmfile.d/services/citoid: apply |
[production] |
| 07:59 |
<filippo@deploy1002> |
helmfile [staging] DONE helmfile.d/services/citoid: apply |
[production] |
| 07:59 |
<filippo@deploy1002> |
helmfile [staging] START helmfile.d/services/citoid: apply |
[production] |
| 07:56 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1134 (re)pooling @ 3%: Repooling after cloning another host', diff saved to https://phabricator.wikimedia.org/P52389 and previous config saved to /var/cache/conftool/dbconfig/20230911-075621-root.json |
[production] |
| 07:53 |
<kharlan@deploy1002> |
kharlan: Continuing with sync |
[production] |
| 07:50 |
<kharlan@deploy1002> |
kharlan: Backport for [[gerrit:953999|ReportIncident: Default deployment to false (T339275)]] synced to the testservers mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) |
[production] |
| 07:49 |
<kharlan@deploy1002> |
Started scap: Backport for [[gerrit:953999|ReportIncident: Default deployment to false (T339275)]] |
[production] |
| 07:46 |
<kharlan@deploy1002> |
Finished scap: Backport for [[gerrit:953998|Add ReportIncident extension (T339275)]] (duration: 22m 44s) |
[production] |
| 07:46 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1128 (re)pooling @ 10%: Repooling after being recloned T345509', diff saved to https://phabricator.wikimedia.org/P52388 and previous config saved to /var/cache/conftool/dbconfig/20230911-074629-root.json |
[production] |
| 07:41 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1134 (re)pooling @ 1%: Repooling after cloning another host', diff saved to https://phabricator.wikimedia.org/P52387 and previous config saved to /var/cache/conftool/dbconfig/20230911-074116-root.json |
[production] |
| 07:36 |
<kharlan@deploy1002> |
kharlan: Continuing with sync |
[production] |
| 07:35 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3005.esams.wmnet |
[production] |
| 07:35 |
<elukey@deploy1002> |
helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . |
[production] |
| 07:33 |
<kharlan@deploy1002> |
kharlan: Backport for [[gerrit:953998|Add ReportIncident extension (T339275)]] synced to the testservers mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) |
[production] |
| 07:32 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3005.esams.wmnet |
[production] |
| 07:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1128 (re)pooling @ 5%: Repooling after being recloned T345509', diff saved to https://phabricator.wikimedia.org/P52386 and previous config saved to /var/cache/conftool/dbconfig/20230911-073124-root.json |
[production] |
| 07:23 |
<kharlan@deploy1002> |
Started scap: Backport for [[gerrit:953998|Add ReportIncident extension (T339275)]] |
[production] |
| 07:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1128 (re)pooling @ 3%: Repooling after being recloned T345509', diff saved to https://phabricator.wikimedia.org/P52385 and previous config saved to /var/cache/conftool/dbconfig/20230911-071619-root.json |
[production] |
| 07:11 |
<kharlan@deploy1002> |
Started scap: Backport for [[gerrit:953998|Add ReportIncident extension (T339275)]] |
[production] |
| 07:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1128 (re)pooling @ 1%: Repooling after being recloned T345509', diff saved to https://phabricator.wikimedia.org/P52384 and previous config saved to /var/cache/conftool/dbconfig/20230911-070114-root.json |
[production] |
| 06:57 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1134.eqiad.wmnet onto db1128.eqiad.wmnet |
[production] |
| 06:12 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 136065 |
[production] |
| 06:11 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.peering with action 'configure' for AS: 136065 |
[production] |
| 05:40 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Add db1119 back to s1 depooled T339185', diff saved to https://phabricator.wikimedia.org/P52383 and previous config saved to /var/cache/conftool/dbconfig/20230911-054057-marostegui.json |
[production] |
| 05:00 |
<marostegui@cumin1001> |
START - Cookbook sre.mysql.clone of db1134.eqiad.wmnet onto db1128.eqiad.wmnet |
[production] |
| 04:59 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1134', diff saved to https://phabricator.wikimedia.org/P52382 and previous config saved to /var/cache/conftool/dbconfig/20230911-045907-root.json |
[production] |
| 01:29 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'Depooling db2127 (T343198)', diff saved to https://phabricator.wikimedia.org/P52381 and previous config saved to /var/cache/conftool/dbconfig/20230911-012911-arnaudb.json |
[production] |
| 01:29 |
<arnaudb@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2127.codfw.wmnet with reason: Maintenance |
[production] |
| 01:28 |
<arnaudb@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2127.codfw.wmnet with reason: Maintenance |
[production] |
| 01:28 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2109 (T343198)', diff saved to https://phabricator.wikimedia.org/P52380 and previous config saved to /var/cache/conftool/dbconfig/20230911-012850-arnaudb.json |
[production] |
| 01:13 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P52379 and previous config saved to /var/cache/conftool/dbconfig/20230911-011343-arnaudb.json |
[production] |
| 00:58 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P52378 and previous config saved to /var/cache/conftool/dbconfig/20230911-005837-arnaudb.json |
[production] |
| 00:43 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2109 (T343198)', diff saved to https://phabricator.wikimedia.org/P52377 and previous config saved to /var/cache/conftool/dbconfig/20230911-004331-arnaudb.json |
[production] |