|
2023-09-11
§
|
| 08:02 |
<filippo@deploy1002> |
helmfile [staging] DONE helmfile.d/services/citoid: apply |
[production] |
| 08:01 |
<filippo@deploy1002> |
helmfile [staging] START helmfile.d/services/citoid: apply |
[production] |
| 08:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1128 (re)pooling @ 25%: Repooling after being recloned T345509', diff saved to https://phabricator.wikimedia.org/P52390 and previous config saved to /var/cache/conftool/dbconfig/20230911-080133-root.json |
[production] |
| 08:00 |
<filippo@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/citoid: apply |
[production] |
| 08:00 |
<filippo@deploy1002> |
helmfile [eqiad] START helmfile.d/services/citoid: apply |
[production] |
| 08:00 |
<kharlan@deploy1002> |
Finished scap: Backport for [[gerrit:953999|ReportIncident: Default deployment to false (T339275)]] (duration: 11m 15s) |
[production] |
| 08:00 |
<filippo@deploy1002> |
helmfile [staging] DONE helmfile.d/services/citoid: apply |
[production] |
| 08:00 |
<filippo@deploy1002> |
helmfile [staging] START helmfile.d/services/citoid: apply |
[production] |
| 07:59 |
<filippo@deploy1002> |
helmfile [staging] DONE helmfile.d/services/citoid: apply |
[production] |
| 07:59 |
<filippo@deploy1002> |
helmfile [staging] START helmfile.d/services/citoid: apply |
[production] |
| 07:56 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1134 (re)pooling @ 3%: Repooling after cloning another host', diff saved to https://phabricator.wikimedia.org/P52389 and previous config saved to /var/cache/conftool/dbconfig/20230911-075621-root.json |
[production] |
| 07:53 |
<kharlan@deploy1002> |
kharlan: Continuing with sync |
[production] |
| 07:50 |
<kharlan@deploy1002> |
kharlan: Backport for [[gerrit:953999|ReportIncident: Default deployment to false (T339275)]] synced to the testservers mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) |
[production] |
| 07:49 |
<kharlan@deploy1002> |
Started scap: Backport for [[gerrit:953999|ReportIncident: Default deployment to false (T339275)]] |
[production] |
| 07:46 |
<kharlan@deploy1002> |
Finished scap: Backport for [[gerrit:953998|Add ReportIncident extension (T339275)]] (duration: 22m 44s) |
[production] |
| 07:46 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1128 (re)pooling @ 10%: Repooling after being recloned T345509', diff saved to https://phabricator.wikimedia.org/P52388 and previous config saved to /var/cache/conftool/dbconfig/20230911-074629-root.json |
[production] |
| 07:41 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1134 (re)pooling @ 1%: Repooling after cloning another host', diff saved to https://phabricator.wikimedia.org/P52387 and previous config saved to /var/cache/conftool/dbconfig/20230911-074116-root.json |
[production] |
| 07:36 |
<kharlan@deploy1002> |
kharlan: Continuing with sync |
[production] |
| 07:35 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3005.esams.wmnet |
[production] |
| 07:35 |
<elukey@deploy1002> |
helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . |
[production] |
| 07:33 |
<kharlan@deploy1002> |
kharlan: Backport for [[gerrit:953998|Add ReportIncident extension (T339275)]] synced to the testservers mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) |
[production] |
| 07:32 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3005.esams.wmnet |
[production] |
| 07:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1128 (re)pooling @ 5%: Repooling after being recloned T345509', diff saved to https://phabricator.wikimedia.org/P52386 and previous config saved to /var/cache/conftool/dbconfig/20230911-073124-root.json |
[production] |
| 07:23 |
<kharlan@deploy1002> |
Started scap: Backport for [[gerrit:953998|Add ReportIncident extension (T339275)]] |
[production] |
| 07:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1128 (re)pooling @ 3%: Repooling after being recloned T345509', diff saved to https://phabricator.wikimedia.org/P52385 and previous config saved to /var/cache/conftool/dbconfig/20230911-071619-root.json |
[production] |
| 07:11 |
<kharlan@deploy1002> |
Started scap: Backport for [[gerrit:953998|Add ReportIncident extension (T339275)]] |
[production] |
| 07:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1128 (re)pooling @ 1%: Repooling after being recloned T345509', diff saved to https://phabricator.wikimedia.org/P52384 and previous config saved to /var/cache/conftool/dbconfig/20230911-070114-root.json |
[production] |
| 06:57 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1134.eqiad.wmnet onto db1128.eqiad.wmnet |
[production] |
| 06:12 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 136065 |
[production] |
| 06:11 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.peering with action 'configure' for AS: 136065 |
[production] |
| 05:40 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Add db1119 back to s1 depooled T339185', diff saved to https://phabricator.wikimedia.org/P52383 and previous config saved to /var/cache/conftool/dbconfig/20230911-054057-marostegui.json |
[production] |
| 05:00 |
<marostegui@cumin1001> |
START - Cookbook sre.mysql.clone of db1134.eqiad.wmnet onto db1128.eqiad.wmnet |
[production] |
| 04:59 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1134', diff saved to https://phabricator.wikimedia.org/P52382 and previous config saved to /var/cache/conftool/dbconfig/20230911-045907-root.json |
[production] |
| 01:29 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'Depooling db2127 (T343198)', diff saved to https://phabricator.wikimedia.org/P52381 and previous config saved to /var/cache/conftool/dbconfig/20230911-012911-arnaudb.json |
[production] |
| 01:29 |
<arnaudb@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2127.codfw.wmnet with reason: Maintenance |
[production] |
| 01:28 |
<arnaudb@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2127.codfw.wmnet with reason: Maintenance |
[production] |
| 01:28 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2109 (T343198)', diff saved to https://phabricator.wikimedia.org/P52380 and previous config saved to /var/cache/conftool/dbconfig/20230911-012850-arnaudb.json |
[production] |
| 01:13 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P52379 and previous config saved to /var/cache/conftool/dbconfig/20230911-011343-arnaudb.json |
[production] |
| 00:58 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P52378 and previous config saved to /var/cache/conftool/dbconfig/20230911-005837-arnaudb.json |
[production] |
| 00:43 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2109 (T343198)', diff saved to https://phabricator.wikimedia.org/P52377 and previous config saved to /var/cache/conftool/dbconfig/20230911-004331-arnaudb.json |
[production] |
|
2023-09-10
§
|
| 17:35 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'Depooling db2109 (T343198)', diff saved to https://phabricator.wikimedia.org/P52375 and previous config saved to /var/cache/conftool/dbconfig/20230910-173502-arnaudb.json |
[production] |
| 17:34 |
<arnaudb@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2109.codfw.wmnet with reason: Maintenance |
[production] |
| 17:34 |
<arnaudb@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2109.codfw.wmnet with reason: Maintenance |
[production] |
| 11:19 |
<arnaudb@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance |
[production] |
| 11:19 |
<arnaudb@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance |
[production] |
| 11:19 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1223 (T343198)', diff saved to https://phabricator.wikimedia.org/P52374 and previous config saved to /var/cache/conftool/dbconfig/20230910-111941-arnaudb.json |
[production] |
| 11:04 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P52373 and previous config saved to /var/cache/conftool/dbconfig/20230910-110435-arnaudb.json |
[production] |
| 10:49 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P52372 and previous config saved to /var/cache/conftool/dbconfig/20230910-104929-arnaudb.json |
[production] |
| 10:34 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1223 (T343198)', diff saved to https://phabricator.wikimedia.org/P52371 and previous config saved to /var/cache/conftool/dbconfig/20230910-103422-arnaudb.json |
[production] |
| 04:23 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'Depooling db1223 (T343198)', diff saved to https://phabricator.wikimedia.org/P52370 and previous config saved to /var/cache/conftool/dbconfig/20230910-042338-arnaudb.json |
[production] |