4401-4450 of 10000 results (89ms)
2023-09-11 §
08:05 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3007.esams.wmnet [production]
08:05 <kharlan@deploy1002> kharlan: Backport for [[gerrit:955732|[beta] ReportIncident: Enable on kowiki beta (T339275)]], [[gerrit:955735|[beta] Enable ReportIncident for configured beta wikis (T339275)]], [[gerrit:955821|ReportIncident: Set default help page (T343382)]] synced to the testservers mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, and mw-debug kubernetes deplo [production]
08:03 <filippo@deploy1002> helmfile [codfw] DONE helmfile.d/services/citoid: apply [production]
08:03 <kharlan@deploy1002> Started scap: Backport for [[gerrit:955732|[beta] ReportIncident: Enable on kowiki beta (T339275)]], [[gerrit:955735|[beta] Enable ReportIncident for configured beta wikis (T339275)]], [[gerrit:955821|ReportIncident: Set default help page (T343382)]] [production]
08:03 <filippo@deploy1002> helmfile [codfw] START helmfile.d/services/citoid: apply [production]
08:02 <filippo@deploy1002> helmfile [eqiad] DONE helmfile.d/services/citoid: apply [production]
08:02 <filippo@deploy1002> helmfile [eqiad] START helmfile.d/services/citoid: apply [production]
08:02 <filippo@deploy1002> helmfile [staging] DONE helmfile.d/services/citoid: apply [production]
08:01 <filippo@deploy1002> helmfile [staging] START helmfile.d/services/citoid: apply [production]
08:01 <marostegui@cumin1001> dbctl commit (dc=all): 'db1128 (re)pooling @ 25%: Repooling after being recloned T345509', diff saved to https://phabricator.wikimedia.org/P52390 and previous config saved to /var/cache/conftool/dbconfig/20230911-080133-root.json [production]
08:00 <filippo@deploy1002> helmfile [eqiad] DONE helmfile.d/services/citoid: apply [production]
08:00 <filippo@deploy1002> helmfile [eqiad] START helmfile.d/services/citoid: apply [production]
08:00 <kharlan@deploy1002> Finished scap: Backport for [[gerrit:953999|ReportIncident: Default deployment to false (T339275)]] (duration: 11m 15s) [production]
08:00 <filippo@deploy1002> helmfile [staging] DONE helmfile.d/services/citoid: apply [production]
08:00 <filippo@deploy1002> helmfile [staging] START helmfile.d/services/citoid: apply [production]
07:59 <filippo@deploy1002> helmfile [staging] DONE helmfile.d/services/citoid: apply [production]
07:59 <filippo@deploy1002> helmfile [staging] START helmfile.d/services/citoid: apply [production]
07:56 <marostegui@cumin1001> dbctl commit (dc=all): 'db1134 (re)pooling @ 3%: Repooling after cloning another host', diff saved to https://phabricator.wikimedia.org/P52389 and previous config saved to /var/cache/conftool/dbconfig/20230911-075621-root.json [production]
07:53 <kharlan@deploy1002> kharlan: Continuing with sync [production]
07:50 <kharlan@deploy1002> kharlan: Backport for [[gerrit:953999|ReportIncident: Default deployment to false (T339275)]] synced to the testservers mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) [production]
07:49 <kharlan@deploy1002> Started scap: Backport for [[gerrit:953999|ReportIncident: Default deployment to false (T339275)]] [production]
07:46 <kharlan@deploy1002> Finished scap: Backport for [[gerrit:953998|Add ReportIncident extension (T339275)]] (duration: 22m 44s) [production]
07:46 <marostegui@cumin1001> dbctl commit (dc=all): 'db1128 (re)pooling @ 10%: Repooling after being recloned T345509', diff saved to https://phabricator.wikimedia.org/P52388 and previous config saved to /var/cache/conftool/dbconfig/20230911-074629-root.json [production]
07:41 <marostegui@cumin1001> dbctl commit (dc=all): 'db1134 (re)pooling @ 1%: Repooling after cloning another host', diff saved to https://phabricator.wikimedia.org/P52387 and previous config saved to /var/cache/conftool/dbconfig/20230911-074116-root.json [production]
07:36 <kharlan@deploy1002> kharlan: Continuing with sync [production]
07:35 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3005.esams.wmnet [production]
07:35 <elukey@deploy1002> helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . [production]
07:33 <kharlan@deploy1002> kharlan: Backport for [[gerrit:953998|Add ReportIncident extension (T339275)]] synced to the testservers mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) [production]
07:32 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3005.esams.wmnet [production]
07:31 <marostegui@cumin1001> dbctl commit (dc=all): 'db1128 (re)pooling @ 5%: Repooling after being recloned T345509', diff saved to https://phabricator.wikimedia.org/P52386 and previous config saved to /var/cache/conftool/dbconfig/20230911-073124-root.json [production]
07:23 <kharlan@deploy1002> Started scap: Backport for [[gerrit:953998|Add ReportIncident extension (T339275)]] [production]
07:16 <marostegui@cumin1001> dbctl commit (dc=all): 'db1128 (re)pooling @ 3%: Repooling after being recloned T345509', diff saved to https://phabricator.wikimedia.org/P52385 and previous config saved to /var/cache/conftool/dbconfig/20230911-071619-root.json [production]
07:11 <kharlan@deploy1002> Started scap: Backport for [[gerrit:953998|Add ReportIncident extension (T339275)]] [production]
07:01 <marostegui@cumin1001> dbctl commit (dc=all): 'db1128 (re)pooling @ 1%: Repooling after being recloned T345509', diff saved to https://phabricator.wikimedia.org/P52384 and previous config saved to /var/cache/conftool/dbconfig/20230911-070114-root.json [production]
06:57 <marostegui@cumin1001> END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1134.eqiad.wmnet onto db1128.eqiad.wmnet [production]
06:12 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 136065 [production]
06:11 <ayounsi@cumin1001> START - Cookbook sre.network.peering with action 'configure' for AS: 136065 [production]
05:40 <marostegui@cumin1001> dbctl commit (dc=all): 'Add db1119 back to s1 depooled T339185', diff saved to https://phabricator.wikimedia.org/P52383 and previous config saved to /var/cache/conftool/dbconfig/20230911-054057-marostegui.json [production]
05:00 <marostegui@cumin1001> START - Cookbook sre.mysql.clone of db1134.eqiad.wmnet onto db1128.eqiad.wmnet [production]
04:59 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1134', diff saved to https://phabricator.wikimedia.org/P52382 and previous config saved to /var/cache/conftool/dbconfig/20230911-045907-root.json [production]
01:29 <arnaudb@cumin1001> dbctl commit (dc=all): 'Depooling db2127 (T343198)', diff saved to https://phabricator.wikimedia.org/P52381 and previous config saved to /var/cache/conftool/dbconfig/20230911-012911-arnaudb.json [production]
01:29 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2127.codfw.wmnet with reason: Maintenance [production]
01:28 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2127.codfw.wmnet with reason: Maintenance [production]
01:28 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2109 (T343198)', diff saved to https://phabricator.wikimedia.org/P52380 and previous config saved to /var/cache/conftool/dbconfig/20230911-012850-arnaudb.json [production]
01:13 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P52379 and previous config saved to /var/cache/conftool/dbconfig/20230911-011343-arnaudb.json [production]
00:58 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P52378 and previous config saved to /var/cache/conftool/dbconfig/20230911-005837-arnaudb.json [production]
00:43 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2109 (T343198)', diff saved to https://phabricator.wikimedia.org/P52377 and previous config saved to /var/cache/conftool/dbconfig/20230911-004331-arnaudb.json [production]
2023-09-10 §
17:35 <arnaudb@cumin1001> dbctl commit (dc=all): 'Depooling db2109 (T343198)', diff saved to https://phabricator.wikimedia.org/P52375 and previous config saved to /var/cache/conftool/dbconfig/20230910-173502-arnaudb.json [production]
17:34 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2109.codfw.wmnet with reason: Maintenance [production]
17:34 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2109.codfw.wmnet with reason: Maintenance [production]