1201-1250 of 10000 results (94ms)
2023-12-12 ยง
14:31 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1211.eqiad.wmnet with reason: provisionning db1226.eqiad.wmnet - T344036 [production]
14:30 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1211.eqiad.wmnet with reason: provisionning db1226.eqiad.wmnet - T344036 [production]
14:24 <phuedx@deploy2002> phuedx and dani: Backport for [[gerrit:982178|Partially undeploy Reader Demographics 2 survey (T344393)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
14:22 <phuedx@deploy2002> Started scap: Backport for [[gerrit:982178|Partially undeploy Reader Demographics 2 survey (T344393)]] [production]
13:56 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1002.eqiad.wmnet [production]
13:50 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host sretest1002.eqiad.wmnet [production]
13:46 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
13:46 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
13:45 <brouberol> increasing max container memory requests in dse-k8s from 3GB to 8GB - T351722 [production]
13:20 <arnaudb@cumin1001> END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1128.eqiad.wmnet onto db1228.eqiad.wmnet [production]
13:19 <arnaudb@cumin1001> END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1129.eqiad.wmnet onto db1229.eqiad.wmnet [production]
13:16 <jayme@cumin2002> END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM kubemaster2002.codfw.wmnet [production]
13:16 <jayme@cumin2002> END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM kubemaster1002.eqiad.wmnet [production]
13:09 <jayme@cumin2002> START - Cookbook sre.ganeti.reboot-vm for VM kubemaster1002.eqiad.wmnet [production]
13:09 <jayme@cumin2002> START - Cookbook sre.ganeti.reboot-vm for VM kubemaster2002.codfw.wmnet [production]
13:06 <arnaudb@cumin1001> END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1147.eqiad.wmnet onto db1247.eqiad.wmnet [production]
13:00 <jayme@cumin2002> END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM kubemaster1001.eqiad.wmnet [production]
12:57 <jayme@cumin2002> END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM kubemaster2001.codfw.wmnet [production]
12:56 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host backup1011.eqiad.wmnet [production]
12:55 <jayme@cumin2002> START - Cookbook sre.ganeti.reboot-vm for VM kubemaster1001.eqiad.wmnet [production]
12:53 <kharlan@deploy2002> helmfile [eqiad] DONE helmfile.d/services/ipoid: apply [production]
12:52 <kharlan@deploy2002> helmfile [eqiad] START helmfile.d/services/ipoid: apply [production]
12:51 <jayme@cumin2002> START - Cookbook sre.ganeti.reboot-vm for VM kubemaster2001.codfw.wmnet [production]
12:46 <jmm@cumin2002> START - Cookbook sre.puppet.migrate-host for host backup1011.eqiad.wmnet [production]
12:45 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host backup1010.eqiad.wmnet [production]
12:45 <jayme> increasing memory of ganeti instance kubemaster2001.codfw.wmnet from 4G to 12G (requires reboot) - T353233 [production]
12:38 <claime> Uncordoning kubernetes10[59-62].eqiad.wmnet - T353135 [production]
12:37 <claime> Pooling kubernetes10[59-62].eqiad.wmnet - T353135 [production]
12:33 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host backup2011.codfw.wmnet [production]
12:24 <jmm@cumin2002> START - Cookbook sre.puppet.migrate-host for host backup2011.codfw.wmnet [production]
12:15 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host backup2010.codfw.wmnet [production]
12:03 <jmm@cumin2002> START - Cookbook sre.puppet.migrate-host for host backup2010.codfw.wmnet [production]
11:43 <kharlan@deploy2002> helmfile [eqiad] DONE helmfile.d/services/ipoid: apply [production]
11:43 <kharlan@deploy2002> helmfile [eqiad] START helmfile.d/services/ipoid: apply [production]
11:28 <moritzm> installing postgresql-11 security updates [production]
10:50 <samtar@deploy2002> Finished scap: Backport for [[gerrit:981423|testwiki: Enable the Edit Recovery feature (T353041)]] (duration: 09m 51s) [production]
10:47 <arnaudb@cumin1001> START - Cookbook sre.mysql.clone of db1129.eqiad.wmnet onto db1229.eqiad.wmnet [production]
10:44 <arnaudb@cumin1001> dbctl commit (dc=all): 'Cloning db1129 in db1229 for T344036', diff saved to https://phabricator.wikimedia.org/P54335 and previous config saved to /var/cache/conftool/dbconfig/20231212-104404-arnaudb.json [production]
10:43 <samtar@deploy2002> samtar and samwilson: Continuing with sync [production]
10:42 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1229.eqiad.wmnet with reason: provisionning db1229.eqiad.wmnet - T344036 [production]
10:42 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1229.eqiad.wmnet with reason: provisionning db1229.eqiad.wmnet - T344036 [production]
10:42 <arnaudb@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1129.eqiad.wmnet with reason: provisionning db1229.eqiad.wmnet - T344036 [production]
10:41 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1129.eqiad.wmnet with reason: provisionning db1229.eqiad.wmnet - T344036 [production]
10:41 <samtar@deploy2002> samtar and samwilson: Backport for [[gerrit:981423|testwiki: Enable the Edit Recovery feature (T353041)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
10:40 <samtar@deploy2002> Started scap: Backport for [[gerrit:981423|testwiki: Enable the Edit Recovery feature (T353041)]] [production]
10:30 <moritzm> installing nghttp2 security updates [production]
10:16 <elukey@deploy2002> helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. [production]
10:15 <elukey@deploy2002> helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. [production]
10:13 <kharlan@deploy2002> helmfile [eqiad] DONE helmfile.d/services/ipoid: apply [production]
10:13 <kharlan@deploy2002> helmfile [eqiad] START helmfile.d/services/ipoid: apply [production]