251-300 of 10000 results (82ms)
2024-04-18 ยง
11:54 <moritzm> upgrading PHP security updates on eqiad baremetal servers T362511 [production]
11:52 <cgoubert@cumin1002> conftool action : set/weight=10:pooled=yes; selector: name=(mw2302.codfw.wmnet|mw2303.codfw.wmnet|mw2304.codfw.wmnet|mw2332.codfw.wmnet|mw2333.codfw.wmnet|mw2334.codfw.wmnet),cluster=kubernetes,service=kubesvc [production]
11:52 <claime> Pooling and uncordoning mw2302.codfw.wmnet,mw2303.codfw.wmnet,mw2304.codfw.wmnet,mw2332.codfw.wmnet,mw2333.codfw.wmnet,mw2334.codfw.wmnet - T351074 [production]
11:45 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P60914 and previous config saved to /var/cache/conftool/dbconfig/20240418-114544-marostegui.json [production]
11:42 <claime> Running homer 'cr*codfw*' commit 'T351074' [production]
11:35 <cgoubert@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw2333.codfw.wmnet with OS bullseye [production]
11:33 <cgoubert@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [production]
11:32 <cgoubert@deploy1002> helmfile [codfw] START helmfile.d/services/mw-debug: apply [production]
11:32 <cgoubert@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply [production]
11:31 <cgoubert@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-debug: apply [production]
11:30 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1231 (T361627)', diff saved to https://phabricator.wikimedia.org/P60913 and previous config saved to /var/cache/conftool/dbconfig/20240418-113037-marostegui.json [production]
11:30 <cgoubert@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw2302.codfw.wmnet with OS bullseye [production]
11:29 <cgoubert@deploy1002> Finished scap: Redeploy mw-on-k8s with full rebuild - Fix setting php.timeout - T358308 (duration: 37m 04s) [production]
11:28 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db1231 (T361627)', diff saved to https://phabricator.wikimedia.org/P60912 and previous config saved to /var/cache/conftool/dbconfig/20240418-112827-marostegui.json [production]
11:28 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db1172 (T352010)', diff saved to https://phabricator.wikimedia.org/P60911 and previous config saved to /var/cache/conftool/dbconfig/20240418-112816-ladsgroup.json [production]
11:28 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1231.eqiad.wmnet with reason: Maintenance [production]
11:28 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1172.eqiad.wmnet with reason: Maintenance [production]
11:28 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on db1231.eqiad.wmnet with reason: Maintenance [production]
11:27 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1172.eqiad.wmnet with reason: Maintenance [production]
11:26 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es1021.eqiad.wmnet [production]
11:25 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1225.eqiad.wmnet with reason: Maintenance [production]
11:25 <cgoubert@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw2304.codfw.wmnet with OS bullseye [production]
11:25 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on db1225.eqiad.wmnet with reason: Maintenance [production]
11:25 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1224 (T361627)', diff saved to https://phabricator.wikimedia.org/P60910 and previous config saved to /var/cache/conftool/dbconfig/20240418-112459-marostegui.json [production]
11:23 <cgoubert@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw2334.codfw.wmnet with OS bullseye [production]
11:20 <cgoubert@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw2332.codfw.wmnet with OS bullseye [production]
11:18 <jmm@cumin2002> START - Cookbook sre.puppet.migrate-host for host es1021.eqiad.wmnet [production]
11:16 <cgoubert@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw2303.codfw.wmnet with OS bullseye [production]
11:13 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es1020.eqiad.wmnet [production]
11:11 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1183 (T355609)', diff saved to https://phabricator.wikimedia.org/P60909 and previous config saved to /var/cache/conftool/dbconfig/20240418-111132-marostegui.json [production]
11:10 <cgoubert@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2302.codfw.wmnet with reason: host reimage [production]
11:09 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1224', diff saved to https://phabricator.wikimedia.org/P60908 and previous config saved to /var/cache/conftool/dbconfig/20240418-110950-marostegui.json [production]
11:08 <cgoubert@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2333.codfw.wmnet with reason: host reimage [production]
11:05 <cgoubert@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2304.codfw.wmnet with reason: host reimage [production]
11:03 <jmm@cumin2002> START - Cookbook sre.puppet.migrate-host for host es1020.eqiad.wmnet [production]
11:02 <cgoubert@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2334.codfw.wmnet with reason: host reimage [production]
11:01 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es2022.codfw.wmnet [production]
11:00 <cgoubert@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2332.codfw.wmnet with reason: host reimage [production]
10:57 <cgoubert@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2303.codfw.wmnet with reason: host reimage [production]
10:56 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1183', diff saved to https://phabricator.wikimedia.org/P60907 and previous config saved to /var/cache/conftool/dbconfig/20240418-105624-marostegui.json [production]
10:56 <cgoubert@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on mw2334.codfw.wmnet with reason: host reimage [production]
10:55 <cgoubert@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on mw2333.codfw.wmnet with reason: host reimage [production]
10:55 <cgoubert@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on mw2304.codfw.wmnet with reason: host reimage [production]
10:54 <cgoubert@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on mw2332.codfw.wmnet with reason: host reimage [production]
10:54 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1224', diff saved to https://phabricator.wikimedia.org/P60906 and previous config saved to /var/cache/conftool/dbconfig/20240418-105441-marostegui.json [production]
10:54 <cgoubert@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on mw2302.codfw.wmnet with reason: host reimage [production]
10:54 <cgoubert@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on mw2303.codfw.wmnet with reason: host reimage [production]
10:52 <cgoubert@deploy1002> Started scap: Redeploy mw-on-k8s with full rebuild - Fix setting php.timeout - T358308 [production]
10:52 <jmm@cumin2002> START - Cookbook sre.puppet.migrate-host for host es2022.codfw.wmnet [production]
10:48 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es2021.codfw.wmnet [production]