2151-2200 of 10000 results (51ms)
2022-04-06 ยง
12:36 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1099:3311 (T300775)', diff saved to https://phabricator.wikimedia.org/P24157 and previous config saved to /var/cache/conftool/dbconfig/20220406-123603-marostegui.json [production]
12:35 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1099.eqiad.wmnet with reason: Maintenance [production]
12:35 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1099.eqiad.wmnet with reason: Maintenance [production]
12:35 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1163', diff saved to https://phabricator.wikimedia.org/P24156 and previous config saved to /var/cache/conftool/dbconfig/20220406-123505-root.json [production]
12:35 <btullis@deploy1002> helmfile [staging] DONE helmfile.d/services/datahub: sync on main [production]
12:32 <elukey@deploy1002> helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. [production]
12:12 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1163', diff saved to https://phabricator.wikimedia.org/P24152 and previous config saved to /var/cache/conftool/dbconfig/20220406-121222-ladsgroup.json [production]
12:11 <aborrero@cumin1001> START - Cookbook sre.hosts.reboot-single for host cloudgw1002.eqiad.wmnet [production]
12:10 <kartik@deploy1002> helmfile [codfw] DONE helmfile.d/services/cxserver: apply [production]
12:09 <kartik@deploy1002> helmfile [codfw] START helmfile.d/services/cxserver: apply [production]
12:03 <kartik@deploy1002> helmfile [staging] DONE helmfile.d/services/cxserver: apply [production]
12:02 <kartik@deploy1002> helmfile [staging] START helmfile.d/services/cxserver: apply [production]
12:01 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance [production]
12:01 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance [production]
11:57 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1163 (T298565)', diff saved to https://phabricator.wikimedia.org/P24151 and previous config saved to /var/cache/conftool/dbconfig/20220406-115717-ladsgroup.json [production]
11:51 <mmandere@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4033.ulsfo.wmnet with reason: host reimage [production]
11:49 <btullis@deploy1002> helmfile [staging] DONE helmfile.d/services/datahub: sync on main [production]
11:48 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudgw1002.eqiad.wmnet with OS bullseye [production]
11:47 <mmandere@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cp4033.ulsfo.wmnet with reason: host reimage [production]
11:38 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudgw1002.eqiad.wmnet with reason: host reimage [production]
11:37 <btullis@deploy1002> helmfile [staging] START helmfile.d/services/datahub: apply on main [production]
11:35 <aborrero@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudgw1002.eqiad.wmnet with reason: host reimage [production]
11:33 <btullis@deploy1002> helmfile [staging] DONE helmfile.d/services/datahub: sync on main [production]
11:32 <mmandere@cumin1001> START - Cookbook sre.hosts.reimage for host cp4033.ulsfo.wmnet with OS buster [production]
11:32 <moritzm> installing wavpack security updates [production]
11:24 <mmandere> depool cp4033 for reimage - T290005 [production]
11:23 <marostegui> dbmaint s3@eqiad T297189 [production]
11:23 <aborrero@cumin1001> START - Cookbook sre.hosts.reimage for host cloudgw1002.eqiad.wmnet with OS bullseye [production]
11:22 <btullis@deploy1002> helmfile [staging] START helmfile.d/services/datahub: apply on main [production]
11:20 <mmandere> pool cp4027 with HAProxy as TLS termination layer - T290005 [production]
11:12 <mmandere@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4027.ulsfo.wmnet with OS buster [production]
11:10 <btullis@deploy1002> helmfile [staging] DONE helmfile.d/services/datahub: sync on main [production]
11:03 <mmandere> pool cp3052 with HAProxy as TLS termination layer - T290005 [production]
11:01 <mmandere@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3052.esams.wmnet with OS buster [production]
11:00 <btullis@deploy1002> helmfile [staging] START helmfile.d/services/datahub: apply on main [production]
10:57 <btullis@deploy1002> helmfile [staging] DONE helmfile.d/services/datahub: sync on main [production]
10:48 <elukey@deploy1002> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. [production]
10:47 <btullis@deploy1002> helmfile [staging] START helmfile.d/services/datahub: apply on main [production]
10:39 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1163 (T298565)', diff saved to https://phabricator.wikimedia.org/P24150 and previous config saved to /var/cache/conftool/dbconfig/20220406-103929-ladsgroup.json [production]
10:39 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1163.eqiad.wmnet with reason: Maintenance [production]
10:39 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1163.eqiad.wmnet with reason: Maintenance [production]
10:38 <elukey@deploy1002> helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. [production]
10:32 <mmandere@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3052.esams.wmnet with reason: host reimage [production]
10:30 <jynus> reruning es4 dump on backup2002 [production]
10:29 <mmandere@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4027.ulsfo.wmnet with reason: host reimage [production]
10:29 <elukey@deploy1002> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. [production]
10:28 <mmandere@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cp3052.esams.wmnet with reason: host reimage [production]
10:27 <jmm@cumin2002> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host deploy2002.codfw.wmnet [production]
10:25 <mmandere@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cp4027.ulsfo.wmnet with reason: host reimage [production]
10:24 <aborrero@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudgw1002.eqiad.wmnet with OS bullseye [production]