1151-1200 of 10000 results (64ms)
2022-04-08 ยง
13:35 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
13:35 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
13:35 <marostegui@cumin1001> dbctl commit (dc=all): 'db1184 (re)pooling @ 10%: After schema change', diff saved to https://phabricator.wikimedia.org/P24297 and previous config saved to /var/cache/conftool/dbconfig/20220408-133528-root.json [production]
13:30 <jynus@cumin2002> START - Cookbook sre.hosts.reimage for host backup2008.codfw.wmnet with OS bullseye [production]
13:21 <jayme@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on kubemaster1001.eqiad.wmnet with reason: reimage [production]
13:21 <jayme@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on kubemaster1001.eqiad.wmnet with reason: reimage [production]
13:20 <mmandere> pool cp6001 with HAProxy as TLS termination layer - T290005 [production]
13:20 <marostegui@cumin1001> dbctl commit (dc=all): 'db1184 (re)pooling @ 5%: After schema change', diff saved to https://phabricator.wikimedia.org/P24296 and previous config saved to /var/cache/conftool/dbconfig/20220408-132024-root.json [production]
13:18 <Emperor> exiqgrep -i -r fr-tech-failmail@wikimedia.org | xargs exim -Mrm on mx1001 (again again again; keeping queue below the p.age threshold while fr-tech work) [production]
13:16 <mmandere> pool cp6009 with HAProxy as TLS termination layer - T290005 [production]
13:13 <mmandere@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp6001.drmrs.wmnet with OS buster [production]
13:11 <mmandere@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp6009.drmrs.wmnet with OS buster [production]
13:00 <gmodena@deploy1002> Finished deploy [airflow-dags/research@b029f10]: (no justification provided) (duration: 02m 11s) [production]
12:59 <Emperor> exiqgrep -i -r fr-tech-failmail@wikimedia.org | xargs exim -Mrm on mx1001 (again again) [production]
12:58 <gmodena@deploy1002> Started deploy [airflow-dags/research@b029f10]: (no justification provided) [production]
12:57 <jayme@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on kubemaster1002.eqiad.wmnet with reason: reimage [production]
12:57 <jayme@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on kubemaster1002.eqiad.wmnet with reason: reimage [production]
12:54 <Emperor> exiqgrep -i -r fr-tech-failmail@wikimedia.org | xargs exim -Mrm on mx1001 (again) [production]
12:49 <ejegg> disabled paypal IPN listener failmail [production]
12:44 <mmandere@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp6001.drmrs.wmnet with reason: host reimage [production]
12:40 <mmandere@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cp6001.drmrs.wmnet with reason: host reimage [production]
12:33 <mmandere@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp6009.drmrs.wmnet with reason: host reimage [production]
12:29 <mmandere@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cp6009.drmrs.wmnet with reason: host reimage [production]
12:22 <mmandere@cumin1001> START - Cookbook sre.hosts.reimage for host cp6001.drmrs.wmnet with OS buster [production]
12:15 <mmandere> depool cp6001 for reimage - T290005 [production]
12:11 <mmandere@cumin1001> START - Cookbook sre.hosts.reimage for host cp6009.drmrs.wmnet with OS buster [production]
12:11 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1119 (T298565)', diff saved to https://phabricator.wikimedia.org/P24295 and previous config saved to /var/cache/conftool/dbconfig/20220408-121138-ladsgroup.json [production]
12:11 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1119.eqiad.wmnet with reason: Maintenance [production]
12:11 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1119.eqiad.wmnet with reason: Maintenance [production]
11:45 <Emperor> exiqgrep -i -r fr-tech-failmail@wikimedia.org | xargs exim -Mrm on mx1001 [production]
11:34 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1184', diff saved to https://phabricator.wikimedia.org/P24294 and previous config saved to /var/cache/conftool/dbconfig/20220408-113452-root.json [production]
11:25 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 14 hosts with reason: Maintenance [production]
11:24 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on 14 hosts with reason: Maintenance [production]
11:24 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2103.codfw.wmnet with reason: Maintenance [production]
11:24 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db2103.codfw.wmnet with reason: Maintenance [production]
11:11 <mmandere> depool cp6009 for reimage - T290005 [production]
10:38 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1133.eqiad.wmnet with reason: Maintenance [production]
10:38 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1133.eqiad.wmnet with reason: Maintenance [production]
10:18 <mmandere> pool cp6002 with HAProxy as TLS termination layer - T290005 [production]
10:15 <jynus@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov1003.eqiad.wmnet with OS bullseye [production]
10:11 <mmandere> pool cp6010 with HAProxy as TLS termination layer - T290005 [production]
10:07 <mmandere@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp6010.drmrs.wmnet with OS buster [production]
10:05 <mmandere@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp6002.drmrs.wmnet with OS buster [production]
10:04 <jynus@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov1003.eqiad.wmnet with reason: host reimage [production]
10:00 <jynus@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov1003.eqiad.wmnet with reason: host reimage [production]
09:54 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1127 (T305300)', diff saved to https://phabricator.wikimedia.org/P24293 and previous config saved to /var/cache/conftool/dbconfig/20220408-095458-ladsgroup.json [production]
09:54 <cmooney@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dumpsdata1006.eqiad.wmnet with OS buster [production]
09:48 <jynus@cumin1001> START - Cookbook sre.hosts.reimage for host dbprov1003.eqiad.wmnet with OS bullseye [production]
09:47 <jynus@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2003.codfw.wmnet with OS bullseye [production]
09:43 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1132.eqiad.wmnet with reason: Maintenance [production]