|
2022-07-01
ยง
|
| 11:01 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1172.eqiad.wmnet with reason: Maintenance |
[production] |
| 11:01 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1179 (T309311)', diff saved to https://phabricator.wikimedia.org/P30713 and previous config saved to /var/cache/conftool/dbconfig/20220701-110117-ladsgroup.json |
[production] |
| 10:46 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P30712 and previous config saved to /var/cache/conftool/dbconfig/20220701-104612-ladsgroup.json |
[production] |
| 10:45 |
<bmansurov@deploy1002> |
Finished deploy [airflow-dags/research@b3fe77c]: (no justification provided) (duration: 00m 08s) |
[production] |
| 10:45 |
<bmansurov@deploy1002> |
Started deploy [airflow-dags/research@b3fe77c]: (no justification provided) |
[production] |
| 10:44 |
<bmansurov@deploy1002> |
Finished deploy [airflow-dags/research@b3fe77c]: (no justification provided) (duration: 00m 09s) |
[production] |
| 10:44 |
<bmansurov@deploy1002> |
Started deploy [airflow-dags/research@b3fe77c]: (no justification provided) |
[production] |
| 10:31 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P30711 and previous config saved to /var/cache/conftool/dbconfig/20220701-103107-ladsgroup.json |
[production] |
| 10:28 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1135 (T309311)', diff saved to https://phabricator.wikimedia.org/P30710 and previous config saved to /var/cache/conftool/dbconfig/20220701-102810-ladsgroup.json |
[production] |
| 10:28 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1135.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:27 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1135.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:16 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1179 (T309311)', diff saved to https://phabricator.wikimedia.org/P30709 and previous config saved to /var/cache/conftool/dbconfig/20220701-101602-ladsgroup.json |
[production] |
| 09:49 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1179 (T309311)', diff saved to https://phabricator.wikimedia.org/P30708 and previous config saved to /var/cache/conftool/dbconfig/20220701-094927-ladsgroup.json |
[production] |
| 09:49 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:49 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:39 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 13 hosts with reason: Maintenance |
[production] |
| 09:39 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on 13 hosts with reason: Maintenance |
[production] |
| 09:39 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2103.codfw.wmnet with reason: Maintenance |
[production] |
| 09:39 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db2103.codfw.wmnet with reason: Maintenance |
[production] |
| 08:35 |
<marostegui> |
Stop mysql on db2073 for cloning db2155 |
[production] |
| 07:47 |
<mmandere> |
kubemaster2001, restart rsyslog |
[production] |
| 07:46 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Add db2154 to s8 T311493', diff saved to https://phabricator.wikimedia.org/P30705 and previous config saved to /var/cache/conftool/dbconfig/20220701-074607-marostegui.json |
[production] |
| 07:35 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Add db2153 to s1 T311493', diff saved to https://phabricator.wikimedia.org/P30704 and previous config saved to /var/cache/conftool/dbconfig/20220701-073512-marostegui.json |
[production] |
| 06:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Remove db2091 from dbctl T311803', diff saved to https://phabricator.wikimedia.org/P30703 and previous config saved to /var/cache/conftool/dbconfig/20220701-060000-marostegui.json |
[production] |
| 05:41 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Remove db2092 from dbctl T311802', diff saved to https://phabricator.wikimedia.org/P30701 and previous config saved to /var/cache/conftool/dbconfig/20220701-054102-marostegui.json |
[production] |
| 02:31 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2165.codfw.wmnet with OS bullseye |
[production] |
| 02:16 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2165.codfw.wmnet with reason: host reimage |
[production] |
| 02:13 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db2165.codfw.wmnet with reason: host reimage |
[production] |
| 02:06 |
<krinkle@deploy1002> |
Synchronized wmf-config/: I60edfb0f60 (3/3) (duration: 03m 31s) |
[production] |
| 02:01 |
<krinkle@deploy1002> |
Synchronized multiversion/: I60edfb0f60 (2/3) (duration: 03m 34s) |
[production] |
| 01:54 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.reimage for host db2165.codfw.wmnet with OS bullseye |
[production] |
| 01:49 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2163.codfw.wmnet with OS bullseye |
[production] |
| 01:39 |
<krinkle@deploy1002> |
Synchronized tests/: I60edfb0f60 (1/3) (duration: 03m 32s) |
[production] |
| 01:35 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2163.codfw.wmnet with reason: host reimage |
[production] |
| 01:33 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 01:32 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 01:32 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 01:31 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db2163.codfw.wmnet with reason: host reimage |
[production] |
| 01:31 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 01:30 |
<krinkle@deploy1002> |
Synchronized src/: I796f38d0f04600c (3/3) (duration: 03m 24s) |
[production] |
| 01:26 |
<krinkle@deploy1002> |
Synchronized multiversion/: I796f38d0f04600c (2/3) (duration: 03m 32s) |
[production] |
| 01:23 |
<krinkle@deploy1002> |
Synchronized tests/: I796f38d0f04600c (1/3) (duration: 03m 41s) |
[production] |
| 01:21 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 01:20 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 01:20 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 01:19 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 01:17 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2162.codfw.wmnet with OS bullseye |
[production] |
| 01:12 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.reimage for host db2163.codfw.wmnet with OS bullseye |
[production] |
| 01:02 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2162.codfw.wmnet with reason: host reimage |
[production] |
| 01:00 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2161.codfw.wmnet with OS bullseye |
[production] |