|
2023-04-13
§
|
| 06:24 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1221 (re)pooling @ 1%: Pooling', diff saved to https://phabricator.wikimedia.org/P46629 and previous config saved to /var/cache/conftool/dbconfig/20230413-062446-root.json |
[production] |
| 06:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Add db1221 to dbctl T326669', diff saved to https://phabricator.wikimedia.org/P46628 and previous config saved to /var/cache/conftool/dbconfig/20230413-062231-marostegui.json |
[production] |
| 06:19 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1210 (re)pooling @ 1%: Pooling T326669', diff saved to https://phabricator.wikimedia.org/P46627 and previous config saved to /var/cache/conftool/dbconfig/20230413-061913-root.json |
[production] |
| 06:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Add db1210 to dbctl T326669', diff saved to https://phabricator.wikimedia.org/P46626 and previous config saved to /var/cache/conftool/dbconfig/20230413-061716-marostegui.json |
[production] |
| 02:08 |
<fab@deploy2002> |
Finished deploy [airflow-dags/research@f8dad05]: (no justification provided) (duration: 00m 10s) |
[production] |
| 02:07 |
<fab@deploy2002> |
Started deploy [airflow-dags/research@f8dad05]: (no justification provided) |
[production] |
| 02:01 |
<fab@deploy2002> |
Finished deploy [airflow-dags/research@f8dad05]: (no justification provided) (duration: 00m 11s) |
[production] |
| 02:01 |
<fab@deploy2002> |
Started deploy [airflow-dags/research@f8dad05]: (no justification provided) |
[production] |
| 02:00 |
<ejegg> |
civicrm upgraded from 0f37f981 to 2d5ede8d |
[production] |
| 01:41 |
<fab@deploy2002> |
Finished deploy [airflow-dags/research@f8dad05]: (no justification provided) (duration: 00m 10s) |
[production] |
| 01:41 |
<fab@deploy2002> |
Started deploy [airflow-dags/research@f8dad05]: (no justification provided) |
[production] |
| 01:23 |
<fab@deploy2002> |
Finished deploy [airflow-dags/research@f8dad05]: (no justification provided) (duration: 00m 10s) |
[production] |
| 01:23 |
<fab@deploy2002> |
Started deploy [airflow-dags/research@f8dad05]: (no justification provided) |
[production] |
| 00:22 |
<krinkle@deploy2002> |
Finished deploy [integration/docroot@f68055d]: (no justification provided) (duration: 00m 28s) |
[production] |
| 00:21 |
<krinkle@deploy2002> |
Started deploy [integration/docroot@f68055d]: (no justification provided) |
[production] |
|
2023-04-12
§
|
| 23:09 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2182 (T333332)', diff saved to https://phabricator.wikimedia.org/P46625 and previous config saved to /var/cache/conftool/dbconfig/20230412-230933-ladsgroup.json |
[production] |
| 22:54 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P46624 and previous config saved to /var/cache/conftool/dbconfig/20230412-225427-ladsgroup.json |
[production] |
| 22:39 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P46623 and previous config saved to /var/cache/conftool/dbconfig/20230412-223921-ladsgroup.json |
[production] |
| 22:24 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2182 (T333332)', diff saved to https://phabricator.wikimedia.org/P46622 and previous config saved to /var/cache/conftool/dbconfig/20230412-222414-ladsgroup.json |
[production] |
| 22:21 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2182 (T333332)', diff saved to https://phabricator.wikimedia.org/P46621 and previous config saved to /var/cache/conftool/dbconfig/20230412-222141-ladsgroup.json |
[production] |
| 22:21 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2182.codfw.wmnet with reason: Maintenance |
[production] |
| 22:21 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db2182.codfw.wmnet with reason: Maintenance |
[production] |
| 22:21 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2169:3317 (T333332)', diff saved to https://phabricator.wikimedia.org/P46620 and previous config saved to /var/cache/conftool/dbconfig/20230412-222117-ladsgroup.json |
[production] |
| 22:06 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2169:3317', diff saved to https://phabricator.wikimedia.org/P46619 and previous config saved to /var/cache/conftool/dbconfig/20230412-220611-ladsgroup.json |
[production] |
| 21:56 |
<brett@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs2007.codfw.wmnet with OS bullseye |
[production] |
| 21:52 |
<eevans@cumin1001> |
END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for sessionstore1001.eqiad.wmnet |
[production] |
| 21:52 |
<eevans@cumin1001> |
START - Cookbook sre.hosts.remove-downtime for sessionstore1001.eqiad.wmnet |
[production] |
| 21:51 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2169:3317', diff saved to https://phabricator.wikimedia.org/P46618 and previous config saved to /var/cache/conftool/dbconfig/20230412-215104-ladsgroup.json |
[production] |
| 21:38 |
<brett@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs2007.codfw.wmnet with reason: host reimage |
[production] |
| 21:35 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2169:3317 (T333332)', diff saved to https://phabricator.wikimedia.org/P46617 and previous config saved to /var/cache/conftool/dbconfig/20230412-213558-ladsgroup.json |
[production] |
| 21:35 |
<urandom> |
restarting Cassandra —sessionstore1001— to reenable native transport — T327954 |
[production] |
| 21:35 |
<brett@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on lvs2007.codfw.wmnet with reason: host reimage |
[production] |
| 21:33 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2169:3317 (T333332)', diff saved to https://phabricator.wikimedia.org/P46616 and previous config saved to /var/cache/conftool/dbconfig/20230412-213325-ladsgroup.json |
[production] |
| 21:33 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2169.codfw.wmnet with reason: Maintenance |
[production] |
| 21:33 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db2169.codfw.wmnet with reason: Maintenance |
[production] |
| 21:33 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2168:3317 (T333332)', diff saved to https://phabricator.wikimedia.org/P46615 and previous config saved to /var/cache/conftool/dbconfig/20230412-213301-ladsgroup.json |
[production] |
| 21:17 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2168:3317', diff saved to https://phabricator.wikimedia.org/P46614 and previous config saved to /var/cache/conftool/dbconfig/20230412-211755-ladsgroup.json |
[production] |
| 21:16 |
<brett@cumin2002> |
START - Cookbook sre.hosts.reimage for host lvs2007.codfw.wmnet with OS bullseye |
[production] |
| 21:04 |
<mutante> |
gerrit1001 - pushing data over to gerrit1003 via rsync, with bwlimit option: rsync -avp --bwlimit=1m /srv/gerrit/ rsync://gerrit1003.wikimedia.org/gerrit-data/ (T326368) |
[production] |
| 21:02 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2168:3317', diff saved to https://phabricator.wikimedia.org/P46613 and previous config saved to /var/cache/conftool/dbconfig/20230412-210249-ladsgroup.json |
[production] |
| 21:01 |
<brett@cumin2002> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host lvs2007.codfw.wmnet with OS bullseye |
[production] |
| 21:01 |
<brett@cumin2002> |
START - Cookbook sre.hosts.reimage for host lvs2007.codfw.wmnet with OS bullseye |
[production] |
| 20:58 |
<brett> |
Disable Puppet/PyBal on lvs2007 in preparation for reimaging - T321309 |
[production] |
| 20:47 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2168:3317 (T333332)', diff saved to https://phabricator.wikimedia.org/P46612 and previous config saved to /var/cache/conftool/dbconfig/20230412-204742-ladsgroup.json |
[production] |
| 20:47 |
<andrew@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudvirtlocal1002.eqiad.wmnet with OS bullseye |
[production] |
| 20:45 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2168:3317 (T333332)', diff saved to https://phabricator.wikimedia.org/P46611 and previous config saved to /var/cache/conftool/dbconfig/20230412-204508-ladsgroup.json |
[production] |
| 20:45 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2168.codfw.wmnet with reason: Maintenance |
[production] |
| 20:44 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db2168.codfw.wmnet with reason: Maintenance |
[production] |
| 20:44 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2159 (T333332)', diff saved to https://phabricator.wikimedia.org/P46610 and previous config saved to /var/cache/conftool/dbconfig/20230412-204445-ladsgroup.json |
[production] |
| 20:38 |
<andrew@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye |
[production] |