851-900 of 10000 results (86ms)
2024-04-03 ยง
06:31 <marostegui@cumin1002> dbctl commit (dc=all): 'db1222 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P59247 and previous config saved to /var/cache/conftool/dbconfig/20240403-063159-root.json [production]
06:26 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P59246 and previous config saved to /var/cache/conftool/dbconfig/20240403-062602-marostegui.json [production]
06:25 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2148.codfw.wmnet with OS bookworm [production]
06:24 <marostegui@cumin1002> dbctl commit (dc=all): 'db2148 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P59245 and previous config saved to /var/cache/conftool/dbconfig/20240403-062436-root.json [production]
06:23 <logmsgbot> @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
06:23 <logmsgbot> @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
06:16 <marostegui@cumin1002> dbctl commit (dc=all): 'db1222 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P59244 and previous config saved to /var/cache/conftool/dbconfig/20240403-061653-root.json [production]
06:13 <logmsgbot> @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
06:13 <logmsgbot> @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
06:10 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1167 (T356166)', diff saved to https://phabricator.wikimedia.org/P59243 and previous config saved to /var/cache/conftool/dbconfig/20240403-061055-marostegui.json [production]
06:04 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2148.codfw.wmnet with reason: host reimage [production]
06:01 <marostegui@cumin1002> dbctl commit (dc=all): 'db1222 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P59242 and previous config saved to /var/cache/conftool/dbconfig/20240403-060147-root.json [production]
06:01 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on db2148.codfw.wmnet with reason: host reimage [production]
05:51 <logmsgbot> @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
05:50 <logmsgbot> @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
05:49 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1222.eqiad.wmnet with OS bookworm [production]
05:46 <marostegui@cumin1002> dbctl commit (dc=all): 'db1222 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P59241 and previous config saved to /var/cache/conftool/dbconfig/20240403-054641-root.json [production]
05:44 <marostegui@cumin1002> START - Cookbook sre.hosts.reimage for host db2148.codfw.wmnet with OS bookworm [production]
05:43 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2148 T361543', diff saved to https://phabricator.wikimedia.org/P59240 and previous config saved to /var/cache/conftool/dbconfig/20240403-054310-root.json [production]
05:28 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1222.eqiad.wmnet with reason: host reimage [production]
05:25 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on db1222.eqiad.wmnet with reason: host reimage [production]
05:13 <marostegui@cumin1002> START - Cookbook sre.hosts.reimage for host db1222.eqiad.wmnet with OS bookworm [production]
05:11 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1222 T361543', diff saved to https://phabricator.wikimedia.org/P59239 and previous config saved to /var/cache/conftool/dbconfig/20240403-051149-root.json [production]
05:10 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db1167 (T356166)', diff saved to https://phabricator.wikimedia.org/P59238 and previous config saved to /var/cache/conftool/dbconfig/20240403-051029-marostegui.json [production]
05:10 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
05:10 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 16:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
05:10 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1167.eqiad.wmnet with reason: Maintenance [production]
05:09 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 8:00:00 on db1167.eqiad.wmnet with reason: Maintenance [production]
01:58 <logmsgbot> @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
01:58 <logmsgbot> @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
01:51 <logmsgbot> @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
01:51 <logmsgbot> @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
01:40 <cwhite@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host logging-hd2002.codfw.wmnet with OS bookworm [production]
01:19 <cwhite@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on logging-hd2002.codfw.wmnet with reason: host reimage [production]
01:15 <cwhite@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on logging-hd2002.codfw.wmnet with reason: host reimage [production]
01:06 <logmsgbot> @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
01:06 <logmsgbot> @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
01:00 <logmsgbot> @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
01:00 <logmsgbot> @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
00:46 <cwhite@cumin2002> START - Cookbook sre.hosts.reimage for host logging-hd2002.codfw.wmnet with OS bookworm [production]
00:44 <logmsgbot> @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
00:44 <logmsgbot> @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
00:43 <cwhite@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host logging-hd2002.codfw.wmnet with OS bookworm [production]
00:37 <logmsgbot> @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
00:36 <logmsgbot> @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
00:30 <logmsgbot> @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
00:30 <logmsgbot> @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
00:25 <cwhite@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host logging-hd2003.codfw.wmnet with OS bookworm [production]
00:25 <logmsgbot> @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
00:25 <logmsgbot> @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]