5151-5200 of 10000 results (92ms)
2022-11-28 ยง
16:18 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1199 (T321126)', diff saved to https://phabricator.wikimedia.org/P41376 and previous config saved to /var/cache/conftool/dbconfig/20221128-161820-marostegui.json [production]
16:17 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1199 (T321126)', diff saved to https://phabricator.wikimedia.org/P41375 and previous config saved to /var/cache/conftool/dbconfig/20221128-161610-marostegui.json [production]
16:17 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1199.eqiad.wmnet with reason: Maintenance [production]
16:16 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on db1199.eqiad.wmnet with reason: Maintenance [production]
16:15 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1190 (T321126)', diff saved to https://phabricator.wikimedia.org/P41374 and previous config saved to /var/cache/conftool/dbconfig/20221128-161549-marostegui.json [production]
16:14 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2138:3312', diff saved to https://phabricator.wikimedia.org/P41373 and previous config saved to /var/cache/conftool/dbconfig/20221128-161417-ladsgroup.json [production]
16:12 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P41372 and previous config saved to /var/cache/conftool/dbconfig/20221128-161247-ladsgroup.json [production]
16:09 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P41371 and previous config saved to /var/cache/conftool/dbconfig/20221128-160929-ladsgroup.json [production]
16:06 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P41370 and previous config saved to /var/cache/conftool/dbconfig/20221128-160641-ladsgroup.json [production]
16:06 <isaranto@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . [production]
16:01 <oblivian@deploy1002> helmfile [eqiad] DONE helmfile.d/services/thumbor: apply [production]
16:01 <jbond@cumin2002> START - Cookbook sre.hosts.reimage for host ms-be2050.codfw.wmnet with OS bullseye [production]
16:00 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P41369 and previous config saved to /var/cache/conftool/dbconfig/20221128-160042-marostegui.json [production]
16:00 <jbond@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ms-be2050.codfw.wmnet with OS bullseye [production]
15:59 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2138:3312', diff saved to https://phabricator.wikimedia.org/P41368 and previous config saved to /var/cache/conftool/dbconfig/20221128-155910-ladsgroup.json [production]
15:57 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P41367 and previous config saved to /var/cache/conftool/dbconfig/20221128-155740-ladsgroup.json [production]
15:54 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P41366 and previous config saved to /var/cache/conftool/dbconfig/20221128-155423-ladsgroup.json [production]
15:53 <jbond@cumin2002> START - Cookbook sre.hosts.reimage for host ms-be2050.codfw.wmnet with OS bullseye [production]
15:52 <isaranto@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . [production]
15:51 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P41365 and previous config saved to /var/cache/conftool/dbconfig/20221128-155135-ladsgroup.json [production]
15:45 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P41364 and previous config saved to /var/cache/conftool/dbconfig/20221128-154536-marostegui.json [production]
15:44 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2138:3312 (T323827)', diff saved to https://phabricator.wikimedia.org/P41363 and previous config saved to /var/cache/conftool/dbconfig/20221128-154404-ladsgroup.json [production]
15:42 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1166 (T323907)', diff saved to https://phabricator.wikimedia.org/P41362 and previous config saved to /var/cache/conftool/dbconfig/20221128-154234-ladsgroup.json [production]
15:41 <oblivian@deploy1002> helmfile [eqiad] START helmfile.d/services/thumbor: apply [production]
15:41 <oblivian@deploy1002> helmfile [codfw] DONE helmfile.d/services/thumbor: apply [production]
15:39 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1129 (T323827)', diff saved to https://phabricator.wikimedia.org/P41361 and previous config saved to /var/cache/conftool/dbconfig/20221128-153916-ladsgroup.json [production]
15:39 <oblivian@deploy1002> helmfile [codfw] START helmfile.d/services/thumbor: apply [production]
15:38 <oblivian@deploy1002> helmfile [staging] DONE helmfile.d/services/thumbor: apply [production]
15:37 <oblivian@deploy1002> helmfile [staging] START helmfile.d/services/thumbor: apply [production]
15:36 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2149 (T323827)', diff saved to https://phabricator.wikimedia.org/P41360 and previous config saved to /var/cache/conftool/dbconfig/20221128-153628-ladsgroup.json [production]
15:34 <filippo@cumin1001> conftool action : set/pooled=true; selector: dnsdisc=thanos-query,name=eqiad [production]
15:33 <godog> revert back to thanos 0.21 - T303154 [production]
15:30 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1190 (T321126)', diff saved to https://phabricator.wikimedia.org/P41359 and previous config saved to /var/cache/conftool/dbconfig/20221128-153029-marostegui.json [production]
15:30 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1129 (T323827)', diff saved to https://phabricator.wikimedia.org/P41358 and previous config saved to /var/cache/conftool/dbconfig/20221128-153016-ladsgroup.json [production]
15:30 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1129.eqiad.wmnet with reason: Maintenance [production]
15:30 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 10:00:00 on db1129.eqiad.wmnet with reason: Maintenance [production]
15:29 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1122 (T323827)', diff saved to https://phabricator.wikimedia.org/P41357 and previous config saved to /var/cache/conftool/dbconfig/20221128-152955-ladsgroup.json [production]
15:28 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1190 (T321126)', diff saved to https://phabricator.wikimedia.org/P41356 and previous config saved to /var/cache/conftool/dbconfig/20221128-152820-marostegui.json [production]
15:28 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1190.eqiad.wmnet with reason: Maintenance [production]
15:28 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on db1190.eqiad.wmnet with reason: Maintenance [production]
15:27 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1160 (T321126)', diff saved to https://phabricator.wikimedia.org/P41355 and previous config saved to /var/cache/conftool/dbconfig/20221128-152758-marostegui.json [production]
15:26 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2138:3312 (T323827)', diff saved to https://phabricator.wikimedia.org/P41354 and previous config saved to /var/cache/conftool/dbconfig/20221128-152631-ladsgroup.json [production]
15:26 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2138.codfw.wmnet with reason: Maintenance [production]
15:26 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 10:00:00 on db2138.codfw.wmnet with reason: Maintenance [production]
15:26 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2126 (T323827)', diff saved to https://phabricator.wikimedia.org/P41353 and previous config saved to /var/cache/conftool/dbconfig/20221128-152609-ladsgroup.json [production]
15:14 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P41352 and previous config saved to /var/cache/conftool/dbconfig/20221128-151448-ladsgroup.json [production]
15:13 <jbond@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-be2050.codfw.wmnet with OS bullseye [production]
15:12 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1160', diff saved to https://phabricator.wikimedia.org/P41351 and previous config saved to /var/cache/conftool/dbconfig/20221128-151252-marostegui.json [production]
15:11 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P41350 and previous config saved to /var/cache/conftool/dbconfig/20221128-151103-ladsgroup.json [production]
15:07 <btullis@cumin1001> END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto analytics cluster: Roll restart of all Presto's jvm daemons. [production]