2901-2950 of 10000 results (117ms)
2024-11-27 ยง
21:38 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1216.eqiad.wmnet with reason: Maintenance [production]
21:38 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 8:00:00 on db1216.eqiad.wmnet with reason: Maintenance [production]
21:38 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1214 (T370903)', diff saved to https://phabricator.wikimedia.org/P71258 and previous config saved to /var/cache/conftool/dbconfig/20241127-213759-ladsgroup.json [production]
21:37 <bking@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1026.eqiad.wmnet with OS bullseye [production]
21:37 <bking@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - bking@cumin2002" [production]
21:37 <cjming@deploy2002> cjming, cscott: Continuing with sync [production]
21:37 <cjming@deploy2002> cjming, cscott: Backport for [[gerrit:1098567|Revert "Normalize ref html before comparison" (T380977)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
21:31 <cjming@deploy2002> Started scap sync-world: Backport for [[gerrit:1098567|Revert "Normalize ref html before comparison" (T380977)]] [production]
21:22 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P71257 and previous config saved to /var/cache/conftool/dbconfig/20241127-212252-ladsgroup.json [production]
21:17 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance es2029 (T376905)', diff saved to https://phabricator.wikimedia.org/P71256 and previous config saved to /var/cache/conftool/dbconfig/20241127-211704-ladsgroup.json [production]
21:07 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P71255 and previous config saved to /var/cache/conftool/dbconfig/20241127-210745-ladsgroup.json [production]
21:01 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance es2029', diff saved to https://phabricator.wikimedia.org/P71254 and previous config saved to /var/cache/conftool/dbconfig/20241127-210157-ladsgroup.json [production]
20:52 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1214 (T370903)', diff saved to https://phabricator.wikimedia.org/P71253 and previous config saved to /var/cache/conftool/dbconfig/20241127-205238-ladsgroup.json [production]
20:46 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance es2029', diff saved to https://phabricator.wikimedia.org/P71252 and previous config saved to /var/cache/conftool/dbconfig/20241127-204650-ladsgroup.json [production]
20:45 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2237.codfw.wmnet with reason: Optimize (T379813) [production]
20:45 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2237.codfw.wmnet with reason: Optimize (T379813) [production]
20:44 <ladsgroup@cumin1002> dbctl commit (dc=all): 'db2237 depool (T379813)', diff saved to https://phabricator.wikimedia.org/P71251 and previous config saved to /var/cache/conftool/dbconfig/20241127-204450-ladsgroup.json [production]
20:38 <bking@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - bking@cumin2002" [production]
20:37 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db1214 (T370903)', diff saved to https://phabricator.wikimedia.org/P71250 and previous config saved to /var/cache/conftool/dbconfig/20241127-203724-ladsgroup.json [production]
20:37 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1214.eqiad.wmnet with reason: Maintenance [production]
20:36 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 8:00:00 on db1214.eqiad.wmnet with reason: Maintenance [production]
20:36 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1211 (T370903)', diff saved to https://phabricator.wikimedia.org/P71249 and previous config saved to /var/cache/conftool/dbconfig/20241127-203650-ladsgroup.json [production]
20:31 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance es2029 (T376905)', diff saved to https://phabricator.wikimedia.org/P71248 and previous config saved to /var/cache/conftool/dbconfig/20241127-203143-ladsgroup.json [production]
20:24 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling es2029 (T376905)', diff saved to https://phabricator.wikimedia.org/P71247 and previous config saved to /var/cache/conftool/dbconfig/20241127-202446-ladsgroup.json [production]
20:24 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2029.codfw.wmnet with reason: Maintenance [production]
20:24 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on es2029.codfw.wmnet with reason: Maintenance [production]
20:24 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance es2034 (T376905)', diff saved to https://phabricator.wikimedia.org/P71246 and previous config saved to /var/cache/conftool/dbconfig/20241127-202420-ladsgroup.json [production]
20:21 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1211', diff saved to https://phabricator.wikimedia.org/P71245 and previous config saved to /var/cache/conftool/dbconfig/20241127-202143-ladsgroup.json [production]
20:20 <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1026.eqiad.wmnet with reason: host reimage [production]
20:18 <bking@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1026.eqiad.wmnet with reason: host reimage [production]
20:09 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance es2034', diff saved to https://phabricator.wikimedia.org/P71244 and previous config saved to /var/cache/conftool/dbconfig/20241127-200913-ladsgroup.json [production]
20:06 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1211', diff saved to https://phabricator.wikimedia.org/P71243 and previous config saved to /var/cache/conftool/dbconfig/20241127-200636-ladsgroup.json [production]
19:54 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance es2034', diff saved to https://phabricator.wikimedia.org/P71242 and previous config saved to /var/cache/conftool/dbconfig/20241127-195406-ladsgroup.json [production]
19:51 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1211 (T370903)', diff saved to https://phabricator.wikimedia.org/P71241 and previous config saved to /var/cache/conftool/dbconfig/20241127-195129-ladsgroup.json [production]
19:50 <bking@cumin2002> START - Cookbook sre.hosts.reimage for host wdqs1026.eqiad.wmnet with OS bullseye [production]
19:50 <bking@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host wdqs1025.eqiad.wmnet with OS bullseye [production]
19:38 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance es2034 (T376905)', diff saved to https://phabricator.wikimedia.org/P71240 and previous config saved to /var/cache/conftool/dbconfig/20241127-193858-ladsgroup.json [production]
19:36 <moritzm> imported jenkins 2.479.2 to thirdparty/ci for bullseye-wikimedia [production]
19:35 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db1211 (T370903)', diff saved to https://phabricator.wikimedia.org/P71239 and previous config saved to /var/cache/conftool/dbconfig/20241127-193529-ladsgroup.json [production]
19:35 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1211.eqiad.wmnet with reason: Maintenance [production]
19:35 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 8:00:00 on db1211.eqiad.wmnet with reason: Maintenance [production]
19:35 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1209 (T370903)', diff saved to https://phabricator.wikimedia.org/P71238 and previous config saved to /var/cache/conftool/dbconfig/20241127-193507-ladsgroup.json [production]
19:34 <bking@cumin2002> START - Cookbook sre.hosts.reimage for host wdqs1025.eqiad.wmnet with OS bullseye [production]
19:32 <bking@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host wdqs1025.eqiad.wmnet with OS bullseye [production]
19:32 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling es2034 (T376905)', diff saved to https://phabricator.wikimedia.org/P71237 and previous config saved to /var/cache/conftool/dbconfig/20241127-193202-ladsgroup.json [production]
19:31 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2034.codfw.wmnet with reason: Maintenance [production]
19:31 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on es2034.codfw.wmnet with reason: Maintenance [production]
19:25 <jclark@cumin1002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wdqs1027.eqiad.wmnet with OS bullseye [production]
19:24 <bking@cumin2002> START - Cookbook sre.hosts.reimage for host wdqs1025.eqiad.wmnet with OS bullseye [production]
19:23 <bking@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wdqs1025.eqiad.wmnet with OS bullseye [production]