6451-6500 of 10000 results (42ms)
2024-11-27 ยง
22:11 <bking@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1027.eqiad.wmnet with reason: host reimage [production]
22:09 <cjming@deploy2002> arlolra, cjming: Backport for [[gerrit:1098581|Bump wikimedia/parsoid to 0.21.0-a9 (T373035 T380664)]], [[gerrit:1098583|Bump wikimedia/parsoid to 0.21.0-a9 (T380664)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
22:07 <bking@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1027.eqiad.wmnet with reason: host reimage [production]
22:06 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1226 (T370903)', diff saved to https://phabricator.wikimedia.org/P71260 and previous config saved to /var/cache/conftool/dbconfig/20241127-220638-ladsgroup.json [production]
21:54 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db1226 (T370903)', diff saved to https://phabricator.wikimedia.org/P71259 and previous config saved to /var/cache/conftool/dbconfig/20241127-215407-ladsgroup.json [production]
21:54 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1226.eqiad.wmnet with reason: Maintenance [production]
21:53 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 8:00:00 on db1226.eqiad.wmnet with reason: Maintenance [production]
21:45 <cjming@deploy2002> Started scap sync-world: Backport for [[gerrit:1098581|Bump wikimedia/parsoid to 0.21.0-a9 (T373035 T380664)]], [[gerrit:1098583|Bump wikimedia/parsoid to 0.21.0-a9 (T380664)]] [production]
21:43 <cjming@deploy2002> Finished scap sync-world: Backport for [[gerrit:1098567|Revert "Normalize ref html before comparison" (T380977)]] (duration: 12m 49s) [production]
21:40 <bking@cumin1002> START - Cookbook sre.hosts.reimage for host wdqs1027.eqiad.wmnet with OS bullseye [production]
21:38 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1216.eqiad.wmnet with reason: Maintenance [production]
21:38 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 8:00:00 on db1216.eqiad.wmnet with reason: Maintenance [production]
21:38 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1214 (T370903)', diff saved to https://phabricator.wikimedia.org/P71258 and previous config saved to /var/cache/conftool/dbconfig/20241127-213759-ladsgroup.json [production]
21:37 <bking@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1026.eqiad.wmnet with OS bullseye [production]
21:37 <bking@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - bking@cumin2002" [production]
21:37 <cjming@deploy2002> cjming, cscott: Continuing with sync [production]
21:37 <cjming@deploy2002> cjming, cscott: Backport for [[gerrit:1098567|Revert "Normalize ref html before comparison" (T380977)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
21:31 <cjming@deploy2002> Started scap sync-world: Backport for [[gerrit:1098567|Revert "Normalize ref html before comparison" (T380977)]] [production]
21:22 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P71257 and previous config saved to /var/cache/conftool/dbconfig/20241127-212252-ladsgroup.json [production]
21:17 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance es2029 (T376905)', diff saved to https://phabricator.wikimedia.org/P71256 and previous config saved to /var/cache/conftool/dbconfig/20241127-211704-ladsgroup.json [production]
21:07 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P71255 and previous config saved to /var/cache/conftool/dbconfig/20241127-210745-ladsgroup.json [production]
21:01 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance es2029', diff saved to https://phabricator.wikimedia.org/P71254 and previous config saved to /var/cache/conftool/dbconfig/20241127-210157-ladsgroup.json [production]
20:52 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1214 (T370903)', diff saved to https://phabricator.wikimedia.org/P71253 and previous config saved to /var/cache/conftool/dbconfig/20241127-205238-ladsgroup.json [production]
20:46 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance es2029', diff saved to https://phabricator.wikimedia.org/P71252 and previous config saved to /var/cache/conftool/dbconfig/20241127-204650-ladsgroup.json [production]
20:45 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2237.codfw.wmnet with reason: Optimize (T379813) [production]
20:45 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2237.codfw.wmnet with reason: Optimize (T379813) [production]
20:44 <ladsgroup@cumin1002> dbctl commit (dc=all): 'db2237 depool (T379813)', diff saved to https://phabricator.wikimedia.org/P71251 and previous config saved to /var/cache/conftool/dbconfig/20241127-204450-ladsgroup.json [production]
20:38 <bking@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - bking@cumin2002" [production]
20:37 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db1214 (T370903)', diff saved to https://phabricator.wikimedia.org/P71250 and previous config saved to /var/cache/conftool/dbconfig/20241127-203724-ladsgroup.json [production]
20:37 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1214.eqiad.wmnet with reason: Maintenance [production]
20:36 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 8:00:00 on db1214.eqiad.wmnet with reason: Maintenance [production]
20:36 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1211 (T370903)', diff saved to https://phabricator.wikimedia.org/P71249 and previous config saved to /var/cache/conftool/dbconfig/20241127-203650-ladsgroup.json [production]
20:31 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance es2029 (T376905)', diff saved to https://phabricator.wikimedia.org/P71248 and previous config saved to /var/cache/conftool/dbconfig/20241127-203143-ladsgroup.json [production]
20:24 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling es2029 (T376905)', diff saved to https://phabricator.wikimedia.org/P71247 and previous config saved to /var/cache/conftool/dbconfig/20241127-202446-ladsgroup.json [production]
20:24 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2029.codfw.wmnet with reason: Maintenance [production]
20:24 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on es2029.codfw.wmnet with reason: Maintenance [production]
20:24 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance es2034 (T376905)', diff saved to https://phabricator.wikimedia.org/P71246 and previous config saved to /var/cache/conftool/dbconfig/20241127-202420-ladsgroup.json [production]
20:21 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1211', diff saved to https://phabricator.wikimedia.org/P71245 and previous config saved to /var/cache/conftool/dbconfig/20241127-202143-ladsgroup.json [production]
20:20 <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1026.eqiad.wmnet with reason: host reimage [production]
20:18 <bking@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1026.eqiad.wmnet with reason: host reimage [production]
20:09 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance es2034', diff saved to https://phabricator.wikimedia.org/P71244 and previous config saved to /var/cache/conftool/dbconfig/20241127-200913-ladsgroup.json [production]
20:06 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1211', diff saved to https://phabricator.wikimedia.org/P71243 and previous config saved to /var/cache/conftool/dbconfig/20241127-200636-ladsgroup.json [production]
19:54 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance es2034', diff saved to https://phabricator.wikimedia.org/P71242 and previous config saved to /var/cache/conftool/dbconfig/20241127-195406-ladsgroup.json [production]
19:51 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1211 (T370903)', diff saved to https://phabricator.wikimedia.org/P71241 and previous config saved to /var/cache/conftool/dbconfig/20241127-195129-ladsgroup.json [production]
19:50 <bking@cumin2002> START - Cookbook sre.hosts.reimage for host wdqs1026.eqiad.wmnet with OS bullseye [production]
19:50 <bking@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host wdqs1025.eqiad.wmnet with OS bullseye [production]
19:39 <mforns> re-ran wikidata_wikitext_history for 2024-08 [analytics]
19:38 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance es2034 (T376905)', diff saved to https://phabricator.wikimedia.org/P71240 and previous config saved to /var/cache/conftool/dbconfig/20241127-193858-ladsgroup.json [production]
19:36 <moritzm> imported jenkins 2.479.2 to thirdparty/ci for bullseye-wikimedia [production]
19:35 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db1211 (T370903)', diff saved to https://phabricator.wikimedia.org/P71239 and previous config saved to /var/cache/conftool/dbconfig/20241127-193529-ladsgroup.json [production]