2051-2100 of 10000 results (89ms)
2023-10-05 §
05:56 <arnaudb@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2165.codfw.wmnet with reason: Maintenance [production]
05:56 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2164 (T343198)', diff saved to https://phabricator.wikimedia.org/P52830 and previous config saved to /var/cache/conftool/dbconfig/20231005-055615-arnaudb.json [production]
05:41 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P52829 and previous config saved to /var/cache/conftool/dbconfig/20231005-054109-arnaudb.json [production]
05:29 <denisse> Deleting old Jenkins builds on pcc-worker1002 to free disk space [production]
05:26 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P52828 and previous config saved to /var/cache/conftool/dbconfig/20231005-052602-arnaudb.json [production]
05:10 <arnaudb@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2164 (T343198)', diff saved to https://phabricator.wikimedia.org/P52827 and previous config saved to /var/cache/conftool/dbconfig/20231005-051056-arnaudb.json [production]
02:50 <eileen> civicrm upgraded from 44800fc0 to 05545fbc [production]
2023-10-04 §
23:44 <jclark@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1067.eqiad.wmnet with OS bullseye [production]
23:38 <jclark@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1064.eqiad.wmnet with OS bullseye [production]
23:32 <jclark@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1063.eqiad.wmnet with OS bullseye [production]
22:40 <jclark@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1065.eqiad.wmnet with OS bullseye [production]
22:40 <jclark@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1001" [production]
22:39 <jclark@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1001" [production]
22:24 <jclark@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1065.eqiad.wmnet with reason: host reimage [production]
22:23 <jclark@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirt1067.eqiad.wmnet with OS bullseye [production]
22:21 <jclark@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1065.eqiad.wmnet with reason: host reimage [production]
22:18 <jclark@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirt1064.eqiad.wmnet with OS bullseye [production]
22:13 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kubernetes2054.codfw.wmnet with OS bullseye [production]
22:11 <jclark@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirt1063.eqiad.wmnet with OS bullseye [production]
22:06 <jclark@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1067.eqiad.wmnet with OS bullseye [production]
22:06 <jclark@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1064.eqiad.wmnet with OS bullseye [production]
22:06 <jclark@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1063.eqiad.wmnet with OS bullseye [production]
22:05 <jclark@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirt1065.eqiad.wmnet with OS bullseye [production]
22:02 <urandom> starting Cassandra rebuild, restbase1030-b — T346803 [production]
22:02 <brennen@deploy2002> Finished scap: Backport for [[gerrit:963351|Revert "Deprecate TOC mutation in OutputPageParserOutput hook" (T348134)]] (duration: 09m 13s) [production]
21:59 <jclark@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudvirt1065.eqiad.wmnet with OS bullseye [production]
21:58 <volans> uploaded spicerack_7.3.1 to apt.wikimedia.org bullseye-wikimedia [production]
21:56 <brennen@deploy2002> brennen and ssastry: Continuing with sync [production]
21:54 <jclark@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1066.eqiad.wmnet with OS bullseye [production]
21:54 <jclark@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1001" [production]
21:54 <brennen@deploy2002> brennen and ssastry: Backport for [[gerrit:963351|Revert "Deprecate TOC mutation in OutputPageParserOutput hook" (T348134)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
21:53 <jclark@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1001" [production]
21:53 <brennen@deploy2002> Started scap: Backport for [[gerrit:963351|Revert "Deprecate TOC mutation in OutputPageParserOutput hook" (T348134)]] [production]
21:46 <jclark@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirt1065.eqiad.wmnet with OS bullseye [production]
21:44 <jclark@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudvirt1065.eqiad.wmnet with OS bullseye [production]
21:40 <jclark@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1062.eqiad.wmnet with OS bullseye [production]
21:40 <jclark@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1001" [production]
21:38 <jclark@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1001" [production]
21:38 <jclark@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1066.eqiad.wmnet with reason: host reimage [production]
21:34 <jclark@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1066.eqiad.wmnet with reason: host reimage [production]
21:23 <jclark@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1062.eqiad.wmnet with reason: host reimage [production]
21:20 <jclark@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1062.eqiad.wmnet with reason: host reimage [production]
21:02 <jclark@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirt1062.eqiad.wmnet with OS bullseye [production]
20:59 <jclark@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudvirt1062.eqiad.wmnet with OS bullseye [production]
20:58 <jhancock@cumin2002> START - Cookbook sre.hosts.reimage for host kubernetes2054.codfw.wmnet with OS bullseye [production]
20:54 <urbanecm@deploy2002> Finished scap: Backport for [[gerrit:963347|SpecialManageMentors: Skip OOUI initialization when transcluding (T346760)]], [[gerrit:963348|SpecialManageMentors: Skip OOUI initialization when transcluding (T346760)]], [[gerrit:963349|Fix phan for GrowthExperiments (T347571)]] (duration: 07m 49s) [production]
20:48 <urbanecm@deploy2002> urbanecm: Continuing with sync [production]
20:48 <urbanecm@deploy2002> urbanecm: Backport for [[gerrit:963347|SpecialManageMentors: Skip OOUI initialization when transcluding (T346760)]], [[gerrit:963348|SpecialManageMentors: Skip OOUI initialization when transcluding (T346760)]], [[gerrit:963349|Fix phan for GrowthExperiments (T347571)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
20:46 <urbanecm@deploy2002> Started scap: Backport for [[gerrit:963347|SpecialManageMentors: Skip OOUI initialization when transcluding (T346760)]], [[gerrit:963348|SpecialManageMentors: Skip OOUI initialization when transcluding (T346760)]], [[gerrit:963349|Fix phan for GrowthExperiments (T347571)]] [production]
20:46 <jclark@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirt1067.eqiad.wmnet with OS bullseye [production]