151-200 of 10000 results (107ms)
2024-10-31 §
22:29 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P70802 and previous config saved to /var/cache/conftool/dbconfig/20241031-222939-ladsgroup.json [production]
22:21 <bking@cumin2002> START - Cookbook sre.hosts.reimage for host an-presto1019.eqiad.wmnet with OS bullseye [production]
22:14 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P70801 and previous config saved to /var/cache/conftool/dbconfig/20241031-221432-ladsgroup.json [production]
21:59 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2164 (T376905)', diff saved to https://phabricator.wikimedia.org/P70800 and previous config saved to /var/cache/conftool/dbconfig/20241031-215925-ladsgroup.json [production]
21:52 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db2164 (T376905)', diff saved to https://phabricator.wikimedia.org/P70799 and previous config saved to /var/cache/conftool/dbconfig/20241031-215056-ladsgroup.json [production]
21:51 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance [production]
21:51 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance [production]
21:51 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2164.codfw.wmnet with reason: Maintenance [production]
21:50 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2164.codfw.wmnet with reason: Maintenance [production]
21:50 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2163 (T376905)', diff saved to https://phabricator.wikimedia.org/P70798 and previous config saved to /var/cache/conftool/dbconfig/20241031-215025-ladsgroup.json [production]
21:50 <bking@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host an-presto1019.eqiad.wmnet with OS bullseye [production]
21:40 <bking@cumin2002> START - Cookbook sre.hosts.reimage for host an-presto1019.eqiad.wmnet with OS bullseye [production]
21:40 <bking@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['an-presto1019.eqiad.wmnet'] [production]
21:37 <bking@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-presto1019.eqiad.wmnet'] [production]
21:37 <bking@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['an-presto1019.eqiad.wmnet'] [production]
21:35 <bking@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-presto1019.eqiad.wmnet'] [production]
21:35 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P70797 and previous config saved to /var/cache/conftool/dbconfig/20241031-213518-ladsgroup.json [production]
21:35 <bking@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['an-presto1019.eqiad.wmnet'] [production]
21:22 <bking@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-presto1019.eqiad.wmnet'] [production]
21:22 <urandom> Bootstrapping Cassandra/aqs1022-b — T378725 [production]
21:20 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P70796 and previous config saved to /var/cache/conftool/dbconfig/20241031-212011-ladsgroup.json [production]
21:19 <bking@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host an-presto1019.eqiad.wmnet with OS bullseye [production]
21:18 <bking@cumin2002> START - Cookbook sre.hosts.reimage for host an-presto1019.eqiad.wmnet with OS bullseye [production]
21:18 <dancy@deploy2002> Installing scap version "4.119.3" for 210 hosts [production]
21:05 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2163 (T376905)', diff saved to https://phabricator.wikimedia.org/P70795 and previous config saved to /var/cache/conftool/dbconfig/20241031-210504-ladsgroup.json [production]
20:56 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db2163 (T376905)', diff saved to https://phabricator.wikimedia.org/P70794 and previous config saved to /var/cache/conftool/dbconfig/20241031-205631-ladsgroup.json [production]
20:56 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2163.codfw.wmnet with reason: Maintenance [production]
20:56 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2163.codfw.wmnet with reason: Maintenance [production]
20:56 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2162 (T376905)', diff saved to https://phabricator.wikimedia.org/P70793 and previous config saved to /var/cache/conftool/dbconfig/20241031-205604-ladsgroup.json [production]
20:55 <jsn@deploy2002> Finished scap sync-world: Backport for [[gerrit:1084889|Translations for configuration for same-user-same-page reverts in Automoderator (T370795)]], [[gerrit:1084891|Add follow-up message (T372476)]] (duration: 27m 10s) [production]
20:46 <jsn@deploy2002> jsn: Continuing with sync [production]
20:46 <jsn@deploy2002> jsn: Backport for [[gerrit:1084889|Translations for configuration for same-user-same-page reverts in Automoderator (T370795)]], [[gerrit:1084891|Add follow-up message (T372476)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
20:40 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2162', diff saved to https://phabricator.wikimedia.org/P70792 and previous config saved to /var/cache/conftool/dbconfig/20241031-204057-ladsgroup.json [production]
20:28 <jsn@deploy2002> Started scap sync-world: Backport for [[gerrit:1084889|Translations for configuration for same-user-same-page reverts in Automoderator (T370795)]], [[gerrit:1084891|Add follow-up message (T372476)]] [production]
20:25 <swfrench@deploy2002> helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply [production]
20:25 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2162', diff saved to https://phabricator.wikimedia.org/P70791 and previous config saved to /var/cache/conftool/dbconfig/20241031-202549-ladsgroup.json [production]
20:25 <swfrench@deploy2002> helmfile [eqiad] START helmfile.d/services/shellbox-video: apply [production]
20:23 <swfrench@deploy2002> helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply [production]
20:22 <swfrench@deploy2002> helmfile [codfw] START helmfile.d/services/shellbox-video: apply [production]
20:15 <dancy@deploy2002> Finished scap sync-world: Backport for [[gerrit:1085454|tcywikisource: fix typo of author namespace (T378555)]] (duration: 07m 46s) [production]
20:10 <dancy@deploy2002> dancy, anzx: Continuing with sync [production]
20:10 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2162 (T376905)', diff saved to https://phabricator.wikimedia.org/P70790 and previous config saved to /var/cache/conftool/dbconfig/20241031-201042-ladsgroup.json [production]
20:10 <dancy@deploy2002> dancy, anzx: Backport for [[gerrit:1085454|tcywikisource: fix typo of author namespace (T378555)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
20:07 <dancy@deploy2002> Started scap sync-world: Backport for [[gerrit:1085454|tcywikisource: fix typo of author namespace (T378555)]] [production]
20:03 <dancy@deploy2002> Installation of scap version "4.119.2" completed for 210 hosts [production]
20:02 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db2162 (T376905)', diff saved to https://phabricator.wikimedia.org/P70789 and previous config saved to /var/cache/conftool/dbconfig/20241031-200214-ladsgroup.json [production]
20:02 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2162.codfw.wmnet with reason: Maintenance [production]
20:01 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2162.codfw.wmnet with reason: Maintenance [production]
20:01 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2161 (T376905)', diff saved to https://phabricator.wikimedia.org/P70788 and previous config saved to /var/cache/conftool/dbconfig/20241031-200148-ladsgroup.json [production]
19:58 <dancy@deploy2002> Installing scap version "4.119.2" for 210 hosts [production]