4101-4150 of 10000 results (122ms)
2024-08-28 ยง
12:27 <arnaudb@cumin1002> START - Cookbook sre.switchdc.databases.finalize for the switch from test-s1 to test-s1 [production]
12:23 <MichaelG_WMF> T371228 running foreachwikiindblist growthexperiments ./extensions/CommunityConfiguration/maintenance/setVersionData.php HelpPanel 1.0.0 [production]
12:22 <arnaudb@cumin1002> END (ERROR) - Cookbook sre.switchdc.databases.finalize (exit_code=97) for the switch from test-s1 to test-s1 [production]
12:22 <arnaudb@cumin1002> START - Cookbook sre.switchdc.databases.finalize for the switch from test-s1 to test-s1 [production]
12:19 <MichaelG_WMF> T371228 running mwscript --wiki testwiki ./extensions/CommunityConfiguration/maintenance/setVersionData.php HelpPanel 1.0.0 [production]
11:51 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db2155 (T371742)', diff saved to https://phabricator.wikimedia.org/P68039 and previous config saved to /var/cache/conftool/dbconfig/20240828-115123-ladsgroup.json [production]
11:51 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance [production]
11:51 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance [production]
11:51 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2155.codfw.wmnet with reason: Maintenance [production]
11:51 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 12:00:00 on db2155.codfw.wmnet with reason: Maintenance [production]
11:50 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2147 (T371742)', diff saved to https://phabricator.wikimedia.org/P68038 and previous config saved to /var/cache/conftool/dbconfig/20240828-115057-ladsgroup.json [production]
11:47 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db1175 (T370903)', diff saved to https://phabricator.wikimedia.org/P68037 and previous config saved to /var/cache/conftool/dbconfig/20240828-114745-ladsgroup.json [production]
11:47 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1175.eqiad.wmnet with reason: Maintenance [production]
11:47 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 8:00:00 on db1175.eqiad.wmnet with reason: Maintenance [production]
11:47 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1166 (T370903)', diff saved to https://phabricator.wikimedia.org/P68036 and previous config saved to /var/cache/conftool/dbconfig/20240828-114722-ladsgroup.json [production]
11:44 <hnowlan@deploy1003> helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply [production]
11:43 <hnowlan@deploy1003> helmfile [eqiad] START helmfile.d/services/shellbox-video: apply [production]
11:43 <hnowlan@deploy1003> helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply [production]
11:42 <hnowlan@deploy1003> helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply [production]
11:42 <hnowlan@deploy1003> helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply [production]
11:42 <hnowlan@deploy1003> helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply [production]
11:41 <hnowlan@deploy1003> helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply [production]
11:40 <hnowlan@deploy1003> helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply [production]
11:35 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P68035 and previous config saved to /var/cache/conftool/dbconfig/20240828-113549-ladsgroup.json [production]
11:32 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P68034 and previous config saved to /var/cache/conftool/dbconfig/20240828-113215-ladsgroup.json [production]
11:23 <hnowlan@deploy1003> helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply [production]
11:22 <hnowlan@deploy1003> helmfile [codfw] START helmfile.d/services/shellbox-video: apply [production]
11:20 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P68033 and previous config saved to /var/cache/conftool/dbconfig/20240828-112042-ladsgroup.json [production]
11:17 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P68032 and previous config saved to /var/cache/conftool/dbconfig/20240828-111708-ladsgroup.json [production]
11:17 <hnowlan@deploy1003> helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply [production]
11:14 <hnowlan@deploy1003> helmfile [codfw] DONE helmfile.d/admin 'apply'. [production]
11:12 <hnowlan@deploy1003> helmfile [codfw] START helmfile.d/admin 'apply'. [production]
11:10 <dreamyjazz@deploy1003> Finished scap sync-world: Backport for [[gerrit:1067938|Maintain ranked order of candidates in STV vote summary (T373499)]] (duration: 06m 44s) [production]
11:06 <hnowlan@deploy1003> helmfile [codfw] START helmfile.d/services/shellbox-video: apply [production]
11:06 <dreamyjazz@deploy1003> dreamyjazz: Continuing with sync [production]
11:06 <dreamyjazz@deploy1003> dreamyjazz: Backport for [[gerrit:1067938|Maintain ranked order of candidates in STV vote summary (T373499)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
11:05 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2147 (T371742)', diff saved to https://phabricator.wikimedia.org/P68031 and previous config saved to /var/cache/conftool/dbconfig/20240828-110535-ladsgroup.json [production]
11:03 <dreamyjazz@deploy1003> Started scap sync-world: Backport for [[gerrit:1067938|Maintain ranked order of candidates in STV vote summary (T373499)]] [production]
11:02 <hnowlan@deploy1003> helmfile [staging] DONE helmfile.d/services/shellbox-video: apply [production]
11:02 <hnowlan@deploy1003> helmfile [staging] START helmfile.d/services/shellbox-video: apply [production]
11:02 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1166 (T370903)', diff saved to https://phabricator.wikimedia.org/P68030 and previous config saved to /var/cache/conftool/dbconfig/20240828-110200-ladsgroup.json [production]
10:57 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db1166 (T370903)', diff saved to https://phabricator.wikimedia.org/P68029 and previous config saved to /var/cache/conftool/dbconfig/20240828-105757-ladsgroup.json [production]
10:57 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1166.eqiad.wmnet with reason: Maintenance [production]
10:57 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 8:00:00 on db1166.eqiad.wmnet with reason: Maintenance [production]
10:57 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1157 (T370903)', diff saved to https://phabricator.wikimedia.org/P68028 and previous config saved to /var/cache/conftool/dbconfig/20240828-105735-ladsgroup.json [production]
10:50 <cmooney@cumin1002> END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin1002.eqiad.wmnet with reason: Relase v0.7.0 with updated plugin - cmooney@cumin1002 [production]
10:48 <ladsgroup@deploy1003> Finished scap sync-world: Backport for [[gerrit:1067930|Set ruwiki to non simple UI (T372694)]] (duration: 10m 48s) [production]
10:44 <ladsgroup@deploy1003> ladsgroup: Continuing with sync [production]
10:42 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P68027 and previous config saved to /var/cache/conftool/dbconfig/20240828-104228-ladsgroup.json [production]
10:42 <ladsgroup@deploy1003> ladsgroup: Backport for [[gerrit:1067930|Set ruwiki to non simple UI (T372694)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]