5201-5250 of 10000 results (103ms)
2024-03-11 §
14:27 <hashar@deploy2002> Finished deploy [gerrit/gerrit@737c475]: Gerrit to 3.7.8 on gerrit2002 (duration: 00m 03s) [production]
14:27 <hashar@deploy2002> Started deploy [gerrit/gerrit@737c475]: Gerrit to 3.7.8 on gerrit2002 [production]
14:20 <mvolz@deploy2002> Finished scap: Backport for [[gerrit:1009740|editcheckreferenceurl: don't error when aborting the lookupPromise (T359601)]] (duration: 19m 20s) [production]
14:19 <arnaudb@cumin1002> dbctl commit (dc=all): 'db1179 (re)pooling @ 1%: Post upgrade', diff saved to https://phabricator.wikimedia.org/P58704 and previous config saved to /var/cache/conftool/dbconfig/20240311-141945-arnaudb.json [production]
14:17 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1179.eqiad.wmnet with OS bookworm [production]
14:10 <mvolz@deploy2002> mvolz: Continuing with sync [production]
14:02 <mvolz@deploy2002> mvolz: Backport for [[gerrit:1009740|editcheckreferenceurl: don't error when aborting the lookupPromise (T359601)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
14:02 <logmsgbot> @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
14:01 <logmsgbot> @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
14:00 <mvolz@deploy2002> Started scap: Backport for [[gerrit:1009740|editcheckreferenceurl: don't error when aborting the lookupPromise (T359601)]] [production]
13:56 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1179.eqiad.wmnet with reason: host reimage [production]
13:54 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on db1179.eqiad.wmnet with reason: host reimage [production]
13:50 <samtar@deploy2002> Finished scap: Backport for [[gerrit:1010155|nnwiki: Enable sandbox link (T359788)]] (duration: 12m 18s) [production]
13:49 <logmsgbot> @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
13:49 <logmsgbot> @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
13:42 <arnaudb@cumin1002> START - Cookbook sre.hosts.reimage for host db1179.eqiad.wmnet with OS bookworm [production]
13:42 <logmsgbot> @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
13:41 <logmsgbot> @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
13:41 <samtar@deploy2002> jhsoby and samtar: Continuing with sync [production]
13:40 <samtar@deploy2002> jhsoby and samtar: Backport for [[gerrit:1010155|nnwiki: Enable sandbox link (T359788)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
13:40 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1179.eqiad.wmnet with reason: Silence for upgrade [production]
13:40 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 3:00:00 on db1179.eqiad.wmnet with reason: Silence for upgrade [production]
13:38 <samtar@deploy2002> Started scap: Backport for [[gerrit:1010155|nnwiki: Enable sandbox link (T359788)]] [production]
13:36 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depool db1179 T359790', diff saved to https://phabricator.wikimedia.org/P58703 and previous config saved to /var/cache/conftool/dbconfig/20240311-133631-arnaudb.json [production]
13:36 <Dreamy_Jazz> Running `foreachwikiindblist group2.dblist extensions/MediaModeration/maintenance/scanFilesInScanTable.php --wiki=commonswiki --use-jobqueue --sleep 30 --verbose 2>&1 | tee ~/scan-files-in-scan-table-group2-sleep-30-no-render-now.txt` on a tmux session [production]
13:35 <samtar@deploy2002> Finished scap: Backport for [[gerrit:1009731|[itwiki] Set 'wgBlockAllowsUTEdit' to true]] (duration: 13m 13s) [production]
13:34 <arnaudb@cumin1002> dbctl commit (dc=all): 'Promote db1220 to x1 primary T359790', diff saved to https://phabricator.wikimedia.org/P58702 and previous config saved to /var/cache/conftool/dbconfig/20240311-133405-arnaudb.json [production]
13:32 <arnaudb> Starting x1 eqiad failover from db1179 to db1220 - T359790 [production]
13:32 <logmsgbot> @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
13:32 <logmsgbot> @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
13:27 <logmsgbot> @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
13:27 <logmsgbot> @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
13:25 <samtar@deploy2002> superpes and samtar: Continuing with sync [production]
13:24 <samtar@deploy2002> superpes and samtar: Backport for [[gerrit:1009731|[itwiki] Set 'wgBlockAllowsUTEdit' to true]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
13:23 <arnaudb@cumin1002> dbctl commit (dc=all): 'Set db1220 with weight 0 T359790', diff saved to https://phabricator.wikimedia.org/P58701 and previous config saved to /var/cache/conftool/dbconfig/20240311-132259-arnaudb.json [production]
13:22 <logmsgbot> @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
13:22 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 16 hosts with reason: Primary switchover x1 T359790 [production]
13:22 <logmsgbot> @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
13:22 <samtar@deploy2002> Started scap: Backport for [[gerrit:1009731|[itwiki] Set 'wgBlockAllowsUTEdit' to true]] [production]
13:22 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on 16 hosts with reason: Primary switchover x1 T359790 [production]
12:59 <dani@deploy2002> helmfile [codfw] DONE helmfile.d/services/miscweb: apply [production]
12:59 <dani@deploy2002> helmfile [codfw] START helmfile.d/services/miscweb: apply [production]
12:59 <dani@deploy2002> helmfile [eqiad] DONE helmfile.d/services/miscweb: apply [production]
12:57 <dani@deploy2002> helmfile [eqiad] START helmfile.d/services/miscweb: apply [production]
12:57 <dani@deploy2002> helmfile [staging] DONE helmfile.d/services/miscweb: apply [production]
12:56 <dani@deploy2002> helmfile [staging] START helmfile.d/services/miscweb: apply [production]
12:52 <Dreamy_Jazz> Re-starting MediaModeration scanning script [production]
12:50 <logmsgbot> @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
12:50 <logmsgbot> @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
10:32 <logmsgbot> @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]