3551-3600 of 10000 results (84ms)
2024-01-23 ยง
15:37 <cgoubert@deploy2002> helmfile [codfw] START helmfile.d/services/mw-api-ext: apply [production]
15:37 <cgoubert@deploy2002> helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply [production]
15:37 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P55383 and previous config saved to /var/cache/conftool/dbconfig/20240123-153712-ladsgroup.json [production]
15:36 <cgoubert@deploy2002> helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply [production]
15:36 <claime> Bumping mw-api-ext replicas - T355532 [production]
15:36 <cgoubert@deploy2002> helmfile [codfw] DONE helmfile.d/services/mw-web: apply [production]
15:36 <cgoubert@deploy2002> helmfile [codfw] START helmfile.d/services/mw-web: apply [production]
15:35 <cgoubert@deploy2002> helmfile [eqiad] DONE helmfile.d/services/mw-web: apply [production]
15:35 <cgoubert@deploy2002> helmfile [eqiad] START helmfile.d/services/mw-web: apply [production]
15:35 <claime> Bumping mw-web replicas - T355532 [production]
15:33 <logmsgbot> lucaswerkmeister-wmde@deploy2002 helmfile [codfw] DONE helmfile.d/services/termbox: apply [production]
15:32 <logmsgbot> lucaswerkmeister-wmde@deploy2002 helmfile [codfw] START helmfile.d/services/termbox: apply [production]
15:32 <logmsgbot> lucaswerkmeister-wmde@deploy2002 helmfile [eqiad] DONE helmfile.d/services/termbox: apply [production]
15:31 <logmsgbot> lucaswerkmeister-wmde@deploy2002 helmfile [eqiad] START helmfile.d/services/termbox: apply [production]
15:31 <logmsgbot> lucaswerkmeister-wmde@deploy2002 helmfile [staging] DONE helmfile.d/services/termbox: apply [production]
15:31 <logmsgbot> lucaswerkmeister-wmde@deploy2002 helmfile [staging] START helmfile.d/services/termbox: apply [production]
15:29 <marostegui@cumin1002> dbctl commit (dc=all): 'db1173 (re)pooling @ 25%: Upgrade to 10.6.16 and bookworm', diff saved to https://phabricator.wikimedia.org/P55382 and previous config saved to /var/cache/conftool/dbconfig/20240123-152911-root.json [production]
15:22 <logmsgbot> lucaswerkmeister-wmde@deploy2002 helmfile [codfw] DONE helmfile.d/services/termbox: apply [production]
15:22 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P55381 and previous config saved to /var/cache/conftool/dbconfig/20240123-152206-ladsgroup.json [production]
15:21 <logmsgbot> lucaswerkmeister-wmde@deploy2002 helmfile [codfw] START helmfile.d/services/termbox: apply [production]
15:21 <logmsgbot> lucaswerkmeister-wmde@deploy2002 helmfile [eqiad] DONE helmfile.d/services/termbox: apply [production]
15:20 <logmsgbot> lucaswerkmeister-wmde@deploy2002 helmfile [eqiad] START helmfile.d/services/termbox: apply [production]
15:20 <logmsgbot> lucaswerkmeister-wmde@deploy2002 helmfile [staging] DONE helmfile.d/services/termbox: apply [production]
15:19 <logmsgbot> lucaswerkmeister-wmde@deploy2002 helmfile [staging] START helmfile.d/services/termbox: apply [production]
15:14 <marostegui@cumin1002> dbctl commit (dc=all): 'db1173 (re)pooling @ 10%: Upgrade to 10.6.16 and bookworm', diff saved to https://phabricator.wikimedia.org/P55380 and previous config saved to /var/cache/conftool/dbconfig/20240123-151406-root.json [production]
15:13 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2165.codfw.wmnet with reason: Maintenance [production]
15:13 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 8:00:00 on db2165.codfw.wmnet with reason: Maintenance [production]
15:08 <logmsgbot> lucaswerkmeister-wmde@deploy2002 helmfile [codfw] DONE helmfile.d/services/termbox: apply [production]
15:08 <logmsgbot> lucaswerkmeister-wmde@deploy2002 helmfile [codfw] START helmfile.d/services/termbox: apply [production]
15:08 <logmsgbot> lucaswerkmeister-wmde@deploy2002 helmfile [eqiad] DONE helmfile.d/services/termbox: apply [production]
15:08 <logmsgbot> lucaswerkmeister-wmde@deploy2002 helmfile [eqiad] START helmfile.d/services/termbox: apply [production]
15:07 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2155 (T352010)', diff saved to https://phabricator.wikimedia.org/P55379 and previous config saved to /var/cache/conftool/dbconfig/20240123-150659-ladsgroup.json [production]
15:06 <logmsgbot> lucaswerkmeister-wmde@deploy2002 helmfile [staging] DONE helmfile.d/services/termbox: apply [production]
15:05 <logmsgbot> lucaswerkmeister-wmde@deploy2002 helmfile [staging] START helmfile.d/services/termbox: apply [production]
15:00 <Lucas_WMDE> UTC afternoon backport+config window done [production]
14:59 <logmsgbot> lucaswerkmeister-wmde@deploy2002 Finished scap: Backport for [[gerrit:992423|ORES: Enable renamed revertrisklanguageagnostic model (T348298)]] (duration: 11m 20s) [production]
14:59 <marostegui@cumin1002> dbctl commit (dc=all): 'db1173 (re)pooling @ 5%: Upgrade to 10.6.16 and bookworm', diff saved to https://phabricator.wikimedia.org/P55378 and previous config saved to /var/cache/conftool/dbconfig/20240123-145901-root.json [production]
14:53 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2195 (T354336)', diff saved to https://phabricator.wikimedia.org/P55377 and previous config saved to /var/cache/conftool/dbconfig/20240123-145353-marostegui.json [production]
14:53 <logmsgbot> lucaswerkmeister-wmde@deploy2002 lucaswerkmeister-wmde and kharlan: Continuing with sync [production]
14:49 <logmsgbot> lucaswerkmeister-wmde@deploy2002 lucaswerkmeister-wmde and kharlan: Backport for [[gerrit:992423|ORES: Enable renamed revertrisklanguageagnostic model (T348298)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
14:47 <logmsgbot> lucaswerkmeister-wmde@deploy2002 Started scap: Backport for [[gerrit:992423|ORES: Enable renamed revertrisklanguageagnostic model (T348298)]] [production]
14:47 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1173.eqiad.wmnet with OS bookworm [production]
14:43 <marostegui@cumin1002> dbctl commit (dc=all): 'db1173 (re)pooling @ 1%: Upgrade to 10.6.16 and bookworm', diff saved to https://phabricator.wikimedia.org/P55376 and previous config saved to /var/cache/conftool/dbconfig/20240123-144356-root.json [production]
14:42 <logmsgbot> lucaswerkmeister-wmde@deploy2002 Finished scap: Backport for [[gerrit:992376|Restore support for matching 'LIKE' patterns/wildcards (T355478)]] (duration: 07m 50s) [production]
14:38 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P55375 and previous config saved to /var/cache/conftool/dbconfig/20240123-143846-marostegui.json [production]
14:36 <logmsgbot> lucaswerkmeister-wmde@deploy2002 matmarex and lucaswerkmeister-wmde: Continuing with sync [production]
14:36 <logmsgbot> lucaswerkmeister-wmde@deploy2002 matmarex and lucaswerkmeister-wmde: Backport for [[gerrit:992376|Restore support for matching 'LIKE' patterns/wildcards (T355478)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
14:34 <logmsgbot> lucaswerkmeister-wmde@deploy2002 Started scap: Backport for [[gerrit:992376|Restore support for matching 'LIKE' patterns/wildcards (T355478)]] [production]
14:33 <logmsgbot> lucaswerkmeister-wmde@deploy2002 Finished scap: Backport for [[gerrit:992377|Restore support for matching 'LIKE' patterns/wildcards (T355478)]] (duration: 10m 29s) [production]
14:32 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts sretest1003.eqiad.wmnet [production]