651-700 of 10000 results (95ms)
2024-04-17 ยง
15:27 <sukhe@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on cp1114.eqiad.wmnet with reason: host reimage [production]
15:20 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2156 (T361627)', diff saved to https://phabricator.wikimedia.org/P60794 and previous config saved to /var/cache/conftool/dbconfig/20240417-152023-marostegui.json [production]
15:18 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2120 depool T358741', diff saved to https://phabricator.wikimedia.org/P60793 and previous config saved to /var/cache/conftool/dbconfig/20240417-151811-arnaudb.json [production]
15:17 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db2127 (T360332)', diff saved to https://phabricator.wikimedia.org/P60792 and previous config saved to /var/cache/conftool/dbconfig/20240417-151653-arnaudb.json [production]
15:16 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2127.codfw.wmnet with reason: Maintenance [production]
15:16 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 12:00:00 on db2127.codfw.wmnet with reason: Maintenance [production]
15:13 <elukey@deploy1002> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. [production]
15:12 <elukey@deploy1002> helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. [production]
15:09 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.dhcp (exit_code=99) for host cp1115.eqiad.wmnet [production]
15:07 <sukhe@cumin1002> START - Cookbook sre.hosts.reimage for host cp1114.eqiad.wmnet with OS bullseye [production]
15:06 <vgutierrez> repool ncredir2001 [production]
15:05 <Lucas_WMDE> UTC afternoon backport+config window (belatedly) done [production]
15:04 <Lucas_WMDE> lucaswerkmeister-wmde@mwmaint1002:~$ mwscript namespaceDupes mlwiki --fix # T362653: 0 pages to fix, 0 were resolvable; 82 links to fix, 82 were resolvable, 0 were deleted. [production]
15:03 <logmsgbot> lucaswerkmeister-wmde@deploy1002 Finished scap: Backport for [[gerrit:1020242|mlwiki: create draft namespace (T362653)]] (duration: 32m 43s) [production]
14:59 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db2156 (T361627)', diff saved to https://phabricator.wikimedia.org/P60790 and previous config saved to /var/cache/conftool/dbconfig/20240417-145916-marostegui.json [production]
14:59 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2186.codfw.wmnet with reason: Maintenance [production]
14:58 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 12:00:00 on db2186.codfw.wmnet with reason: Maintenance [production]
14:58 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2156.codfw.wmnet with reason: Maintenance [production]
14:58 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 6:00:00 on db2156.codfw.wmnet with reason: Maintenance [production]
14:58 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2149 (T361627)', diff saved to https://phabricator.wikimedia.org/P60789 and previous config saved to /var/cache/conftool/dbconfig/20240417-145838-marostegui.json [production]
14:51 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db1218 (T352010)', diff saved to https://phabricator.wikimedia.org/P60788 and previous config saved to /var/cache/conftool/dbconfig/20240417-145136-ladsgroup.json [production]
14:51 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1218.eqiad.wmnet with reason: Maintenance [production]
14:51 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1218.eqiad.wmnet with reason: Maintenance [production]
14:51 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1207 (T352010)', diff saved to https://phabricator.wikimedia.org/P60787 and previous config saved to /var/cache/conftool/dbconfig/20240417-145113-ladsgroup.json [production]
14:50 <logmsgbot> lucaswerkmeister-wmde@deploy1002 anzx and lucaswerkmeister-wmde: Continuing with sync [production]
14:44 <elukey@deploy1002> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. [production]
14:44 <elukey@deploy1002> helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. [production]
14:43 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P60786 and previous config saved to /var/cache/conftool/dbconfig/20240417-144330-marostegui.json [production]
14:36 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P60785 and previous config saved to /var/cache/conftool/dbconfig/20240417-143606-ladsgroup.json [production]
14:34 <logmsgbot> lucaswerkmeister-wmde@deploy1002 anzx and lucaswerkmeister-wmde: Backport for [[gerrit:1020242|mlwiki: create draft namespace (T362653)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
14:33 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.migrate-role (exit_code=0) for role: insetup::data_persistence [production]
14:31 <marostegui@cumin1002> dbctl commit (dc=all): 'db2120 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P60784 and previous config saved to /var/cache/conftool/dbconfig/20240417-143103-root.json [production]
14:31 <logmsgbot> lucaswerkmeister-wmde@deploy1002 Started scap: Backport for [[gerrit:1020242|mlwiki: create draft namespace (T362653)]] [production]
14:28 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P60783 and previous config saved to /var/cache/conftool/dbconfig/20240417-142823-marostegui.json [production]
14:22 <jforrester@deploy1002> helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply [production]
14:22 <sukhe@cumin1002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts cp1114.eqiad.wmnet [production]
14:21 <sukhe@cumin1002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp1114.eqiad.wmnet [production]
14:21 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P60782 and previous config saved to /var/cache/conftool/dbconfig/20240417-142057-ladsgroup.json [production]
14:20 <jforrester@deploy1002> helmfile [eqiad] START helmfile.d/services/wikifunctions: apply [production]
14:20 <jforrester@deploy1002> helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply [production]
14:20 <sukhe@puppetmaster1001> conftool action : set/pooled=no; selector: name=cp1114.eqiad.wmnet,service=(cdn|ats-be) [production]
14:20 <sukhe> depool cp1114.eqiad.wmnet for PXE boot testing issues and downgrade NIC firmware: T350179 [production]
14:19 <jforrester@deploy1002> helmfile [codfw] START helmfile.d/services/wikifunctions: apply [production]
14:19 <jforrester@deploy1002> helmfile [staging] DONE helmfile.d/services/wikifunctions: apply [production]
14:18 <jforrester@deploy1002> helmfile [staging] START helmfile.d/services/wikifunctions: apply [production]
14:15 <marostegui@cumin1002> dbctl commit (dc=all): 'db2120 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P60781 and previous config saved to /var/cache/conftool/dbconfig/20240417-141557-root.json [production]
14:15 <jforrester@deploy1002> helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply [production]
14:13 <jforrester@deploy1002> helmfile [eqiad] START helmfile.d/services/wikifunctions: apply [production]
14:13 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2149 (T361627)', diff saved to https://phabricator.wikimedia.org/P60780 and previous config saved to /var/cache/conftool/dbconfig/20240417-141314-marostegui.json [production]
14:13 <jforrester@deploy1002> helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply [production]