801-850 of 10000 results (87ms)
2024-04-17 ยง
10:35 <jiji@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply [production]
10:35 <jiji@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-api-int: apply [production]
10:35 <akosiaris@deploy1002> helmfile [codfw] DONE helmfile.d/admin 'apply'. [production]
10:35 <akosiaris@deploy1002> helmfile [codfw] START helmfile.d/admin 'apply'. [production]
10:34 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2105 (T361627)', diff saved to https://phabricator.wikimedia.org/P60756 and previous config saved to /var/cache/conftool/dbconfig/20240417-103455-marostegui.json [production]
10:34 <akosiaris> apply the coredns patches for bumping instances from 4 to 6. They are noop, I am applying them to update helm's state. [production]
10:34 <jiji@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply [production]
10:34 <akosiaris@deploy1002> helmfile [eqiad] DONE helmfile.d/admin 'apply'. [production]
10:34 <akosiaris@deploy1002> helmfile [eqiad] START helmfile.d/admin 'apply'. [production]
10:34 <jmm@cumin2002> START - Cookbook sre.puppet.migrate-host for host es1027.eqiad.wmnet [production]
10:33 <jiji@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply [production]
10:33 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es2028.codfw.wmnet [production]
10:22 <jmm@cumin2002> START - Cookbook sre.puppet.migrate-host for host es2028.codfw.wmnet [production]
10:14 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db2105 (T361627)', diff saved to https://phabricator.wikimedia.org/P60755 and previous config saved to /var/cache/conftool/dbconfig/20240417-101446-marostegui.json [production]
10:14 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2105.codfw.wmnet with reason: Maintenance [production]
10:14 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 6:00:00 on db2105.codfw.wmnet with reason: Maintenance [production]
10:08 <hnowlan@deploy1002> helmfile [eqiad] DONE helmfile.d/services/wikifeeds: sync [production]
10:08 <hnowlan@deploy1002> helmfile [eqiad] START helmfile.d/services/wikifeeds: sync [production]
10:06 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance [production]
10:06 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance [production]
10:00 <akosiaris> manually bump coredns in eqiad to 6 [production]
09:59 <akosiaris> manually bump coredns in codfw to 6 [production]
09:57 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1240.eqiad.wmnet with reason: Maintenance [production]
09:57 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 6:00:00 on db1240.eqiad.wmnet with reason: Maintenance [production]
09:57 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1223 (T361627)', diff saved to https://phabricator.wikimedia.org/P60753 and previous config saved to /var/cache/conftool/dbconfig/20240417-095731-marostegui.json [production]
09:44 <cgoubert@cumin1002> conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad [production]
09:44 <cgoubert@cumin1002> conftool action : set/pooled=false; selector: dnsdisc=mw-api-int-ro,name=eqiad [production]
09:44 <cgoubert@cumin1002> conftool action : set/pooled=false; selector: dnsdisc=mw-web-ro,name=eqiad [production]
09:42 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P60750 and previous config saved to /var/cache/conftool/dbconfig/20240417-094223-marostegui.json [production]
09:31 <jiji@deploy1002> scap failed: KeyError 'production' (duration: 22m 21s) [production]
09:29 <marostegui@cumin1002> dbctl commit (dc=all): 'db2150 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P60749 and previous config saved to /var/cache/conftool/dbconfig/20240417-092923-root.json [production]
09:27 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P60748 and previous config saved to /var/cache/conftool/dbconfig/20240417-092714-marostegui.json [production]
09:14 <marostegui@cumin1002> dbctl commit (dc=all): 'db2150 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P60747 and previous config saved to /var/cache/conftool/dbconfig/20240417-091418-root.json [production]
09:12 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1223 (T361627)', diff saved to https://phabricator.wikimedia.org/P60746 and previous config saved to /var/cache/conftool/dbconfig/20240417-091203-marostegui.json [production]
09:08 <jiji@deploy1002> Started scap: Switch mediawiki in eqiad to use node-local mcrouter ds - T346690 [production]
09:05 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db1223 (T361627)', diff saved to https://phabricator.wikimedia.org/P60745 and previous config saved to /var/cache/conftool/dbconfig/20240417-090539-marostegui.json [production]
09:05 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1223.eqiad.wmnet with reason: Maintenance [production]
09:05 <jiji@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply [production]
09:05 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 6:00:00 on db1223.eqiad.wmnet with reason: Maintenance [production]
09:05 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1212 (T361627)', diff saved to https://phabricator.wikimedia.org/P60744 and previous config saved to /var/cache/conftool/dbconfig/20240417-090516-marostegui.json [production]
09:03 <jiji@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-api-int: apply [production]
08:59 <marostegui@cumin1002> dbctl commit (dc=all): 'db2150 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P60743 and previous config saved to /var/cache/conftool/dbconfig/20240417-085912-root.json [production]
08:57 <hashar@deploy1002> Finished scap: Backport for [[gerrit:1019267|logging: pluralize $wmgDefaultMonologHandler (T238838)]] (duration: 16m 37s) [production]
08:50 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P60742 and previous config saved to /var/cache/conftool/dbconfig/20240417-085009-marostegui.json [production]
08:44 <hashar@deploy1002> hashar: Continuing with sync [production]
08:44 <hashar@deploy1002> hashar: Backport for [[gerrit:1019267|logging: pluralize $wmgDefaultMonologHandler (T238838)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
08:44 <marostegui@cumin1002> dbctl commit (dc=all): 'db2150 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P60741 and previous config saved to /var/cache/conftool/dbconfig/20240417-084407-root.json [production]
08:41 <hashar@deploy1002> Started scap: Backport for [[gerrit:1019267|logging: pluralize $wmgDefaultMonologHandler (T238838)]] [production]
08:40 <aqu> Deployed refinery using scap, then deployed onto hdfs [production]
08:35 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P60739 and previous config saved to /var/cache/conftool/dbconfig/20240417-083501-marostegui.json [production]