2301-2350 of 10000 results (75ms)
2023-02-09 ยง
09:47 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2128 (T328817)', diff saved to https://phabricator.wikimedia.org/P43958 and previous config saved to /var/cache/conftool/dbconfig/20230209-094755-marostegui.json [production]
09:42 <marostegui@cumin1001> dbctl commit (dc=all): 'db1107 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43957 and previous config saved to /var/cache/conftool/dbconfig/20230209-094159-root.json [production]
09:41 <marostegui@cumin1001> dbctl commit (dc=all): 'db1132 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P43956 and previous config saved to /var/cache/conftool/dbconfig/20230209-094154-root.json [production]
09:32 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2128', diff saved to https://phabricator.wikimedia.org/P43955 and previous config saved to /var/cache/conftool/dbconfig/20230209-093248-marostegui.json [production]
09:32 <godog> roll-restart opensearch-dashboards to apply memory limit - T327161 [production]
09:29 <kharlan@deploy1002> Finished scap: Backport for [[gerrit:887849|Add StatusValue::hasMessagesExcept() (T272081)]] (duration: 09m 20s) [production]
09:26 <marostegui@cumin1001> dbctl commit (dc=all): 'db1107 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43954 and previous config saved to /var/cache/conftool/dbconfig/20230209-092654-root.json [production]
09:26 <marostegui@cumin1001> dbctl commit (dc=all): 'db1132 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43953 and previous config saved to /var/cache/conftool/dbconfig/20230209-092650-root.json [production]
09:22 <kharlan@deploy1002> kharlan: Backport for [[gerrit:887849|Add StatusValue::hasMessagesExcept() (T272081)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet [production]
09:20 <kharlan@deploy1002> Started scap: Backport for [[gerrit:887849|Add StatusValue::hasMessagesExcept() (T272081)]] [production]
09:17 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2128', diff saved to https://phabricator.wikimedia.org/P43952 and previous config saved to /var/cache/conftool/dbconfig/20230209-091742-marostegui.json [production]
09:13 <moritzm> installing openssl security updates on Bullseye [production]
09:11 <marostegui@cumin1001> dbctl commit (dc=all): 'db1107 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P43951 and previous config saved to /var/cache/conftool/dbconfig/20230209-091149-root.json [production]
09:11 <marostegui@cumin1001> dbctl commit (dc=all): 'db1132 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P43950 and previous config saved to /var/cache/conftool/dbconfig/20230209-091145-root.json [production]
09:10 <marostegui> Install 10.4.28 on db1107 T329011 [production]
09:10 <marostegui> Install 10.6.12 on db1132 T329011 [production]
09:09 <vgutierrez> pool cp4044 with ESI testing enabled [production]
09:08 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1107 db1132', diff saved to https://phabricator.wikimedia.org/P43949 and previous config saved to /var/cache/conftool/dbconfig/20230209-090846-root.json [production]
09:02 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2128 (T328817)', diff saved to https://phabricator.wikimedia.org/P43948 and previous config saved to /var/cache/conftool/dbconfig/20230209-090236-marostegui.json [production]
09:00 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db2128 (T328817)', diff saved to https://phabricator.wikimedia.org/P43947 and previous config saved to /var/cache/conftool/dbconfig/20230209-090018-marostegui.json [production]
09:00 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2094.codfw.wmnet with reason: Maintenance [production]
09:00 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2094.codfw.wmnet with reason: Maintenance [production]
09:00 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2128.codfw.wmnet with reason: Maintenance [production]
08:59 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2128.codfw.wmnet with reason: Maintenance [production]
08:59 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2123 (T328817)', diff saved to https://phabricator.wikimedia.org/P43946 and previous config saved to /var/cache/conftool/dbconfig/20230209-085952-marostegui.json [production]
08:57 <vgutierrez> depool cp4044 - T308799 [production]
08:44 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2123', diff saved to https://phabricator.wikimedia.org/P43945 and previous config saved to /var/cache/conftool/dbconfig/20230209-084446-marostegui.json [production]
08:42 <apergos> UTC morning backport and config training window complete [production]
08:32 <kartik@deploy1002> Finished scap: Backport for [[gerrit:887847|CX: Provide the appropriate arguments to ve.ui.CXSurface constructor (T329154)]] (duration: 13m 10s) [production]
08:29 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2123', diff saved to https://phabricator.wikimedia.org/P43944 and previous config saved to /var/cache/conftool/dbconfig/20230209-082940-marostegui.json [production]
08:20 <kartik@deploy1002> kartik: Backport for [[gerrit:887847|CX: Provide the appropriate arguments to ve.ui.CXSurface constructor (T329154)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet [production]
08:19 <kartik@deploy1002> Started scap: Backport for [[gerrit:887847|CX: Provide the appropriate arguments to ve.ui.CXSurface constructor (T329154)]] [production]
08:14 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2123 (T328817)', diff saved to https://phabricator.wikimedia.org/P43943 and previous config saved to /var/cache/conftool/dbconfig/20230209-081433-marostegui.json [production]
08:11 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db2123 (T328817)', diff saved to https://phabricator.wikimedia.org/P43942 and previous config saved to /var/cache/conftool/dbconfig/20230209-081116-marostegui.json [production]
08:11 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2123.codfw.wmnet with reason: Maintenance [production]
08:10 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2123.codfw.wmnet with reason: Maintenance [production]
08:10 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2111 (T328817)', diff saved to https://phabricator.wikimedia.org/P43941 and previous config saved to /var/cache/conftool/dbconfig/20230209-081054-marostegui.json [production]
07:55 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2111', diff saved to https://phabricator.wikimedia.org/P43940 and previous config saved to /var/cache/conftool/dbconfig/20230209-075548-marostegui.json [production]
07:40 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2111', diff saved to https://phabricator.wikimedia.org/P43939 and previous config saved to /var/cache/conftool/dbconfig/20230209-074042-marostegui.json [production]
07:25 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2111 (T328817)', diff saved to https://phabricator.wikimedia.org/P43938 and previous config saved to /var/cache/conftool/dbconfig/20230209-072535-marostegui.json [production]
07:24 <marostegui> Stop mariadb on db1117:3321 (some dbproxy irc alerts will be triggered) T329143 [production]
07:22 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db2111 (T328817)', diff saved to https://phabricator.wikimedia.org/P43936 and previous config saved to /var/cache/conftool/dbconfig/20230209-072204-marostegui.json [production]
07:21 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2111.codfw.wmnet with reason: Maintenance [production]
07:21 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2111.codfw.wmnet with reason: Maintenance [production]
07:19 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2101.codfw.wmnet with reason: Maintenance [production]
07:18 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2101.codfw.wmnet with reason: Maintenance [production]
07:10 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove db1098 (s6, s7) from dbctl T329171', diff saved to https://phabricator.wikimedia.org/P43935 and previous config saved to /var/cache/conftool/dbconfig/20230209-071013-marostegui.json [production]
07:09 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1162.eqiad.wmnet with reason: Maintenance [production]
07:09 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1162.eqiad.wmnet with reason: Maintenance [production]
07:04 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2107.codfw.wmnet with reason: Maintenance [production]