|
2023-02-09
ยง
|
| 09:32 |
<godog> |
roll-restart opensearch-dashboards to apply memory limit - T327161 |
[production] |
| 09:29 |
<kharlan@deploy1002> |
Finished scap: Backport for [[gerrit:887849|Add StatusValue::hasMessagesExcept() (T272081)]] (duration: 09m 20s) |
[production] |
| 09:26 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1107 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43954 and previous config saved to /var/cache/conftool/dbconfig/20230209-092654-root.json |
[production] |
| 09:26 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1132 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P43953 and previous config saved to /var/cache/conftool/dbconfig/20230209-092650-root.json |
[production] |
| 09:22 |
<kharlan@deploy1002> |
kharlan: Backport for [[gerrit:887849|Add StatusValue::hasMessagesExcept() (T272081)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet |
[production] |
| 09:20 |
<kharlan@deploy1002> |
Started scap: Backport for [[gerrit:887849|Add StatusValue::hasMessagesExcept() (T272081)]] |
[production] |
| 09:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2128', diff saved to https://phabricator.wikimedia.org/P43952 and previous config saved to /var/cache/conftool/dbconfig/20230209-091742-marostegui.json |
[production] |
| 09:13 |
<moritzm> |
installing openssl security updates on Bullseye |
[production] |
| 09:11 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1107 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P43951 and previous config saved to /var/cache/conftool/dbconfig/20230209-091149-root.json |
[production] |
| 09:11 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1132 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P43950 and previous config saved to /var/cache/conftool/dbconfig/20230209-091145-root.json |
[production] |
| 09:10 |
<marostegui> |
Install 10.4.28 on db1107 T329011 |
[production] |
| 09:10 |
<marostegui> |
Install 10.6.12 on db1132 T329011 |
[production] |
| 09:09 |
<vgutierrez> |
pool cp4044 with ESI testing enabled |
[production] |
| 09:08 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1107 db1132', diff saved to https://phabricator.wikimedia.org/P43949 and previous config saved to /var/cache/conftool/dbconfig/20230209-090846-root.json |
[production] |
| 09:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2128 (T328817)', diff saved to https://phabricator.wikimedia.org/P43948 and previous config saved to /var/cache/conftool/dbconfig/20230209-090236-marostegui.json |
[production] |
| 09:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db2128 (T328817)', diff saved to https://phabricator.wikimedia.org/P43947 and previous config saved to /var/cache/conftool/dbconfig/20230209-090018-marostegui.json |
[production] |
| 09:00 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2094.codfw.wmnet with reason: Maintenance |
[production] |
| 09:00 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2094.codfw.wmnet with reason: Maintenance |
[production] |
| 09:00 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2128.codfw.wmnet with reason: Maintenance |
[production] |
| 08:59 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2128.codfw.wmnet with reason: Maintenance |
[production] |
| 08:59 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2123 (T328817)', diff saved to https://phabricator.wikimedia.org/P43946 and previous config saved to /var/cache/conftool/dbconfig/20230209-085952-marostegui.json |
[production] |
| 08:57 |
<vgutierrez> |
depool cp4044 - T308799 |
[production] |
| 08:44 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2123', diff saved to https://phabricator.wikimedia.org/P43945 and previous config saved to /var/cache/conftool/dbconfig/20230209-084446-marostegui.json |
[production] |
| 08:42 |
<apergos> |
UTC morning backport and config training window complete |
[production] |
| 08:32 |
<kartik@deploy1002> |
Finished scap: Backport for [[gerrit:887847|CX: Provide the appropriate arguments to ve.ui.CXSurface constructor (T329154)]] (duration: 13m 10s) |
[production] |
| 08:29 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2123', diff saved to https://phabricator.wikimedia.org/P43944 and previous config saved to /var/cache/conftool/dbconfig/20230209-082940-marostegui.json |
[production] |
| 08:20 |
<kartik@deploy1002> |
kartik: Backport for [[gerrit:887847|CX: Provide the appropriate arguments to ve.ui.CXSurface constructor (T329154)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet |
[production] |
| 08:19 |
<kartik@deploy1002> |
Started scap: Backport for [[gerrit:887847|CX: Provide the appropriate arguments to ve.ui.CXSurface constructor (T329154)]] |
[production] |
| 08:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2123 (T328817)', diff saved to https://phabricator.wikimedia.org/P43943 and previous config saved to /var/cache/conftool/dbconfig/20230209-081433-marostegui.json |
[production] |
| 08:11 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db2123 (T328817)', diff saved to https://phabricator.wikimedia.org/P43942 and previous config saved to /var/cache/conftool/dbconfig/20230209-081116-marostegui.json |
[production] |
| 08:11 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2123.codfw.wmnet with reason: Maintenance |
[production] |
| 08:10 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2123.codfw.wmnet with reason: Maintenance |
[production] |
| 08:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2111 (T328817)', diff saved to https://phabricator.wikimedia.org/P43941 and previous config saved to /var/cache/conftool/dbconfig/20230209-081054-marostegui.json |
[production] |
| 07:55 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2111', diff saved to https://phabricator.wikimedia.org/P43940 and previous config saved to /var/cache/conftool/dbconfig/20230209-075548-marostegui.json |
[production] |
| 07:40 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2111', diff saved to https://phabricator.wikimedia.org/P43939 and previous config saved to /var/cache/conftool/dbconfig/20230209-074042-marostegui.json |
[production] |
| 07:25 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2111 (T328817)', diff saved to https://phabricator.wikimedia.org/P43938 and previous config saved to /var/cache/conftool/dbconfig/20230209-072535-marostegui.json |
[production] |
| 07:24 |
<marostegui> |
Stop mariadb on db1117:3321 (some dbproxy irc alerts will be triggered) T329143 |
[production] |
| 07:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db2111 (T328817)', diff saved to https://phabricator.wikimedia.org/P43936 and previous config saved to /var/cache/conftool/dbconfig/20230209-072204-marostegui.json |
[production] |
| 07:21 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2111.codfw.wmnet with reason: Maintenance |
[production] |
| 07:21 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2111.codfw.wmnet with reason: Maintenance |
[production] |
| 07:19 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2101.codfw.wmnet with reason: Maintenance |
[production] |
| 07:18 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2101.codfw.wmnet with reason: Maintenance |
[production] |
| 07:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Remove db1098 (s6, s7) from dbctl T329171', diff saved to https://phabricator.wikimedia.org/P43935 and previous config saved to /var/cache/conftool/dbconfig/20230209-071013-marostegui.json |
[production] |
| 07:09 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1162.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:09 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db1162.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:04 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2107.codfw.wmnet with reason: Maintenance |
[production] |
| 07:04 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2107.codfw.wmnet with reason: Maintenance |
[production] |
| 07:00 |
<marostegui> |
Failover m3 from db1164 to db1159 - T329141 |
[production] |
| 06:48 |
<oblivian@cumin2002> |
END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) status all services in eqiad: maintenance |
[production] |
| 06:48 |
<oblivian@cumin2002> |
START - Cookbook sre.discovery.datacenter status all services in eqiad: maintenance |
[production] |