|
2023-01-05
§
|
| 11:13 |
<cgoubert@deploy1002> |
helmfile [codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 11:13 |
<cgoubert@deploy1002> |
helmfile [codfw] START helmfile.d/admin 'apply'. |
[production] |
| 11:13 |
<cgoubert@deploy1002> |
helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 11:13 |
<cgoubert@deploy1002> |
helmfile [staging-eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 11:12 |
<cgoubert@deploy1002> |
helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 11:12 |
<cgoubert@deploy1002> |
helmfile [staging-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 10:58 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1134 (re)pooling @ 100%: After cloning db1176', diff saved to https://phabricator.wikimedia.org/P42850 and previous config saved to /var/cache/conftool/dbconfig/20230105-105808-root.json |
[production] |
| 10:43 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1134 (re)pooling @ 75%: After cloning db1176', diff saved to https://phabricator.wikimedia.org/P42849 and previous config saved to /var/cache/conftool/dbconfig/20230105-104303-root.json |
[production] |
| 10:27 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1134 (re)pooling @ 50%: After cloning db1176', diff saved to https://phabricator.wikimedia.org/P42848 and previous config saved to /var/cache/conftool/dbconfig/20230105-102758-root.json |
[production] |
| 10:26 |
<cgoubert@cumin1001> |
START - Cookbook sre.hosts.reboot-cluster |
[production] |
| 10:26 |
<claime> |
Rolling reboot of api_appserver hosts in eqiad |
[production] |
| 10:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2151 (re)pooling @ 100%: Pooling in s6', diff saved to https://phabricator.wikimedia.org/P42847 and previous config saved to /var/cache/conftool/dbconfig/20230105-102357-root.json |
[production] |
| 10:22 |
<cgoubert@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-cluster (exit_code=0) |
[production] |
| 10:12 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1134 (re)pooling @ 25%: After cloning db1176', diff saved to https://phabricator.wikimedia.org/P42846 and previous config saved to /var/cache/conftool/dbconfig/20230105-101253-root.json |
[production] |
| 10:08 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2151 (re)pooling @ 75%: Pooling in s6', diff saved to https://phabricator.wikimedia.org/P42845 and previous config saved to /var/cache/conftool/dbconfig/20230105-100852-root.json |
[production] |
| 10:07 |
<cgoubert@cumin1001> |
START - Cookbook sre.hosts.reboot-cluster |
[production] |
| 10:06 |
<claime> |
Restarting rolling reboot of api_appserver hosts in codfw |
[production] |
| 09:57 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1134 (re)pooling @ 10%: After cloning db1176', diff saved to https://phabricator.wikimedia.org/P42844 and previous config saved to /var/cache/conftool/dbconfig/20230105-095748-root.json |
[production] |
| 09:53 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2151 (re)pooling @ 60%: Pooling in s6', diff saved to https://phabricator.wikimedia.org/P42843 and previous config saved to /var/cache/conftool/dbconfig/20230105-095347-root.json |
[production] |
| 09:42 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1134 (re)pooling @ 5%: After cloning db1176', diff saved to https://phabricator.wikimedia.org/P42841 and previous config saved to /var/cache/conftool/dbconfig/20230105-094243-root.json |
[production] |
| 09:38 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2151 (re)pooling @ 50%: Pooling in s6', diff saved to https://phabricator.wikimedia.org/P42840 and previous config saved to /var/cache/conftool/dbconfig/20230105-093842-root.json |
[production] |
| 09:27 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1134 (re)pooling @ 1%: After cloning db1176', diff saved to https://phabricator.wikimedia.org/P42839 and previous config saved to /var/cache/conftool/dbconfig/20230105-092738-root.json |
[production] |
| 09:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2151 (re)pooling @ 40%: Pooling in s6', diff saved to https://phabricator.wikimedia.org/P42838 and previous config saved to /var/cache/conftool/dbconfig/20230105-092336-root.json |
[production] |
| 09:14 |
<XioNoX> |
turn up BGP to NTT in drmrs - T314929 |
[production] |
| 09:08 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2151 (re)pooling @ 25%: Pooling in s6', diff saved to https://phabricator.wikimedia.org/P42837 and previous config saved to /var/cache/conftool/dbconfig/20230105-090831-root.json |
[production] |
| 08:56 |
<hashar@deploy1002> |
Finished scap: Backport for [[gerrit:830877|[SearchVue] Enable extension on ptwiki, ruwiki & idwiki (T310367)]] (duration: 11m 38s) |
[production] |
| 08:46 |
<hashar@deploy1002> |
hashar and mlitn: Backport for [[gerrit:830877|[SearchVue] Enable extension on ptwiki, ruwiki & idwiki (T310367)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet |
[production] |
| 08:44 |
<hashar@deploy1002> |
Started scap: Backport for [[gerrit:830877|[SearchVue] Enable extension on ptwiki, ruwiki & idwiki (T310367)]] |
[production] |
| 07:58 |
<moritzm> |
installing glibc security updates on bullseye |
[production] |
| 07:50 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'More weight to db2151 in s6 T326206', diff saved to https://phabricator.wikimedia.org/P42836 and previous config saved to /var/cache/conftool/dbconfig/20230105-075046-marostegui.json |
[production] |
| 07:28 |
<oblivian@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 07:27 |
<oblivian@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mw-debug: apply |
[production] |
| 07:26 |
<oblivian@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 07:25 |
<oblivian@deploy1002> |
helmfile [codfw] START helmfile.d/services/mw-debug: apply |
[production] |
| 06:41 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1134 to clone db1176 T326211', diff saved to https://phabricator.wikimedia.org/P42833 and previous config saved to /var/cache/conftool/dbconfig/20230105-064153-marostegui.json |
[production] |
| 06:39 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Pool db2151 for the first time in s6 T326206', diff saved to https://phabricator.wikimedia.org/P42832 and previous config saved to /var/cache/conftool/dbconfig/20230105-063937-marostegui.json |
[production] |
| 06:31 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1157.eqiad.wmnet with reason: Maintenance |
[production] |
| 06:31 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 4:00:00 on db1157.eqiad.wmnet with reason: Maintenance |
[production] |
|
2023-01-04
§
|
| 23:01 |
<mutante> |
deploy2002 - re-arming keyholder T324014 |
[production] |
| 23:00 |
<mutante> |
deploy1002 - re-arming keyholder T324014 |
[production] |
| 22:35 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance |
[production] |
| 22:35 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 4:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance |
[production] |
| 22:35 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1198 (T326011)', diff saved to https://phabricator.wikimedia.org/P42831 and previous config saved to /var/cache/conftool/dbconfig/20230104-223545-marostegui.json |
[production] |
| 22:27 |
<kindrobot> |
finished UTC late backport window |
[production] |
| 22:27 |
<kindrobot@deploy1002> |
Finished scap: Backport for [[gerrit:875371|Fix underlinkedness rescore logic (T301096)]], [[gerrit:875372|Fix underlinkedness rescore logic (T301096)]] (duration: 15m 20s) |
[production] |
| 22:20 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P42828 and previous config saved to /var/cache/conftool/dbconfig/20230104-222038-marostegui.json |
[production] |
| 22:13 |
<kindrobot@deploy1002> |
kindrobot and tgr: Backport for [[gerrit:875371|Fix underlinkedness rescore logic (T301096)]], [[gerrit:875372|Fix underlinkedness rescore logic (T301096)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet |
[production] |
| 22:11 |
<kindrobot@deploy1002> |
Started scap: Backport for [[gerrit:875371|Fix underlinkedness rescore logic (T301096)]], [[gerrit:875372|Fix underlinkedness rescore logic (T301096)]] |
[production] |
| 22:05 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P42827 and previous config saved to /var/cache/conftool/dbconfig/20230104-220532-marostegui.json |
[production] |
| 21:51 |
<kindrobot@deploy1002> |
backport aborted: (duration: 02m 12s) |
[production] |