|
2022-11-03
ยง
|
| 19:48 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1185.eqiad.wmnet with reason: Maintenance |
[production] |
| 19:48 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1161 (T318605)', diff saved to https://phabricator.wikimedia.org/P38091 and previous config saved to /var/cache/conftool/dbconfig/20221103-194818-ladsgroup.json |
[production] |
| 19:40 |
<brennen@deploy1002> |
Finished deploy [phabricator/deployment@ea0ffa7]: initial deploy to phab1004 (duration: 02m 24s) |
[production] |
| 19:37 |
<brennen@deploy1002> |
Started deploy [phabricator/deployment@ea0ffa7]: initial deploy to phab1004 |
[production] |
| 19:33 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P38090 and previous config saved to /var/cache/conftool/dbconfig/20221103-193311-ladsgroup.json |
[production] |
| 19:29 |
<andrew@deploy1002> |
Finished deploy [horizon/deploy@9d02cd6]: (no justification provided) (duration: 00m 48s) |
[production] |
| 19:28 |
<andrew@deploy1002> |
Started deploy [horizon/deploy@9d02cd6]: (no justification provided) |
[production] |
| 19:26 |
<andrew@deploy1002> |
Finished deploy [horizon/deploy@9d02cd6]: (no justification provided) (duration: 00m 48s) |
[production] |
| 19:25 |
<andrew@deploy1002> |
Started deploy [horizon/deploy@9d02cd6]: (no justification provided) |
[production] |
| 19:22 |
<brennen@deploy1002> |
Finished deploy [phabricator/deployment@ea0ffa7]: initial deploy to phab1004 (duration: 00m 25s) |
[production] |
| 19:22 |
<brennen@deploy1002> |
Started deploy [phabricator/deployment@ea0ffa7]: initial deploy to phab1004 |
[production] |
| 19:18 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P38089 and previous config saved to /var/cache/conftool/dbconfig/20221103-191805-ladsgroup.json |
[production] |
| 19:06 |
<andrew@deploy1002> |
Finished deploy [horizon/deploy@9d02cd6]: (no justification provided) (duration: 01m 10s) |
[production] |
| 19:05 |
<andrew@deploy1002> |
Started deploy [horizon/deploy@9d02cd6]: (no justification provided) |
[production] |
| 19:03 |
<ladsgroup@deploy1002> |
Finished scap: Backport for [[gerrit:852883|WikiExporter: Avoid calling reload in processing every row (T298485 T322360)]] (duration: 04m 24s) |
[production] |
| 19:03 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 19:03 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mw-debug: apply |
[production] |
| 19:03 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 19:02 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1161 (T318605)', diff saved to https://phabricator.wikimedia.org/P38088 and previous config saved to /var/cache/conftool/dbconfig/20221103-190258-ladsgroup.json |
[production] |
| 19:02 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mw-debug: apply |
[production] |
| 18:59 |
<ladsgroup@deploy1002> |
ladsgroup and ladsgroup: Backport for [[gerrit:852883|WikiExporter: Avoid calling reload in processing every row (T298485 T322360)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet |
[production] |
| 18:59 |
<ladsgroup@deploy1002> |
Started scap: Backport for [[gerrit:852883|WikiExporter: Avoid calling reload in processing every row (T298485 T322360)]] |
[production] |
| 18:27 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1161 (T318605)', diff saved to https://phabricator.wikimedia.org/P38087 and previous config saved to /var/cache/conftool/dbconfig/20221103-182756-ladsgroup.json |
[production] |
| 18:27 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'increase db1144:3315 load', diff saved to https://phabricator.wikimedia.org/P38086 and previous config saved to /var/cache/conftool/dbconfig/20221103-182750-jynus.json |
[production] |
| 18:23 |
<andrew@deploy1002> |
Finished deploy [horizon/deploy@9d02cd6]: (no justification provided) (duration: 02m 56s) |
[production] |
| 18:20 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance |
[production] |
| 18:20 |
<andrew@deploy1002> |
Started deploy [horizon/deploy@9d02cd6]: (no justification provided) |
[production] |
| 18:20 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance |
[production] |
| 18:20 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1161.eqiad.wmnet with reason: Maintenance |
[production] |
| 18:19 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1161.eqiad.wmnet with reason: Maintenance |
[production] |
| 18:16 |
<bblack> |
lvs1017: restart pybal to clear etcd error states |
[production] |
| 18:15 |
<bblack> |
lvs1018: restart pybal to clear etcd error states |
[production] |
| 18:14 |
<bblack> |
lvs1019: restart pybal to clear etcd error states |
[production] |
| 18:11 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 18:10 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mw-debug: apply |
[production] |
| 18:10 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 18:09 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mw-debug: apply |
[production] |
| 18:09 |
<bblack> |
lvs1020: restart pybal to hopefully clear etcd error states |
[production] |
| 18:07 |
<jhuneidi@deploy1002> |
rebuilt and synchronized wikiversions files: all wikis to 1.40.0-wmf.8 refs T320513 |
[production] |
| 17:46 |
<vgutierrez> |
vgutierrez@conf1007:~$ sudo -i systemctl start etcd |
[production] |
| 17:46 |
<fnegri@cumin1001> |
conftool action : set/pooled=inactive; selector: name=dbproxy1018.eqiad.wmnet,service=wikireplicas-b |
[production] |
| 17:45 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1150.eqiad.wmnet with reason: Maintenance |
[production] |
| 17:45 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1150.eqiad.wmnet with reason: Maintenance |
[production] |
| 17:43 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2121', diff saved to https://phabricator.wikimedia.org/P38084 and previous config saved to /var/cache/conftool/dbconfig/20221103-174306-marostegui.json |
[production] |
| 17:39 |
<elukey> |
`sudo truncate -s 20G /var/log/nginx/etcd_access.log.1` on conf100[7-9], root partition full |
[production] |
| 17:38 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1134', diff saved to https://phabricator.wikimedia.org/P38083 and previous config saved to /var/cache/conftool/dbconfig/20221103-173843-ladsgroup.json |
[production] |
| 17:30 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1144:3315', diff saved to https://phabricator.wikimedia.org/P38082 and previous config saved to /var/cache/conftool/dbconfig/20221103-173022-ladsgroup.json |
[production] |
| 17:28 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2121', diff saved to https://phabricator.wikimedia.org/P38081 and previous config saved to /var/cache/conftool/dbconfig/20221103-172759-marostegui.json |
[production] |
| 17:23 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1134 (T318955)', diff saved to https://phabricator.wikimedia.org/P38080 and previous config saved to /var/cache/conftool/dbconfig/20221103-172338-ladsgroup.json |
[production] |
| 17:22 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2173 (T318955)', diff saved to https://phabricator.wikimedia.org/P38079 and previous config saved to /var/cache/conftool/dbconfig/20221103-172235-ladsgroup.json |
[production] |