|
2022-11-03
ยง
|
| 20:10 |
<thcipriani@deploy1002> |
thcipriani and daniel: Backport for [[gerrit:843955|Enable parsoid cache warming on testwiki. (T320535)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet |
[production] |
| 20:10 |
<thcipriani@deploy1002> |
Started scap: Backport for [[gerrit:843955|Enable parsoid cache warming on testwiki. (T320535)]] |
[production] |
| 20:06 |
<samtar@deploy1002> |
backport aborted: (duration: 05m 23s) |
[production] |
| 19:56 |
<ryankemper> |
FOO Merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/852885; disabled puppet on query service fleet via `ryankemper@cumin1001:~$ sudo -E cumin 'A:wcqs-public or A:wdqs-all' 'sudo disable-puppet "T322037"'`; testing change on `wdqs1009` |
[production] |
| 19:49 |
<cstone> |
civicrm upgraded from d1f286f0 to c0db8f34 |
[production] |
| 19:48 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1185 (T318605)', diff saved to https://phabricator.wikimedia.org/P38092 and previous config saved to /var/cache/conftool/dbconfig/20221103-194839-ladsgroup.json |
[production] |
| 19:48 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1185.eqiad.wmnet with reason: Maintenance |
[production] |
| 19:48 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1185.eqiad.wmnet with reason: Maintenance |
[production] |
| 19:48 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1161 (T318605)', diff saved to https://phabricator.wikimedia.org/P38091 and previous config saved to /var/cache/conftool/dbconfig/20221103-194818-ladsgroup.json |
[production] |
| 19:40 |
<brennen@deploy1002> |
Finished deploy [phabricator/deployment@ea0ffa7]: initial deploy to phab1004 (duration: 02m 24s) |
[production] |
| 19:37 |
<brennen@deploy1002> |
Started deploy [phabricator/deployment@ea0ffa7]: initial deploy to phab1004 |
[production] |
| 19:33 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P38090 and previous config saved to /var/cache/conftool/dbconfig/20221103-193311-ladsgroup.json |
[production] |
| 19:29 |
<andrew@deploy1002> |
Finished deploy [horizon/deploy@9d02cd6]: (no justification provided) (duration: 00m 48s) |
[production] |
| 19:28 |
<andrew@deploy1002> |
Started deploy [horizon/deploy@9d02cd6]: (no justification provided) |
[production] |
| 19:26 |
<andrew@deploy1002> |
Finished deploy [horizon/deploy@9d02cd6]: (no justification provided) (duration: 00m 48s) |
[production] |
| 19:25 |
<andrew@deploy1002> |
Started deploy [horizon/deploy@9d02cd6]: (no justification provided) |
[production] |
| 19:22 |
<brennen@deploy1002> |
Finished deploy [phabricator/deployment@ea0ffa7]: initial deploy to phab1004 (duration: 00m 25s) |
[production] |
| 19:22 |
<brennen@deploy1002> |
Started deploy [phabricator/deployment@ea0ffa7]: initial deploy to phab1004 |
[production] |
| 19:18 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P38089 and previous config saved to /var/cache/conftool/dbconfig/20221103-191805-ladsgroup.json |
[production] |
| 19:06 |
<andrew@deploy1002> |
Finished deploy [horizon/deploy@9d02cd6]: (no justification provided) (duration: 01m 10s) |
[production] |
| 19:05 |
<andrew@deploy1002> |
Started deploy [horizon/deploy@9d02cd6]: (no justification provided) |
[production] |
| 19:03 |
<ladsgroup@deploy1002> |
Finished scap: Backport for [[gerrit:852883|WikiExporter: Avoid calling reload in processing every row (T298485 T322360)]] (duration: 04m 24s) |
[production] |
| 19:03 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 19:03 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mw-debug: apply |
[production] |
| 19:03 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 19:02 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1161 (T318605)', diff saved to https://phabricator.wikimedia.org/P38088 and previous config saved to /var/cache/conftool/dbconfig/20221103-190258-ladsgroup.json |
[production] |
| 19:02 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mw-debug: apply |
[production] |
| 18:59 |
<ladsgroup@deploy1002> |
ladsgroup and ladsgroup: Backport for [[gerrit:852883|WikiExporter: Avoid calling reload in processing every row (T298485 T322360)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet |
[production] |
| 18:59 |
<ladsgroup@deploy1002> |
Started scap: Backport for [[gerrit:852883|WikiExporter: Avoid calling reload in processing every row (T298485 T322360)]] |
[production] |
| 18:27 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1161 (T318605)', diff saved to https://phabricator.wikimedia.org/P38087 and previous config saved to /var/cache/conftool/dbconfig/20221103-182756-ladsgroup.json |
[production] |
| 18:27 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'increase db1144:3315 load', diff saved to https://phabricator.wikimedia.org/P38086 and previous config saved to /var/cache/conftool/dbconfig/20221103-182750-jynus.json |
[production] |
| 18:23 |
<andrew@deploy1002> |
Finished deploy [horizon/deploy@9d02cd6]: (no justification provided) (duration: 02m 56s) |
[production] |
| 18:20 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance |
[production] |
| 18:20 |
<andrew@deploy1002> |
Started deploy [horizon/deploy@9d02cd6]: (no justification provided) |
[production] |
| 18:20 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance |
[production] |
| 18:20 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1161.eqiad.wmnet with reason: Maintenance |
[production] |
| 18:19 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1161.eqiad.wmnet with reason: Maintenance |
[production] |
| 18:16 |
<bblack> |
lvs1017: restart pybal to clear etcd error states |
[production] |
| 18:15 |
<bblack> |
lvs1018: restart pybal to clear etcd error states |
[production] |
| 18:14 |
<bblack> |
lvs1019: restart pybal to clear etcd error states |
[production] |
| 18:11 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 18:10 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mw-debug: apply |
[production] |
| 18:10 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 18:09 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mw-debug: apply |
[production] |
| 18:09 |
<bblack> |
lvs1020: restart pybal to hopefully clear etcd error states |
[production] |
| 18:07 |
<jhuneidi@deploy1002> |
rebuilt and synchronized wikiversions files: all wikis to 1.40.0-wmf.8 refs T320513 |
[production] |
| 17:46 |
<vgutierrez> |
vgutierrez@conf1007:~$ sudo -i systemctl start etcd |
[production] |
| 17:46 |
<fnegri@cumin1001> |
conftool action : set/pooled=inactive; selector: name=dbproxy1018.eqiad.wmnet,service=wikireplicas-b |
[production] |
| 17:45 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1150.eqiad.wmnet with reason: Maintenance |
[production] |
| 17:45 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1150.eqiad.wmnet with reason: Maintenance |
[production] |