|
2022-11-03
ยง
|
| 20:35 |
<ryankemper> |
T322037 Rolling changes in https://gerrit.wikimedia.org/r/c/operations/puppet/+/852885 and https://gerrit.wikimedia.org/r/853006 out to query service fleet, 4 hosts at a time: `ryankemper@cumin1001:~$ sudo -E cumin -b 4 'A:wcqs-public or A:wdqs-all' 'run-puppet-agent --force'` |
[production] |
| 20:35 |
<samtar@deploy1002> |
Finished scap: Backport for [[gerrit:852882|Finish moving to Page Tools naming convention]] (duration: 07m 50s) |
[production] |
| 20:28 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P38094 and previous config saved to /var/cache/conftool/dbconfig/20221103-202810-ladsgroup.json |
[production] |
| 20:27 |
<samtar@deploy1002> |
samtar and jdlrobson: Backport for [[gerrit:852882|Finish moving to Page Tools naming convention]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet |
[production] |
| 20:27 |
<samtar@deploy1002> |
Started scap: Backport for [[gerrit:852882|Finish moving to Page Tools naming convention]] |
[production] |
| 20:23 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 20:22 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mw-debug: apply |
[production] |
| 20:22 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 20:21 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mw-debug: apply |
[production] |
| 20:20 |
<thcipriani@deploy1002> |
Finished scap: Backport for [[gerrit:843955|Enable parsoid cache warming on testwiki. (T320535)]] (duration: 10m 30s) |
[production] |
| 20:16 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 20:15 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mw-debug: apply |
[production] |
| 20:15 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 20:14 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mw-debug: apply |
[production] |
| 20:13 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1185 (T318605)', diff saved to https://phabricator.wikimedia.org/P38093 and previous config saved to /var/cache/conftool/dbconfig/20221103-201303-ladsgroup.json |
[production] |
| 20:10 |
<thcipriani@deploy1002> |
thcipriani and daniel: Backport for [[gerrit:843955|Enable parsoid cache warming on testwiki. (T320535)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet |
[production] |
| 20:10 |
<thcipriani@deploy1002> |
Started scap: Backport for [[gerrit:843955|Enable parsoid cache warming on testwiki. (T320535)]] |
[production] |
| 20:06 |
<samtar@deploy1002> |
backport aborted: (duration: 05m 23s) |
[production] |
| 19:56 |
<ryankemper> |
FOO Merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/852885; disabled puppet on query service fleet via `ryankemper@cumin1001:~$ sudo -E cumin 'A:wcqs-public or A:wdqs-all' 'sudo disable-puppet "T322037"'`; testing change on `wdqs1009` |
[production] |
| 19:49 |
<cstone> |
civicrm upgraded from d1f286f0 to c0db8f34 |
[production] |
| 19:48 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1185 (T318605)', diff saved to https://phabricator.wikimedia.org/P38092 and previous config saved to /var/cache/conftool/dbconfig/20221103-194839-ladsgroup.json |
[production] |
| 19:48 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1185.eqiad.wmnet with reason: Maintenance |
[production] |
| 19:48 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1185.eqiad.wmnet with reason: Maintenance |
[production] |
| 19:48 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1161 (T318605)', diff saved to https://phabricator.wikimedia.org/P38091 and previous config saved to /var/cache/conftool/dbconfig/20221103-194818-ladsgroup.json |
[production] |
| 19:40 |
<brennen@deploy1002> |
Finished deploy [phabricator/deployment@ea0ffa7]: initial deploy to phab1004 (duration: 02m 24s) |
[production] |
| 19:37 |
<brennen@deploy1002> |
Started deploy [phabricator/deployment@ea0ffa7]: initial deploy to phab1004 |
[production] |
| 19:33 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P38090 and previous config saved to /var/cache/conftool/dbconfig/20221103-193311-ladsgroup.json |
[production] |
| 19:29 |
<andrew@deploy1002> |
Finished deploy [horizon/deploy@9d02cd6]: (no justification provided) (duration: 00m 48s) |
[production] |
| 19:28 |
<andrew@deploy1002> |
Started deploy [horizon/deploy@9d02cd6]: (no justification provided) |
[production] |
| 19:26 |
<andrew@deploy1002> |
Finished deploy [horizon/deploy@9d02cd6]: (no justification provided) (duration: 00m 48s) |
[production] |
| 19:25 |
<andrew@deploy1002> |
Started deploy [horizon/deploy@9d02cd6]: (no justification provided) |
[production] |
| 19:22 |
<brennen@deploy1002> |
Finished deploy [phabricator/deployment@ea0ffa7]: initial deploy to phab1004 (duration: 00m 25s) |
[production] |
| 19:22 |
<brennen@deploy1002> |
Started deploy [phabricator/deployment@ea0ffa7]: initial deploy to phab1004 |
[production] |
| 19:18 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P38089 and previous config saved to /var/cache/conftool/dbconfig/20221103-191805-ladsgroup.json |
[production] |
| 19:06 |
<andrew@deploy1002> |
Finished deploy [horizon/deploy@9d02cd6]: (no justification provided) (duration: 01m 10s) |
[production] |
| 19:05 |
<andrew@deploy1002> |
Started deploy [horizon/deploy@9d02cd6]: (no justification provided) |
[production] |
| 19:03 |
<ladsgroup@deploy1002> |
Finished scap: Backport for [[gerrit:852883|WikiExporter: Avoid calling reload in processing every row (T298485 T322360)]] (duration: 04m 24s) |
[production] |
| 19:03 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 19:03 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mw-debug: apply |
[production] |
| 19:03 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 19:02 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1161 (T318605)', diff saved to https://phabricator.wikimedia.org/P38088 and previous config saved to /var/cache/conftool/dbconfig/20221103-190258-ladsgroup.json |
[production] |
| 19:02 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mw-debug: apply |
[production] |
| 18:59 |
<ladsgroup@deploy1002> |
ladsgroup and ladsgroup: Backport for [[gerrit:852883|WikiExporter: Avoid calling reload in processing every row (T298485 T322360)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet |
[production] |
| 18:59 |
<ladsgroup@deploy1002> |
Started scap: Backport for [[gerrit:852883|WikiExporter: Avoid calling reload in processing every row (T298485 T322360)]] |
[production] |
| 18:27 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1161 (T318605)', diff saved to https://phabricator.wikimedia.org/P38087 and previous config saved to /var/cache/conftool/dbconfig/20221103-182756-ladsgroup.json |
[production] |
| 18:27 |
<jynus@cumin1001> |
dbctl commit (dc=all): 'increase db1144:3315 load', diff saved to https://phabricator.wikimedia.org/P38086 and previous config saved to /var/cache/conftool/dbconfig/20221103-182750-jynus.json |
[production] |
| 18:23 |
<andrew@deploy1002> |
Finished deploy [horizon/deploy@9d02cd6]: (no justification provided) (duration: 02m 56s) |
[production] |
| 18:20 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance |
[production] |
| 18:20 |
<andrew@deploy1002> |
Started deploy [horizon/deploy@9d02cd6]: (no justification provided) |
[production] |
| 18:20 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance |
[production] |