|
2022-11-03
ยง
|
| 21:08 |
<ryankemper> |
[WCQS] Pooled `wcqs100[1,2]` |
[production] |
| 21:04 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 21:03 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mw-debug: apply |
[production] |
| 21:03 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 21:02 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mw-debug: apply |
[production] |
| 20:58 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1200 (T318605)', diff saved to https://phabricator.wikimedia.org/P38098 and previous config saved to /var/cache/conftool/dbconfig/20221103-205855-ladsgroup.json |
[production] |
| 20:58 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1200.eqiad.wmnet with reason: Maintenance |
[production] |
| 20:58 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1200.eqiad.wmnet with reason: Maintenance |
[production] |
| 20:58 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1185 (T318605)', diff saved to https://phabricator.wikimedia.org/P38097 and previous config saved to /var/cache/conftool/dbconfig/20221103-205823-ladsgroup.json |
[production] |
| 20:57 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 20:56 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mw-debug: apply |
[production] |
| 20:56 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 20:56 |
<urbanecm@deploy1002> |
Finished scap: Backport for [[gerrit:853026|ApiSetMenteeStatus: Check GEMentorshipEnabled in wiki config (T321805)]], [[gerrit:852172|SpecialManageMentors: Do not include explanatory text on transclusion (T321773)]] (duration: 06m 34s) |
[production] |
| 20:55 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mw-debug: apply |
[production] |
| 20:52 |
<bking@cumin1001> |
END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) |
[production] |
| 20:50 |
<urbanecm@deploy1002> |
urbanecm and urbanecm: Backport for [[gerrit:853026|ApiSetMenteeStatus: Check GEMentorshipEnabled in wiki config (T321805)]], [[gerrit:852172|SpecialManageMentors: Do not include explanatory text on transclusion (T321773)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet |
[production] |
| 20:50 |
<jhathaway@deploy1002> |
helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'. |
[production] |
| 20:50 |
<jhathaway@deploy1002> |
helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'. |
[production] |
| 20:49 |
<urbanecm@deploy1002> |
Started scap: Backport for [[gerrit:853026|ApiSetMenteeStatus: Check GEMentorshipEnabled in wiki config (T321805)]], [[gerrit:852172|SpecialManageMentors: Do not include explanatory text on transclusion (T321773)]] |
[production] |
| 20:47 |
<bking@cumin1001> |
START - Cookbook sre.wdqs.data-transfer |
[production] |
| 20:46 |
<samtar@deploy1002> |
Finished scap: Backport for [[gerrit:853003|Update lv and bn wordmarks (T319223)]] (duration: 06m 36s) |
[production] |
| 20:45 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 20:44 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mw-debug: apply |
[production] |
| 20:44 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 20:43 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mw-debug: apply |
[production] |
| 20:43 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P38096 and previous config saved to /var/cache/conftool/dbconfig/20221103-204316-ladsgroup.json |
[production] |
| 20:39 |
<samtar@deploy1002> |
samtar and jdlrobson: Backport for [[gerrit:853003|Update lv and bn wordmarks (T319223)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet |
[production] |
| 20:39 |
<samtar@deploy1002> |
Started scap: Backport for [[gerrit:853003|Update lv and bn wordmarks (T319223)]] |
[production] |
| 20:35 |
<ryankemper> |
T322037 Rolling changes in https://gerrit.wikimedia.org/r/c/operations/puppet/+/852885 and https://gerrit.wikimedia.org/r/853006 out to query service fleet, 4 hosts at a time: `ryankemper@cumin1001:~$ sudo -E cumin -b 4 'A:wcqs-public or A:wdqs-all' 'run-puppet-agent --force'` |
[production] |
| 20:35 |
<samtar@deploy1002> |
Finished scap: Backport for [[gerrit:852882|Finish moving to Page Tools naming convention]] (duration: 07m 50s) |
[production] |
| 20:28 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P38094 and previous config saved to /var/cache/conftool/dbconfig/20221103-202810-ladsgroup.json |
[production] |
| 20:27 |
<samtar@deploy1002> |
samtar and jdlrobson: Backport for [[gerrit:852882|Finish moving to Page Tools naming convention]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet |
[production] |
| 20:27 |
<samtar@deploy1002> |
Started scap: Backport for [[gerrit:852882|Finish moving to Page Tools naming convention]] |
[production] |
| 20:23 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 20:22 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mw-debug: apply |
[production] |
| 20:22 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 20:21 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mw-debug: apply |
[production] |
| 20:20 |
<thcipriani@deploy1002> |
Finished scap: Backport for [[gerrit:843955|Enable parsoid cache warming on testwiki. (T320535)]] (duration: 10m 30s) |
[production] |
| 20:16 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 20:15 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mw-debug: apply |
[production] |
| 20:15 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 20:14 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mw-debug: apply |
[production] |
| 20:13 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1185 (T318605)', diff saved to https://phabricator.wikimedia.org/P38093 and previous config saved to /var/cache/conftool/dbconfig/20221103-201303-ladsgroup.json |
[production] |
| 20:10 |
<thcipriani@deploy1002> |
thcipriani and daniel: Backport for [[gerrit:843955|Enable parsoid cache warming on testwiki. (T320535)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet |
[production] |
| 20:10 |
<thcipriani@deploy1002> |
Started scap: Backport for [[gerrit:843955|Enable parsoid cache warming on testwiki. (T320535)]] |
[production] |
| 20:06 |
<samtar@deploy1002> |
backport aborted: (duration: 05m 23s) |
[production] |
| 19:56 |
<ryankemper> |
FOO Merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/852885; disabled puppet on query service fleet via `ryankemper@cumin1001:~$ sudo -E cumin 'A:wcqs-public or A:wdqs-all' 'sudo disable-puppet "T322037"'`; testing change on `wdqs1009` |
[production] |
| 19:49 |
<cstone> |
civicrm upgraded from d1f286f0 to c0db8f34 |
[production] |
| 19:48 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1185 (T318605)', diff saved to https://phabricator.wikimedia.org/P38092 and previous config saved to /var/cache/conftool/dbconfig/20221103-194839-ladsgroup.json |
[production] |
| 19:48 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1185.eqiad.wmnet with reason: Maintenance |
[production] |