|
2021-06-11
§
|
| 14:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1143 (re)pooling @ 75%: Repool db1143 after upgrade', diff saved to https://phabricator.wikimedia.org/P16431 and previous config saved to /var/cache/conftool/dbconfig/20210611-144514-root.json |
[production] |
| 14:45 |
<balloons> |
Increase quota to 21 cpu, 42GB ram, 360G disk T284662 |
[logging] |
| 14:44 |
<mbsantos@deploy1002> |
Finished deploy [tilerator/deploy@6bfdab5]: (no justification provided) (duration: 00m 05s) |
[production] |
| 14:44 |
<mbsantos@deploy1002> |
Started deploy [tilerator/deploy@6bfdab5]: (no justification provided) |
[production] |
| 14:43 |
<mbsantos@deploy1002> |
Finished deploy [kartotherian/deploy@5d7c993]: (no justification provided) (duration: 00m 05s) |
[production] |
| 14:42 |
<mbsantos@deploy1002> |
Started deploy [kartotherian/deploy@5d7c993]: (no justification provided) |
[production] |
| 14:39 |
<balloons> |
add 16 vcpu, to 178 total T284507 |
[integration] |
| 14:38 |
<balloons> |
set quota to 16 vcpu, 24G ram T284527 |
[wikisource] |
| 14:36 |
<hnowlan@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on maps1008.eqiad.wmnet with reason: Reparenting from maps1009 |
[production] |
| 14:36 |
<hnowlan@cumin1001> |
START - Cookbook sre.hosts.downtime for 4:00:00 on maps1008.eqiad.wmnet with reason: Reparenting from maps1009 |
[production] |
| 14:35 |
<jiji@deploy1002> |
helmfile [codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 14:35 |
<jiji@deploy1002> |
helmfile [codfw] START helmfile.d/admin 'apply'. |
[production] |
| 14:34 |
<jiji@deploy1002> |
helmfile [eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 14:34 |
<jiji@deploy1002> |
helmfile [eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 14:34 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=maps1008.eqiad.wmnet |
[production] |
| 14:33 |
<jiji@deploy1002> |
helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 14:33 |
<jiji@deploy1002> |
helmfile [staging-eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 14:32 |
<jiji@deploy1002> |
helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 14:31 |
<jiji@deploy1002> |
helmfile [staging-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 14:30 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1143 (re)pooling @ 50%: Repool db1143 after upgrade', diff saved to https://phabricator.wikimedia.org/P16430 and previous config saved to /var/cache/conftool/dbconfig/20210611-143010-root.json |
[production] |
| 14:22 |
<jiji@deploy1002> |
helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 14:22 |
<jiji@deploy1002> |
helmfile [staging-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 14:20 |
<jiji@deploy1002> |
helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 14:20 |
<jiji@deploy1002> |
helmfile [staging-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 14:17 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=maps1008.eqiad.wmnet |
[production] |
| 14:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1143 (re)pooling @ 25%: Repool db1143 after upgrade', diff saved to https://phabricator.wikimedia.org/P16429 and previous config saved to /var/cache/conftool/dbconfig/20210611-141506-root.json |
[production] |
| 14:14 |
<hashar> |
deployment-imagescaler03: delete local mwdeploy user with uid 497 # T73480 |
[releng] |
| 13:53 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=maps1008.eqiad.wmnet |
[production] |
| 13:53 |
<hnowlan@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on maps1008.eqiad.wmnet with reason: Reparenting from maps1009 |
[production] |
| 13:53 |
<hnowlan@cumin1001> |
START - Cookbook sre.hosts.downtime for 4:00:00 on maps1008.eqiad.wmnet with reason: Reparenting from maps1009 |
[production] |
| 13:52 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1143 for kernel upgrade', diff saved to https://phabricator.wikimedia.org/P16428 and previous config saved to /var/cache/conftool/dbconfig/20210611-135248-marostegui.json |
[production] |
| 13:50 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1153', diff saved to https://phabricator.wikimedia.org/P16427 and previous config saved to /var/cache/conftool/dbconfig/20210611-135036-marostegui.json |
[production] |
| 13:35 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1153 for kernel upgrade', diff saved to https://phabricator.wikimedia.org/P16426 and previous config saved to /var/cache/conftool/dbconfig/20210611-133527-marostegui.json |
[production] |
| 12:46 |
<Jayprakash12345> |
Deploying build (I670a77ce84fa - T284818) |
[tools.bodh] |
| 12:45 |
<Jayprakash12345> |
Deploying build (I670a77ce84fa) |
[tools.bodh] |
| 12:31 |
<hashar> |
deployment-prep: removed deployment-shellbox puppet certificate and regenerated it. Ran puppet and it passes all fine. |
[releng] |
| 10:46 |
<oblivian@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'shellbox' for release 'main' . |
[production] |
| 10:30 |
<hashar> |
deployment-prep: cherry picked https://gerrit.wikimedia.org/r/c/operations/puppet/+/699207 to add a motd on all instances # T100837 |
[releng] |
| 07:29 |
<moritzm> |
restarting archiva to pick up OpenJDK security updates |
[production] |
| 07:09 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mwmaint2002.codfw.wmnet |
[production] |
| 07:03 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host mwmaint2002.codfw.wmnet |
[production] |
| 06:56 |
<oblivian@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 05:56 |
<elukey> |
rm -rf empty dir /etc/apache2/sites-enabled/.links2 on webperf1001 to avoid puppet changes at every run |
[production] |
| 05:47 |
<elukey> |
run systemctl reset-failed ifup@en5.service on doh1001 - T273026 |
[production] |
| 02:03 |
<Reedy> |
beta-update-databases-eqiad seemingly broken by CategoryTree fix for T271011. Comment left on gerrit patch and task, not reverting patch in master at this stage |
[releng] |
| 01:10 |
<eileen> |
process-control config revision is 2aed6ff89b |
[production] |