|
2020-03-02
ยง
|
| 18:38 |
<ottomata> |
using new eventgate-main LVS ports for eventbus on group0 wikis - T245203 |
[production] |
| 18:37 |
<ebernhardson@deploy1001> |
Started deploy [search/mjolnir/deploy@8195b6f]: Bump python to 3.7, python-kafka to 1.4.7 |
[production] |
| 18:35 |
<XioNoX> |
add BGP to lvs2008 on cr1/2-codfw |
[production] |
| 18:22 |
<brennen> |
Updating dev-images docker-pkg files on contint1001 for T246202 |
[releng] |
| 18:02 |
<pt1979@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 17:59 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 17:50 |
<vgutierrez> |
starting pybal on lvs2009 - T246686 |
[production] |
| 17:41 |
<mutante> |
notebook1003 df: /mnt/hdfs: Input/output error | systemctl restart nagios-nrpe-server (T224682) |
[production] |
| 17:40 |
<mutante> |
notebook1003 systemctl restart nagios-nrpe-server |
[production] |
| 17:06 |
<wm-bot> |
<root> Hard restart of webservice. Running Pod and requiested version in service.manifest did not match. |
[tools.zppixbot] |
| 17:04 |
<vgutierrez> |
Stopping pybal on lvs2009 to let lvs2010 get its traffic - T246686 |
[production] |
| 16:54 |
<arturo> |
[codfw1dev] deleted python3-os-ken debian package in cloudnet2003-dev which was installed by hand and had depedency issues |
[admin] |
| 16:37 |
<rxy> |
Add Urbanecm as maintainer |
[tools.stewardbots] |
| 16:20 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 16:20 |
<moritzm> |
installing netty-3.9 security updates |
[production] |
| 16:18 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 15:57 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 15:54 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 15:38 |
<elukey> |
apply new settings to all stat/notebooks |
[analytics] |
| 15:34 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Increase weight from 350 to 400 on db1111 T246447', diff saved to https://phabricator.wikimedia.org/P10583 and previous config saved to /var/cache/conftool/dbconfig/20200302-153416-marostegui.json |
[production] |
| 15:31 |
<elukey> |
setting new user.slice global memory/cpu settings on notebook1003 |
[analytics] |
| 15:30 |
<vgutierrez> |
reimage lvs3007 with buster - T245984 |
[production] |
| 15:30 |
<bstorm_> |
the correct deployment file is now refill.yaml. The last version is refill-old.yaml (which will certainly not work). |
[tools.refill-api] |
| 15:27 |
<vgutierrez@cumin2001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 15:26 |
<vgutierrez@cumin2001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 15:26 |
<vgutierrez> |
running the decommission cookbook against lvs2004 - T246669 |
[production] |
| 15:25 |
<elukey> |
setting new user-slice global memory/cpu settings on stat1007 |
[analytics] |
| 15:20 |
<otto@deploy1001> |
Synchronized wmf-config/ProductionServices.php: Use new LVS port for EventBus+monolog for eventgate-analytics - T245203 (duration: 00m 56s) |
[production] |
| 15:20 |
<ottomata> |
Use new LVS port for EventBus+monolog for eventgate-analytics - T245203 |
[production] |
| 15:18 |
<bstorm_> |
increasing max cpu per container to 2 |
[tools.refill-api] |
| 15:16 |
<bstorm_> |
reconfigured deployment to work according to the setup of the 2020 kubernetes cluster |
[tools.refill-api] |
| 15:15 |
<bstorm_> |
bumped cpu quota limit to 3 |
[tools.refill-api] |
| 15:11 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Increase weight from 300 to 350 on db1111 T246447', diff saved to https://phabricator.wikimedia.org/P10582 and previous config saved to /var/cache/conftool/dbconfig/20200302-151149-marostegui.json |
[production] |
| 14:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Increase weight from 250 to 300 on db1111 T246447', diff saved to https://phabricator.wikimedia.org/P10581 and previous config saved to /var/cache/conftool/dbconfig/20200302-145130-marostegui.json |
[production] |
| 14:42 |
<vgutierrez> |
Re-enable BGP in lvs5001 - T245984 |
[production] |
| 14:40 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Give weight to es4 and es5 unused eqiad slaves T246072', diff saved to https://phabricator.wikimedia.org/P10579 and previous config saved to /var/cache/conftool/dbconfig/20200302-144033-marostegui.json |
[production] |
| 14:39 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Give weight to es4 and es5 unused codfw slaves T246072', diff saved to https://phabricator.wikimedia.org/P10578 and previous config saved to /var/cache/conftool/dbconfig/20200302-143915-marostegui.json |
[production] |
| 14:38 |
<addshore@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Reading up to Q12M for the new term store everywhere (was Q10M) + warm db1126 & db1111 caches (T219123) cache bust (duration: 00m 56s) |
[production] |
| 14:37 |
<addshore@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Reading up to Q12M for the new term store everywhere (was Q10M) + warm db1126 & db1111 caches (T219123) (duration: 00m 58s) |
[production] |
| 14:37 |
<vgutierrez@cumin2001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 14:36 |
<vgutierrez@cumin2001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 14:36 |
<vgutierrez> |
running the decommission cookbook against lvs2005 - T246666 |
[production] |
| 14:20 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Increase weight from 200 to 250 on db1111 T246447', diff saved to https://phabricator.wikimedia.org/P10577 and previous config saved to /var/cache/conftool/dbconfig/20200302-142017-marostegui.json |
[production] |
| 14:19 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:17 |
<addshore> |
START warm cache for db1111 & db1126 for Q10-12 million T219123 (pass 3) |
[production] |
| 14:15 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:05 |
<vgutierrez> |
update puppet compiler facts |
[production] |
| 13:58 |
<addshore> |
START warm cache for db1111 & db1126 for Q10-12 million T219123 (pass 2) |
[production] |
| 13:55 |
<vgutierrez> |
Switch from globalsign to LE as unified cert vendor on ulsfo - T230687 |
[production] |
| 13:53 |
<vgutierrez> |
Switch from globalsign to LE as unified cert vendor on cp4026 - T230687 |
[production] |