|
2020-03-02
ยง
|
| 23:58 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=wtp1025.eqiad.wmnet |
[production] |
| 23:43 |
<krinkle@deploy1001> |
Synchronized docroot/noc/: Idc26716abef5bff (duration: 00m 56s) |
[production] |
| 23:42 |
<krinkle@deploy1001> |
Synchronized multiversion/: Idc26716abef5bff (duration: 00m 57s) |
[production] |
| 23:41 |
<krinkle@deploy1001> |
Synchronized src/: Idc26716abef5bff (duration: 00m 56s) |
[production] |
| 23:36 |
<krinkle@deploy1001> |
Synchronized wmf-config/CommonSettings.php: I1bdc5e359476 (duration: 00m 56s) |
[production] |
| 21:10 |
<XioNoX> |
re-number AMS-IX peer 64271 |
[production] |
| 20:22 |
<otto@deploy1001> |
Synchronized wmf-config/InitialiseSettings-labs.php: Enable Mediawiki client side error logging on group0 wikis - T246030 (duration: 00m 57s) |
[production] |
| 20:12 |
<krinkle@deploy1001> |
Synchronized multiversion/MWConfigCacheGenerator.php: I1ef0589 (duration: 00m 58s) |
[production] |
| 20:01 |
<otto@deploy1001> |
Synchronized wmf-config/InitialiseSettings-labs.php: Enable Mediawiki client side error logging in beta - T246030 (duration: 00m 56s) |
[production] |
| 19:35 |
<otto@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: EventStreamConfig - allow eventgate-analytics-external to produce error events - T233629 (duration: 00m 56s) |
[production] |
| 19:17 |
<ppchelko@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Enable REST run jobs endpoint on jobrunners T244770 (duration: 00m 56s) |
[production] |
| 19:15 |
<ppchelko@deploy1001> |
Synchronized wmf-config/CommonSettings-labs.php: Enable REST run jobs endpoint on jobrunners T244770 (duration: 00m 56s) |
[production] |
| 19:04 |
<otto@deploy1001> |
Synchronized wmf-config/ProductionServices.php: Use new LVS port for EventBus for eventgate-main on all wikis - T245203 (duration: 00m 56s) |
[production] |
| 19:03 |
<otto@deploy1001> |
Synchronized wmf-config/LabsServices.php: Use new LVS port for EventBus for eventgate-main on all wikis - T245203 (duration: 00m 56s) |
[production] |
| 19:01 |
<otto@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Use new LVS port for EventBus for eventgate-main on all wikis - T245203 (duration: 00m 57s) |
[production] |
| 18:58 |
<otto@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Use new LVS port for EventBus for eventgate-main on all wikis - T245203 (duration: 00m 56s) |
[production] |
| 18:53 |
<otto@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Use new LVS port for EventBus for eventgate-main on group1 wikis - T245203 (duration: 00m 57s) |
[production] |
| 18:45 |
<otto@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Use new LVS port for EventBus for eventgate-main on group0 wikis - T245203 (duration: 00m 57s) |
[production] |
| 18:43 |
<otto@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Use new LVS port for EventBus for eventgate-main on group0 wikis - T245203 (duration: 00m 56s) |
[production] |
| 18:41 |
<XioNoX> |
remove BGP to lvs2004/5/6 on cr1/2-codfw |
[production] |
| 18:41 |
<ebernhardson@deploy1001> |
Finished deploy [search/mjolnir/deploy@8195b6f]: Bump python to 3.7, python-kafka to 1.4.7 (duration: 04m 04s) |
[production] |
| 18:41 |
<otto@deploy1001> |
Synchronized wmf-config/ProductionServices.php: Use new LVS port for EventBus for eventgate-main on group0 wikis - T245203 (duration: 00m 57s) |
[production] |
| 18:39 |
<otto@deploy1001> |
Synchronized wmf-config/LabsServices.php: Use new LVS port for EventBus for eventgate-main on group0 wikis - T245203 (duration: 00m 58s) |
[production] |
| 18:38 |
<ottomata> |
using new eventgate-main LVS ports for eventbus on group0 wikis - T245203 |
[production] |
| 18:37 |
<ebernhardson@deploy1001> |
Started deploy [search/mjolnir/deploy@8195b6f]: Bump python to 3.7, python-kafka to 1.4.7 |
[production] |
| 18:35 |
<XioNoX> |
add BGP to lvs2008 on cr1/2-codfw |
[production] |
| 18:02 |
<pt1979@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 17:59 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 17:50 |
<vgutierrez> |
starting pybal on lvs2009 - T246686 |
[production] |
| 17:41 |
<mutante> |
notebook1003 df: /mnt/hdfs: Input/output error | systemctl restart nagios-nrpe-server (T224682) |
[production] |
| 17:40 |
<mutante> |
notebook1003 systemctl restart nagios-nrpe-server |
[production] |
| 17:04 |
<vgutierrez> |
Stopping pybal on lvs2009 to let lvs2010 get its traffic - T246686 |
[production] |
| 16:20 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 16:20 |
<moritzm> |
installing netty-3.9 security updates |
[production] |
| 16:18 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 15:57 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 15:54 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 15:34 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Increase weight from 350 to 400 on db1111 T246447', diff saved to https://phabricator.wikimedia.org/P10583 and previous config saved to /var/cache/conftool/dbconfig/20200302-153416-marostegui.json |
[production] |
| 15:30 |
<vgutierrez> |
reimage lvs3007 with buster - T245984 |
[production] |
| 15:27 |
<vgutierrez@cumin2001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 15:26 |
<vgutierrez@cumin2001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 15:26 |
<vgutierrez> |
running the decommission cookbook against lvs2004 - T246669 |
[production] |
| 15:20 |
<otto@deploy1001> |
Synchronized wmf-config/ProductionServices.php: Use new LVS port for EventBus+monolog for eventgate-analytics - T245203 (duration: 00m 56s) |
[production] |
| 15:20 |
<ottomata> |
Use new LVS port for EventBus+monolog for eventgate-analytics - T245203 |
[production] |
| 15:11 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Increase weight from 300 to 350 on db1111 T246447', diff saved to https://phabricator.wikimedia.org/P10582 and previous config saved to /var/cache/conftool/dbconfig/20200302-151149-marostegui.json |
[production] |
| 14:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Increase weight from 250 to 300 on db1111 T246447', diff saved to https://phabricator.wikimedia.org/P10581 and previous config saved to /var/cache/conftool/dbconfig/20200302-145130-marostegui.json |
[production] |
| 14:42 |
<vgutierrez> |
Re-enable BGP in lvs5001 - T245984 |
[production] |
| 14:40 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Give weight to es4 and es5 unused eqiad slaves T246072', diff saved to https://phabricator.wikimedia.org/P10579 and previous config saved to /var/cache/conftool/dbconfig/20200302-144033-marostegui.json |
[production] |
| 14:39 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Give weight to es4 and es5 unused codfw slaves T246072', diff saved to https://phabricator.wikimedia.org/P10578 and previous config saved to /var/cache/conftool/dbconfig/20200302-143915-marostegui.json |
[production] |
| 14:38 |
<addshore@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Reading up to Q12M for the new term store everywhere (was Q10M) + warm db1126 & db1111 caches (T219123) cache bust (duration: 00m 56s) |
[production] |