|
2020-03-03
§
|
| 06:49 |
<marostegui> |
Stop MySQL on db1096:3315,3316 for reimage - T246604 |
[production] |
| 06:43 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1096:3315, db1096:3316 for reimage T246604', diff saved to https://phabricator.wikimedia.org/P10587 and previous config saved to /var/cache/conftool/dbconfig/20200303-064316-marostegui.json |
[production] |
| 06:41 |
<elukey@cumin1001> |
START - Cookbook sre.druid.roll-restart-workers |
[production] |
| 06:33 |
<vgutierrez> |
Switch from globalsign to LE as unified cert vendor on eqiad - T230687 |
[production] |
| 06:25 |
<vgutierrez> |
Switch from globalsign to LE as unified cert vendor on codfw - T230687 |
[production] |
| 02:57 |
<mutante> |
manually running "Ancientpages" cron on s3 (T243599) |
[production] |
| 02:39 |
<mutante> |
manually running updateSpecialPages.php maintenance cron on s8 for AncientPages to confirm it was fixed by gerrit:574726 a few days ago (T243599) |
[production] |
| 00:53 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: T240055 Update special scandium configuration to load from /srv/parsoid-testing (duration: 00m 58s) |
[production] |
| 00:36 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Touch and secondary sync of IS for cache-busting (duration: 00m 56s) |
[production] |
| 00:35 |
<jforrester@deploy1001> |
Synchronized dblists/mobilemainpagelegacy.dblist: T32405 Drop legacy main page special casing on select projects (duration: 00m 56s) |
[production] |
| 00:31 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Touch and secondary sync of IS for cache-busting (duration: 00m 56s) |
[production] |
| 00:28 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T242030 Enable lead paragraph in user namespace on nlwiki (duration: 00m 56s) |
[production] |
| 00:15 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet |
[production] |
| 00:14 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: T240055: Point the Parsoid cluster at the train version of Parsoid, not a special check-out (duration: 00m 56s) |
[production] |
| 00:13 |
<jforrester@deploy1001> |
sync aborted: wmf-config/CommonSettings.php T240055: Point the Parsoid cluster at the train version of Parsoid, not a special check-out (duration: 00m 03s) |
[production] |
| 00:13 |
<jforrester@deploy1001> |
Started scap: wmf-config/CommonSettings.php T240055: Point the Parsoid cluster at the train version of Parsoid, not a special check-out |
[production] |
| 00:05 |
<mutante> |
wtp1025 - scap pull |
[production] |
|
2020-03-02
§
|
| 23:58 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=wtp1025.eqiad.wmnet |
[production] |
| 23:43 |
<krinkle@deploy1001> |
Synchronized docroot/noc/: Idc26716abef5bff (duration: 00m 56s) |
[production] |
| 23:42 |
<krinkle@deploy1001> |
Synchronized multiversion/: Idc26716abef5bff (duration: 00m 57s) |
[production] |
| 23:41 |
<krinkle@deploy1001> |
Synchronized src/: Idc26716abef5bff (duration: 00m 56s) |
[production] |
| 23:36 |
<krinkle@deploy1001> |
Synchronized wmf-config/CommonSettings.php: I1bdc5e359476 (duration: 00m 56s) |
[production] |
| 21:10 |
<XioNoX> |
re-number AMS-IX peer 64271 |
[production] |
| 20:22 |
<otto@deploy1001> |
Synchronized wmf-config/InitialiseSettings-labs.php: Enable Mediawiki client side error logging on group0 wikis - T246030 (duration: 00m 57s) |
[production] |
| 20:12 |
<krinkle@deploy1001> |
Synchronized multiversion/MWConfigCacheGenerator.php: I1ef0589 (duration: 00m 58s) |
[production] |
| 20:01 |
<otto@deploy1001> |
Synchronized wmf-config/InitialiseSettings-labs.php: Enable Mediawiki client side error logging in beta - T246030 (duration: 00m 56s) |
[production] |
| 19:35 |
<otto@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: EventStreamConfig - allow eventgate-analytics-external to produce error events - T233629 (duration: 00m 56s) |
[production] |
| 19:17 |
<ppchelko@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Enable REST run jobs endpoint on jobrunners T244770 (duration: 00m 56s) |
[production] |
| 19:15 |
<ppchelko@deploy1001> |
Synchronized wmf-config/CommonSettings-labs.php: Enable REST run jobs endpoint on jobrunners T244770 (duration: 00m 56s) |
[production] |
| 19:04 |
<otto@deploy1001> |
Synchronized wmf-config/ProductionServices.php: Use new LVS port for EventBus for eventgate-main on all wikis - T245203 (duration: 00m 56s) |
[production] |
| 19:03 |
<otto@deploy1001> |
Synchronized wmf-config/LabsServices.php: Use new LVS port for EventBus for eventgate-main on all wikis - T245203 (duration: 00m 56s) |
[production] |
| 19:01 |
<otto@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Use new LVS port for EventBus for eventgate-main on all wikis - T245203 (duration: 00m 57s) |
[production] |
| 18:58 |
<otto@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Use new LVS port for EventBus for eventgate-main on all wikis - T245203 (duration: 00m 56s) |
[production] |
| 18:53 |
<otto@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Use new LVS port for EventBus for eventgate-main on group1 wikis - T245203 (duration: 00m 57s) |
[production] |
| 18:45 |
<otto@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Use new LVS port for EventBus for eventgate-main on group0 wikis - T245203 (duration: 00m 57s) |
[production] |
| 18:43 |
<otto@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Use new LVS port for EventBus for eventgate-main on group0 wikis - T245203 (duration: 00m 56s) |
[production] |
| 18:41 |
<XioNoX> |
remove BGP to lvs2004/5/6 on cr1/2-codfw |
[production] |
| 18:41 |
<ebernhardson@deploy1001> |
Finished deploy [search/mjolnir/deploy@8195b6f]: Bump python to 3.7, python-kafka to 1.4.7 (duration: 04m 04s) |
[production] |
| 18:41 |
<otto@deploy1001> |
Synchronized wmf-config/ProductionServices.php: Use new LVS port for EventBus for eventgate-main on group0 wikis - T245203 (duration: 00m 57s) |
[production] |
| 18:39 |
<otto@deploy1001> |
Synchronized wmf-config/LabsServices.php: Use new LVS port for EventBus for eventgate-main on group0 wikis - T245203 (duration: 00m 58s) |
[production] |
| 18:38 |
<ottomata> |
using new eventgate-main LVS ports for eventbus on group0 wikis - T245203 |
[production] |
| 18:37 |
<ebernhardson@deploy1001> |
Started deploy [search/mjolnir/deploy@8195b6f]: Bump python to 3.7, python-kafka to 1.4.7 |
[production] |
| 18:35 |
<XioNoX> |
add BGP to lvs2008 on cr1/2-codfw |
[production] |
| 18:02 |
<pt1979@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 17:59 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 17:50 |
<vgutierrez> |
starting pybal on lvs2009 - T246686 |
[production] |
| 17:41 |
<mutante> |
notebook1003 df: /mnt/hdfs: Input/output error | systemctl restart nagios-nrpe-server (T224682) |
[production] |
| 17:40 |
<mutante> |
notebook1003 systemctl restart nagios-nrpe-server |
[production] |
| 17:04 |
<vgutierrez> |
Stopping pybal on lvs2009 to let lvs2010 get its traffic - T246686 |
[production] |
| 16:20 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |