|
2019-03-27
ยง
|
| 19:43 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.force-shard-allocation |
[production] |
| 19:43 |
<herron> |
removed queued wikidata notification messages for a***a@w**gm**ster.** on mx1001 to address gmail excessive volume rate limiting |
[production] |
| 19:32 |
<jijiki> |
restarting pdfrender on scb1001 |
[production] |
| 19:30 |
<gehel@cumin2001> |
END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) |
[production] |
| 19:27 |
<marxarelli> |
(resent; originally @ 1916) dduvall@deploy1001 rebuilt and synchronized wikiversions files: group1 wikis to 1.33.0-wmf.23 |
[production] |
| 19:23 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.force-shard-allocation |
[production] |
| 19:18 |
<dduvall@deploy1001> |
Synchronized php: group1 wikis to 1.33.0-wmf.23 (duration: 01m 45s) |
[production] |
| 19:16 |
<dduvall@deploy1001> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.33.0-wmf.23 |
[production] |
| 19:14 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.e6-upgrade |
[production] |
| 18:48 |
<thcipriani> |
restarting gerrit process |
[production] |
| 18:12 |
<jynus> |
update grants on db1115 for new provisioning hosts on codfw T218336 |
[production] |
| 18:10 |
<elukey> |
interface::rps applied to all the mc10XX hosts - T203786 |
[production] |
| 17:41 |
<gehel@cumin2001> |
END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) |
[production] |
| 17:41 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.force-shard-allocation |
[production] |
| 17:10 |
<ema> |
fermium: /usr/local/sbin/disable_list wikimetrics T211835 |
[production] |
| 16:55 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T214075 SDC: Enable Wikidata federation on Commons (duration: 00m 57s) |
[production] |
| 16:38 |
<elukey> |
mc20XX and mc1022 have interface::rps enabled - T203786 |
[production] |
| 16:28 |
<jforrester@deploy1001> |
Synchronized php-1.33.0-wmf.23/extensions/GlobalPreferences/includes/GlobalPreferencesFactory.php: Hot-fix T219380 GlobalPreferences: Allow modifiedPrefs to be set even if no UI control (duration: 00m 58s) |
[production] |
| 16:18 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: SDC: Use feature flag for enabling depicts in UW (duration: 00m 57s) |
[production] |
| 16:13 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SDC: Add feature flag for enabling depicts in UW (duration: 00m 57s) |
[production] |
| 15:56 |
<jbond42> |
bastion reboots complete |
[production] |
| 15:56 |
<ariel@deploy1001> |
Finished deploy [dumps/dumps@88ddd76]: ability to use lbzip2 for meta-history compression (duration: 00m 03s) |
[production] |
| 15:56 |
<ariel@deploy1001> |
Started deploy [dumps/dumps@88ddd76]: ability to use lbzip2 for meta-history compression |
[production] |
| 15:44 |
<jbond42> |
rebooting bast2001.wikimedia.org in 5 minutes |
[production] |
| 15:44 |
<gehel@cumin2001> |
END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) |
[production] |
| 15:43 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.force-shard-allocation |
[production] |
| 15:42 |
<jbond42> |
rebooting bast2002.wikimedia.org in 5 minutes |
[production] |
| 15:38 |
<jbond42> |
rebooting bast1002.wikimedia.org in 5 minutes |
[production] |
| 15:34 |
<jbond42> |
rebooting bast4002.wikimedia.org in 5 minutes |
[production] |
| 15:30 |
<jbond42> |
rebooting bast5001.wikimedia.org in 5 minutes |
[production] |
| 15:24 |
<jbond42> |
rebooting iron.wikimedia.org in 5 minutes |
[production] |
| 15:22 |
<gehel@cumin2001> |
END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) |
[production] |
| 15:21 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.force-shard-allocation |
[production] |
| 15:19 |
<elukey> |
slowly rolling out interface::rps to all the mcXXXX nodes - T203786 |
[production] |
| 14:52 |
<gehel@cumin2001> |
END (FAIL) - Cookbook sre.elasticsearch.e6-upgrade (exit_code=99) |
[production] |
| 14:45 |
<gehel@cumin2001> |
END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) |
[production] |
| 14:44 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.force-shard-allocation |
[production] |
| 14:13 |
<gehel@cumin2001> |
END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) |
[production] |
| 14:12 |
<marostegui> |
Sanitize hywwiki on db1124:3313 T212625 |
[production] |
| 14:11 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.force-shard-allocation |
[production] |
| 14:05 |
<godog> |
roll-restart logstash to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/498417 |
[production] |
| 13:38 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.e6-upgrade |
[production] |
| 13:35 |
<gehel@cumin2001> |
END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) |
[production] |
| 13:35 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.force-shard-allocation |
[production] |
| 13:11 |
<ladsgroup@deploy1001> |
Synchronized dblists: (no justification provided) (duration: 00m 57s) |
[production] |
| 12:42 |
<ladsgroup@deploy1001> |
Synchronized dblists: (no justification provided) (duration: 00m 58s) |
[production] |
| 12:41 |
<Amir1> |
scap sync-file dblists |
[production] |
| 12:30 |
<Amir1> |
mwscript extensions/WikimediaMaintenance/addWiki.php --wiki=mediawikiwiki hyw wikipedia hywwiki hyw.wikipedia.org |
[production] |
| 12:25 |
<gehel@cumin2001> |
END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) |
[production] |
| 12:23 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.force-shard-allocation |
[production] |