|
2019-03-27
ยง
|
| 20:14 |
<gehel@cumin2001> |
END (FAIL) - Cookbook sre.elasticsearch.e6-upgrade (exit_code=99) |
[production] |
| 20:14 |
<mholloway-shell@deploy1001> |
Synchronized php-1.33.0-wmf.23/extensions/WikimediaEditorTasks: Fix: Use READ_LOCKING when evaluating whether to update targets_passed (duration: 00m 58s) |
[production] |
| 20:04 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.e6-upgrade |
[production] |
| 20:03 |
<gehel@cumin2001> |
END (ERROR) - Cookbook sre.elasticsearch.e6-upgrade (exit_code=97) |
[production] |
| 19:48 |
<gehel@cumin2001> |
END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) |
[production] |
| 19:43 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.force-shard-allocation |
[production] |
| 19:43 |
<herron> |
removed queued wikidata notification messages for a***a@w**gm**ster.** on mx1001 to address gmail excessive volume rate limiting |
[production] |
| 19:32 |
<jijiki> |
restarting pdfrender on scb1001 |
[production] |
| 19:30 |
<gehel@cumin2001> |
END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) |
[production] |
| 19:27 |
<marxarelli> |
(resent; originally @ 1916) dduvall@deploy1001 rebuilt and synchronized wikiversions files: group1 wikis to 1.33.0-wmf.23 |
[production] |
| 19:23 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.force-shard-allocation |
[production] |
| 19:18 |
<dduvall@deploy1001> |
Synchronized php: group1 wikis to 1.33.0-wmf.23 (duration: 01m 45s) |
[production] |
| 19:16 |
<dduvall@deploy1001> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.33.0-wmf.23 |
[production] |
| 19:14 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.e6-upgrade |
[production] |
| 18:48 |
<thcipriani> |
restarting gerrit process |
[production] |
| 18:12 |
<jynus> |
update grants on db1115 for new provisioning hosts on codfw T218336 |
[production] |
| 18:10 |
<elukey> |
interface::rps applied to all the mc10XX hosts - T203786 |
[production] |
| 17:41 |
<gehel@cumin2001> |
END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) |
[production] |
| 17:41 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.force-shard-allocation |
[production] |
| 17:10 |
<ema> |
fermium: /usr/local/sbin/disable_list wikimetrics T211835 |
[production] |
| 16:55 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T214075 SDC: Enable Wikidata federation on Commons (duration: 00m 57s) |
[production] |
| 16:38 |
<elukey> |
mc20XX and mc1022 have interface::rps enabled - T203786 |
[production] |
| 16:28 |
<jforrester@deploy1001> |
Synchronized php-1.33.0-wmf.23/extensions/GlobalPreferences/includes/GlobalPreferencesFactory.php: Hot-fix T219380 GlobalPreferences: Allow modifiedPrefs to be set even if no UI control (duration: 00m 58s) |
[production] |
| 16:18 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: SDC: Use feature flag for enabling depicts in UW (duration: 00m 57s) |
[production] |
| 16:13 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SDC: Add feature flag for enabling depicts in UW (duration: 00m 57s) |
[production] |
| 15:56 |
<jbond42> |
bastion reboots complete |
[production] |
| 15:56 |
<ariel@deploy1001> |
Finished deploy [dumps/dumps@88ddd76]: ability to use lbzip2 for meta-history compression (duration: 00m 03s) |
[production] |
| 15:56 |
<ariel@deploy1001> |
Started deploy [dumps/dumps@88ddd76]: ability to use lbzip2 for meta-history compression |
[production] |
| 15:53 |
<bd808> |
Restarted webservice to see if that fixes connectivity with tools.wmflabs.org proxy (T219377) |
[tools.wikiloves] |
| 15:44 |
<jbond42> |
rebooting bast2001.wikimedia.org in 5 minutes |
[production] |
| 15:44 |
<gehel@cumin2001> |
END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) |
[production] |
| 15:43 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.force-shard-allocation |
[production] |
| 15:42 |
<jbond42> |
rebooting bast2002.wikimedia.org in 5 minutes |
[production] |
| 15:38 |
<jbond42> |
rebooting bast1002.wikimedia.org in 5 minutes |
[production] |
| 15:34 |
<jbond42> |
rebooting bast4002.wikimedia.org in 5 minutes |
[production] |
| 15:30 |
<jbond42> |
rebooting bast5001.wikimedia.org in 5 minutes |
[production] |
| 15:24 |
<jbond42> |
rebooting iron.wikimedia.org in 5 minutes |
[production] |
| 15:22 |
<gehel@cumin2001> |
END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) |
[production] |
| 15:21 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.force-shard-allocation |
[production] |
| 15:19 |
<elukey> |
slowly rolling out interface::rps to all the mcXXXX nodes - T203786 |
[production] |
| 14:52 |
<gehel@cumin2001> |
END (FAIL) - Cookbook sre.elasticsearch.e6-upgrade (exit_code=99) |
[production] |
| 14:45 |
<gehel@cumin2001> |
END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) |
[production] |
| 14:45 |
<bstorm_> |
cleared several "E" state queues |
[tools] |
| 14:44 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.force-shard-allocation |
[production] |
| 14:43 |
<gtirloni> |
deployed shinken-puppetmaster-01 |
[shinken] |
| 14:16 |
<hashar> |
contint: refreshed all git caches manually from cumin: cumin --force 'name:docker' 'find /srv/git -name '*.git' -type d -print -exec git -C {} fetch --prune \;' |
[releng] |
| 14:16 |
<hashar> |
contint: added repositories to the git caches https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/499482/ |
[releng] |
| 14:13 |
<gehel@cumin2001> |
END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) |
[production] |
| 14:12 |
<marostegui> |
Sanitize hywwiki on db1124:3313 T212625 |
[production] |
| 14:11 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.force-shard-allocation |
[production] |