|
2019-03-27
ยง
|
| 21:58 |
<chaomodus> |
restarting proton1002 to upgrade ram |
[production] |
| 21:57 |
<chaomodus> |
restarting proton2001 in order to upgrade ram |
[production] |
| 21:54 |
<chaomodus> |
restarting proton2002 in order to upgrade ram |
[production] |
| 21:25 |
<dcausse@deploy1001> |
Synchronized wmf-config/Wikibase.php: T219448 (duration: 00m 55s) |
[production] |
| 21:25 |
<eileen> |
civicrm revision changed from 67b8405b60 to 7560af93df, config revision is 5a0cbb3c7d (was actually before the process control one) |
[production] |
| 21:24 |
<eileen> |
process-control config revision is e1bc772c89 |
[production] |
| 21:17 |
<chaomodus> |
restarted proton on proton1001 in response to memory exhaustion and cpu peg |
[production] |
| 21:07 |
<milimetric@deploy1001> |
Finished deploy [analytics/refinery@fdd21a4]: non-deploy changes and two new oozie jobs (duration: 11m 48s) |
[production] |
| 20:55 |
<milimetric@deploy1001> |
Started deploy [analytics/refinery@fdd21a4]: non-deploy changes and two new oozie jobs |
[production] |
| 20:29 |
<mholloway-shell@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Update WikimediaEditorTasks config for DB location split (duration: 00m 57s) |
[production] |
| 20:23 |
<mholloway-shell@deploy1001> |
Synchronized php-1.33.0-wmf.23/extensions/WikimediaEditorTasks: Update DB utils to handle counts and suggestion DBs in different locations (duration: 00m 58s) |
[production] |
| 20:14 |
<gehel@cumin2001> |
END (FAIL) - Cookbook sre.elasticsearch.e6-upgrade (exit_code=99) |
[production] |
| 20:14 |
<mholloway-shell@deploy1001> |
Synchronized php-1.33.0-wmf.23/extensions/WikimediaEditorTasks: Fix: Use READ_LOCKING when evaluating whether to update targets_passed (duration: 00m 58s) |
[production] |
| 20:04 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.e6-upgrade |
[production] |
| 20:03 |
<gehel@cumin2001> |
END (ERROR) - Cookbook sre.elasticsearch.e6-upgrade (exit_code=97) |
[production] |
| 19:48 |
<gehel@cumin2001> |
END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) |
[production] |
| 19:43 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.force-shard-allocation |
[production] |
| 19:43 |
<herron> |
removed queued wikidata notification messages for a***a@w**gm**ster.** on mx1001 to address gmail excessive volume rate limiting |
[production] |
| 19:32 |
<jijiki> |
restarting pdfrender on scb1001 |
[production] |
| 19:30 |
<gehel@cumin2001> |
END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) |
[production] |
| 19:27 |
<marxarelli> |
(resent; originally @ 1916) dduvall@deploy1001 rebuilt and synchronized wikiversions files: group1 wikis to 1.33.0-wmf.23 |
[production] |
| 19:23 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.force-shard-allocation |
[production] |
| 19:18 |
<dduvall@deploy1001> |
Synchronized php: group1 wikis to 1.33.0-wmf.23 (duration: 01m 45s) |
[production] |
| 19:16 |
<dduvall@deploy1001> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.33.0-wmf.23 |
[production] |
| 19:14 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.e6-upgrade |
[production] |
| 18:48 |
<thcipriani> |
restarting gerrit process |
[production] |
| 18:12 |
<jynus> |
update grants on db1115 for new provisioning hosts on codfw T218336 |
[production] |
| 18:10 |
<elukey> |
interface::rps applied to all the mc10XX hosts - T203786 |
[production] |
| 17:41 |
<gehel@cumin2001> |
END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) |
[production] |
| 17:41 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.force-shard-allocation |
[production] |
| 17:10 |
<ema> |
fermium: /usr/local/sbin/disable_list wikimetrics T211835 |
[production] |
| 16:55 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T214075 SDC: Enable Wikidata federation on Commons (duration: 00m 57s) |
[production] |
| 16:38 |
<elukey> |
mc20XX and mc1022 have interface::rps enabled - T203786 |
[production] |
| 16:28 |
<jforrester@deploy1001> |
Synchronized php-1.33.0-wmf.23/extensions/GlobalPreferences/includes/GlobalPreferencesFactory.php: Hot-fix T219380 GlobalPreferences: Allow modifiedPrefs to be set even if no UI control (duration: 00m 58s) |
[production] |
| 16:18 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: SDC: Use feature flag for enabling depicts in UW (duration: 00m 57s) |
[production] |
| 16:13 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SDC: Add feature flag for enabling depicts in UW (duration: 00m 57s) |
[production] |
| 15:56 |
<jbond42> |
bastion reboots complete |
[production] |
| 15:56 |
<ariel@deploy1001> |
Finished deploy [dumps/dumps@88ddd76]: ability to use lbzip2 for meta-history compression (duration: 00m 03s) |
[production] |
| 15:56 |
<ariel@deploy1001> |
Started deploy [dumps/dumps@88ddd76]: ability to use lbzip2 for meta-history compression |
[production] |
| 15:44 |
<jbond42> |
rebooting bast2001.wikimedia.org in 5 minutes |
[production] |
| 15:44 |
<gehel@cumin2001> |
END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) |
[production] |
| 15:43 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.force-shard-allocation |
[production] |
| 15:42 |
<jbond42> |
rebooting bast2002.wikimedia.org in 5 minutes |
[production] |
| 15:38 |
<jbond42> |
rebooting bast1002.wikimedia.org in 5 minutes |
[production] |
| 15:34 |
<jbond42> |
rebooting bast4002.wikimedia.org in 5 minutes |
[production] |
| 15:30 |
<jbond42> |
rebooting bast5001.wikimedia.org in 5 minutes |
[production] |
| 15:24 |
<jbond42> |
rebooting iron.wikimedia.org in 5 minutes |
[production] |
| 15:22 |
<gehel@cumin2001> |
END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) |
[production] |
| 15:21 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.force-shard-allocation |
[production] |
| 15:19 |
<elukey> |
slowly rolling out interface::rps to all the mcXXXX nodes - T203786 |
[production] |