|
2014-11-13
§
|
| 20:32 |
<bd808> |
Dropped replica count of all logstash indices except today to 0. Should make rolling restarts faster during hardware upgrade. |
[production] |
| 20:25 |
<awight> |
Synchronized php-1.25wmf8/extensions/CentralNotice: push CentralNotice updates (duration: 00m 05s) |
[production] |
| 20:19 |
<csteipp> |
patched bugs 71111 and 71394 in wmf7 and wmf8 |
[production] |
| 20:14 |
<cmjohnson> |
powering down logstash1003 for a few mins to add disks |
[production] |
| 19:52 |
<ottomata> |
starting upgrade to trusty on analytics1023 |
[production] |
| 19:15 |
<awight> |
campaigns reenabled |
[production] |
| 18:55 |
<awight> |
disabling CentralNotice campaigns |
[production] |
| 17:49 |
<ottomata> |
preparing for trusty upgrade of analytics1003 |
[production] |
| 16:57 |
<bd808> |
dropped replica count to 0 for logstash indices from 2014-10-30 and 2014-10-31. |
[production] |
| 16:49 |
<bd808> |
restarted elasticsearch on logstash1002 |
[production] |
| 16:46 |
<bd808> |
dropped replica count to 0 for logstash indices from 2014-10-14 through 2014-10-29. See https://phabricator.wikimedia.org/P73 for the commands. |
[production] |
| 16:45 |
<ottomata> |
preparing to upgrade analytics1026 to trusty |
[production] |
| 16:21 |
<bd808> |
disk utilization is 94% on logstash1002, 92% on logstash1001 and 91% on logstash1003. Too much data in indices even with replica count bumped down to 1 for the small disks we have today. |
[production] |
| 16:16 |
<bd808> |
logstash elasticsearch cluster is pretty messed up. logstash1002 has lost shards for all indices except for today, and it's master for that one. |
[production] |
| 16:16 |
<manybubbles> |
Synchronized php-1.25wmf8/extensions/CirrusSearch/: SWAT update cirrussearch to fix slow prefix queries (duration: 00m 05s) |
[production] |
| 16:14 |
<manybubbles> |
Synchronized wmf-config/CirrusSearch-production.php: SWAT reenable regex search now that it will not crash elasticsearch (duration: 00m 04s) |
[production] |
| 16:13 |
<manybubbles> |
Synchronized wmf-config/CirrusSearch-common.php: SWAT reenable accelerated regex search (regex search still disabled) (duration: 00m 03s) |
[production] |
| 16:11 |
<manybubbles> |
Synchronized wmf-config/InitialiseSettings.php: SWAT force summary when running checkuser query on all wikis (duration: 00m 04s) |
[production] |
| 16:01 |
<manybubbles> |
Synchronized wmf-config/InitialiseSettings.php: SWAT revert JPG thumbnail chaining on all wikis except commons (duration: 00m 05s) |
[production] |
| 15:27 |
<hashar:> |
deleted all content from https://doc.wikimedia.org/ :-( Will regenerate. |
[production] |
| 15:09 |
<godog> |
rolling restart of object-auditor in swift codfw/eqiad to pick up changes |
[production] |
| 15:06 |
<yurik> |
Synchronized php-1.25wmf8/extensions/ZeroPortal: updatidng ZeroPortal to master (duration: 01m 13s) |
[production] |
| 15:04 |
<chasemp> |
phabricator upgrades T1203 |
[production] |
| 14:43 |
<hashar:> |
restarted zuul-merger on gallium |
[production] |
| 14:42 |
<hashar:> |
restarting Jenkins and Zuul |
[production] |
| 12:45 |
<godog> |
investigating high iops on swift eqiad with paravoid, stopped object-auditor on ms-be1005 and ms-be1015 |
[production] |
| 11:09 |
<hashar> |
resurrected morebots in #wikimedia-operations (see [[Morebots]]). |
[production] |
| 11:08 |
<hashar> |
Killed Jenkins due to a deadlock |
[production] |
| 11:08 |
<hashar> |
Killing Jenkins due to a deadlock |
[production] |
| 02:52 |
<mutante> |
beta puppet freshness - UNKNOWN: No valid datapoints found .. since 13d |
[production] |
| 02:30 |
<LocalisationUpdate> |
completed (1.25wmf8) at 2014-11-13 02:30:00+00:00 |
[production] |
| 02:18 |
<LocalisationUpdate> |
completed (1.25wmf7) at 2014-11-13 02:18:44+00:00 |
[production] |
| 00:46 |
<mutante> |
thulium - Could not intern from pson: expected value in object at '"[PHP]\\n\\n; puppet:t'! |
[production] |
|
2014-11-12
§
|
| 21:59 |
<reedy> |
Synchronized wmf-config/: Set useLegacyUsageIndex = true for Wikibase client (duration: 00m 17s) |
[production] |
| 20:57 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.25wmf8 |
[production] |
| 20:55 |
<hashar> |
Restarting Jenkins, deadlock on deployment-bastion |
[production] |
| 20:28 |
<reedy> |
Synchronized wmf-config/: (no message) (duration: 00m 15s) |
[production] |
| 20:28 |
<reedy> |
Synchronized database lists: (no message) (duration: 00m 15s) |
[production] |
| 20:25 |
<manybubbles> |
restarting elastic1021 to pick up new plugins |
[production] |
| 20:21 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.25wmf7 |
[production] |
| 20:13 |
<reedy> |
Finished scap: testwiki to 1.25wmf8 and build l10n cache (duration: 53m 57s) |
[production] |
| 19:37 |
<hoo> |
Made myself oauthadmin on mediawikiwiki |
[production] |
| 19:19 |
<reedy> |
Started scap: testwiki to 1.25wmf8 and build l10n cache |
[production] |
| 19:05 |
<mutante> |
installing package upgrades on bast1001 (incl. PHP version) |
[production] |
| 19:04 |
<mutante> |
installing package upgrades on iron |
[production] |
| 18:38 |
<YuviPanda> |
turned off yurik's zerosms cronjob on stat1002 (already discussed with him, he was ok with it being stopped until he could find time to fix it) |
[production] |
| 17:58 |
<_joe_> |
gracefulling apache on problematic API hosts |
[production] |
| 17:05 |
<demon> |
Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 04s) |
[production] |
| 16:51 |
<anomie> |
Synchronized php-1.25wmf7/extensions/SecurePoll/: SWAT: SecurePoll fix for jump-text and title on create/edit [[gerrit:172718]] (for real this time) (duration: 00m 09s) |
[production] |
| 16:48 |
<anomie> |
Finished scap: SWAT: SecurePoll fix for jump-text and title on create/edit [[gerrit:172718]] (duration: 22m 13s) |
[production] |