|
2014-11-13
§
|
| 22:19 |
<jgage> |
hadoop: analytics1010 is again active namenode |
[production] |
| 22:14 |
<awight> |
Synchronized php-1.25wmf8/extensions/CentralNotice: push CentralNotice updates (duration: 00m 04s) |
[production] |
| 22:13 |
<awight> |
Synchronized php-1.25wmf7/extensions/CentralNotice: push CentralNotice updates (duration: 00m 05s) |
[production] |
| 22:12 |
<qchris> |
restarted EventLogging jobs that write to disk, to pick up config changes |
[production] |
| 22:03 |
<jgage> |
failed over hadoop namenode to analytics1004 |
[production] |
| 21:42 |
<awight> |
Synchronized wmf-config: Enabling CentralNotice banner choice on testwiki, take 2 (duration: 00m 06s) |
[production] |
| 21:15 |
<cscott> |
updated Parsoid to version dabff010 |
[production] |
| 20:51 |
<cmjohnson> |
powering down logstash1001 to add disks |
[production] |
| 20:39 |
<cmjohnson> |
powering down logstash1002 to add disks |
[production] |
| 20:37 |
<awight> |
CentralNotice noops deployed to all wikis |
[production] |
| 20:36 |
<awight> |
Synchronized php-1.25wmf7/extensions/CentralNotice: push CentralNotice updates (duration: 00m 05s) |
[production] |
| 20:33 |
<awight> |
Synchronized wmf-config: Enabling CentralNotice banner choice on testwiki (duration: 00m 04s) |
[production] |
| 20:32 |
<bd808> |
Dropped replica count of all logstash indices except today to 0. Should make rolling restarts faster during hardware upgrade. |
[production] |
| 20:25 |
<awight> |
Synchronized php-1.25wmf8/extensions/CentralNotice: push CentralNotice updates (duration: 00m 05s) |
[production] |
| 20:19 |
<csteipp> |
patched bugs 71111 and 71394 in wmf7 and wmf8 |
[production] |
| 20:14 |
<cmjohnson> |
powering down logstash1003 for a few mins to add disks |
[production] |
| 19:52 |
<ottomata> |
starting upgrade to trusty on analytics1023 |
[production] |
| 19:15 |
<awight> |
campaigns reenabled |
[production] |
| 18:55 |
<awight> |
disabling CentralNotice campaigns |
[production] |
| 17:49 |
<ottomata> |
preparing for trusty upgrade of analytics1003 |
[production] |
| 16:57 |
<bd808> |
dropped replica count to 0 for logstash indices from 2014-10-30 and 2014-10-31. |
[production] |
| 16:49 |
<bd808> |
restarted elasticsearch on logstash1002 |
[production] |
| 16:46 |
<bd808> |
dropped replica count to 0 for logstash indices from 2014-10-14 through 2014-10-29. See https://phabricator.wikimedia.org/P73 for the commands. |
[production] |
| 16:45 |
<ottomata> |
preparing to upgrade analytics1026 to trusty |
[production] |
| 16:21 |
<bd808> |
disk utilization is 94% on logstash1002, 92% on logstash1001 and 91% on logstash1003. Too much data in indices even with replica count bumped down to 1 for the small disks we have today. |
[production] |
| 16:16 |
<bd808> |
logstash elasticsearch cluster is pretty messed up. logstash1002 has lost shards for all indices except for today, and it's master for that one. |
[production] |
| 16:16 |
<manybubbles> |
Synchronized php-1.25wmf8/extensions/CirrusSearch/: SWAT update cirrussearch to fix slow prefix queries (duration: 00m 05s) |
[production] |
| 16:14 |
<manybubbles> |
Synchronized wmf-config/CirrusSearch-production.php: SWAT reenable regex search now that it will not crash elasticsearch (duration: 00m 04s) |
[production] |
| 16:13 |
<manybubbles> |
Synchronized wmf-config/CirrusSearch-common.php: SWAT reenable accelerated regex search (regex search still disabled) (duration: 00m 03s) |
[production] |
| 16:11 |
<manybubbles> |
Synchronized wmf-config/InitialiseSettings.php: SWAT force summary when running checkuser query on all wikis (duration: 00m 04s) |
[production] |
| 16:01 |
<manybubbles> |
Synchronized wmf-config/InitialiseSettings.php: SWAT revert JPG thumbnail chaining on all wikis except commons (duration: 00m 05s) |
[production] |
| 15:27 |
<hashar:> |
deleted all content from https://doc.wikimedia.org/ :-( Will regenerate. |
[production] |
| 15:09 |
<godog> |
rolling restart of object-auditor in swift codfw/eqiad to pick up changes |
[production] |
| 15:06 |
<yurik> |
Synchronized php-1.25wmf8/extensions/ZeroPortal: updatidng ZeroPortal to master (duration: 01m 13s) |
[production] |
| 15:04 |
<chasemp> |
phabricator upgrades T1203 |
[production] |
| 14:43 |
<hashar:> |
restarted zuul-merger on gallium |
[production] |
| 14:42 |
<hashar:> |
restarting Jenkins and Zuul |
[production] |
| 12:45 |
<godog> |
investigating high iops on swift eqiad with paravoid, stopped object-auditor on ms-be1005 and ms-be1015 |
[production] |
| 11:09 |
<hashar> |
resurrected morebots in #wikimedia-operations (see [[Morebots]]). |
[production] |
| 11:08 |
<hashar> |
Killed Jenkins due to a deadlock |
[production] |
| 11:08 |
<hashar> |
Killing Jenkins due to a deadlock |
[production] |
| 02:52 |
<mutante> |
beta puppet freshness - UNKNOWN: No valid datapoints found .. since 13d |
[production] |
| 02:30 |
<LocalisationUpdate> |
completed (1.25wmf8) at 2014-11-13 02:30:00+00:00 |
[production] |
| 02:18 |
<LocalisationUpdate> |
completed (1.25wmf7) at 2014-11-13 02:18:44+00:00 |
[production] |
| 00:46 |
<mutante> |
thulium - Could not intern from pson: expected value in object at '"[PHP]\\n\\n; puppet:t'! |
[production] |