|
2014-11-14
§
|
| 09:14 |
<hashar> |
Zuul is flapping |
[production] |
| 07:38 |
<jgage> |
logstash hosts: elasticsearch moved to bigger disks |
[production] |
| 04:36 |
<Tim> |
on mw1114 restarting hhvm |
[production] |
| 04:17 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Fri Nov 14 04:17:16 UTC 2014 (duration 17m 15s) |
[production] |
| 04:03 |
<jgage> |
logstash1002 migration to new md0 complete |
[production] |
| 02:30 |
<LocalisationUpdate> |
completed (1.25wmf8) at 2014-11-14 02:30:34+00:00 |
[production] |
| 02:28 |
<Tim> |
progressively increasing load on mw1114, attempting to reproduce the previous overload |
[production] |
| 02:19 |
<jgage> |
logstash1003 elasticsearch migration to new raid0 complete |
[production] |
| 02:18 |
<LocalisationUpdate> |
completed (1.25wmf7) at 2014-11-14 02:18:01+00:00 |
[production] |
| 01:39 |
<kaldari> |
Synchronized wmf-config/mobile.php: updating WikiGrok A/B test times (duration: 00m 03s) |
[production] |
| 01:04 |
<kaldari> |
Synchronized php-1.25wmf7/extensions/MobileFrontend: (no message) (duration: 00m 05s) |
[production] |
| 01:04 |
<kaldari> |
Synchronized php-1.25wmf7/extensions/WikiGrok: (no message) (duration: 00m 03s) |
[production] |
| 00:49 |
<jgage> |
logstash1003: migrating elasticsearch data to new raid volume |
[production] |
| 00:42 |
<kaldari> |
Synchronized wmf-config/mobile.php: Update WikiGrok A/B test times (duration: 00m 03s) |
[production] |
| 00:20 |
<demon> |
Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 06s) |
[production] |
| 00:15 |
<mutante> |
nickel - shutdown |
[production] |
| 00:15 |
<demon> |
Synchronized php-1.25wmf8/extensions/VisualEditor: (no message) (duration: 00m 04s) |
[production] |
| 00:14 |
<demon> |
Synchronized php-1.25wmf7/extensions/VisualEditor: (no message) (duration: 00m 05s) |
[production] |
| 00:08 |
<demon> |
Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 04s) |
[production] |
| 00:06 |
<demon> |
Synchronized wmf-config/: (no message) (duration: 00m 07s) |
[production] |
| 00:04 |
<demon> |
Synchronized php-1.25wmf8/extensions/Echo: (no message) (duration: 00m 04s) |
[production] |
| 00:04 |
<demon> |
Synchronized php-1.25wmf7/extensions/Echo: (no message) (duration: 00m 06s) |
[production] |
|
2014-11-13
§
|
| 23:55 |
<mutante> |
nickel - remove from puppet,salt,icinga,stop services... |
[production] |
| 23:52 |
<^d> |
restarted gitblit on antimony |
[production] |
| 23:39 |
<kaldari> |
Synchronized wmf-config/mobile.php: Adding WikiGrok A/B test start and end times (duration: 00m 03s) |
[production] |
| 22:19 |
<jgage> |
hadoop: analytics1010 is again active namenode |
[production] |
| 22:14 |
<awight> |
Synchronized php-1.25wmf8/extensions/CentralNotice: push CentralNotice updates (duration: 00m 04s) |
[production] |
| 22:13 |
<awight> |
Synchronized php-1.25wmf7/extensions/CentralNotice: push CentralNotice updates (duration: 00m 05s) |
[production] |
| 22:12 |
<qchris> |
restarted EventLogging jobs that write to disk, to pick up config changes |
[production] |
| 22:03 |
<jgage> |
failed over hadoop namenode to analytics1004 |
[production] |
| 21:42 |
<awight> |
Synchronized wmf-config: Enabling CentralNotice banner choice on testwiki, take 2 (duration: 00m 06s) |
[production] |
| 21:15 |
<cscott> |
updated Parsoid to version dabff010 |
[production] |
| 20:51 |
<cmjohnson> |
powering down logstash1001 to add disks |
[production] |
| 20:39 |
<cmjohnson> |
powering down logstash1002 to add disks |
[production] |
| 20:37 |
<awight> |
CentralNotice noops deployed to all wikis |
[production] |
| 20:36 |
<awight> |
Synchronized php-1.25wmf7/extensions/CentralNotice: push CentralNotice updates (duration: 00m 05s) |
[production] |
| 20:33 |
<awight> |
Synchronized wmf-config: Enabling CentralNotice banner choice on testwiki (duration: 00m 04s) |
[production] |
| 20:32 |
<bd808> |
Dropped replica count of all logstash indices except today to 0. Should make rolling restarts faster during hardware upgrade. |
[production] |
| 20:25 |
<awight> |
Synchronized php-1.25wmf8/extensions/CentralNotice: push CentralNotice updates (duration: 00m 05s) |
[production] |
| 20:19 |
<csteipp> |
patched bugs 71111 and 71394 in wmf7 and wmf8 |
[production] |
| 20:14 |
<cmjohnson> |
powering down logstash1003 for a few mins to add disks |
[production] |
| 19:52 |
<ottomata> |
starting upgrade to trusty on analytics1023 |
[production] |
| 19:15 |
<awight> |
campaigns reenabled |
[production] |
| 18:55 |
<awight> |
disabling CentralNotice campaigns |
[production] |
| 17:49 |
<ottomata> |
preparing for trusty upgrade of analytics1003 |
[production] |
| 16:57 |
<bd808> |
dropped replica count to 0 for logstash indices from 2014-10-30 and 2014-10-31. |
[production] |
| 16:49 |
<bd808> |
restarted elasticsearch on logstash1002 |
[production] |
| 16:46 |
<bd808> |
dropped replica count to 0 for logstash indices from 2014-10-14 through 2014-10-29. See https://phabricator.wikimedia.org/P73 for the commands. |
[production] |
| 16:45 |
<ottomata> |
preparing to upgrade analytics1026 to trusty |
[production] |
| 16:21 |
<bd808> |
disk utilization is 94% on logstash1002, 92% on logstash1001 and 91% on logstash1003. Too much data in indices even with replica count bumped down to 1 for the small disks we have today. |
[production] |