|
2014-11-14
§
|
| 21:47 |
<bd808> |
restarted /etc/init.d/ganglia-monitor on logstash1003 |
[production] |
| 18:44 |
<legoktm> |
running scripts to fix bug 72927 |
[production] |
| 18:01 |
<akosiaris> |
reimaging xenon |
[production] |
| 16:40 |
<bd808> |
Increased replica count from 0 to 2 for all logstash elasticsearch indices. Expect icinga warnings as replicas are populated. |
[production] |
| 15:56 |
<ottomata> |
upgrading analytics1024 to trusty |
[production] |
| 15:46 |
<ottomata> |
analytics1003 (a cisco) is acting crazy, stuck in some loop while trying to boot. Am attempting to fix with power cycle |
[production] |
| 14:35 |
<paravoid> |
cr1-ulsfo: setting up BGP with new transit provider |
[production] |
| 09:16 |
<hashar> |
Zuul is back |
[production] |
| 09:14 |
<hashar> |
Zuul is flapping |
[production] |
| 07:38 |
<jgage> |
logstash hosts: elasticsearch moved to bigger disks |
[production] |
| 04:36 |
<Tim> |
on mw1114 restarting hhvm |
[production] |
| 04:17 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Fri Nov 14 04:17:16 UTC 2014 (duration 17m 15s) |
[production] |
| 04:03 |
<jgage> |
logstash1002 migration to new md0 complete |
[production] |
| 02:30 |
<LocalisationUpdate> |
completed (1.25wmf8) at 2014-11-14 02:30:34+00:00 |
[production] |
| 02:28 |
<Tim> |
progressively increasing load on mw1114, attempting to reproduce the previous overload |
[production] |
| 02:19 |
<jgage> |
logstash1003 elasticsearch migration to new raid0 complete |
[production] |
| 02:18 |
<LocalisationUpdate> |
completed (1.25wmf7) at 2014-11-14 02:18:01+00:00 |
[production] |
| 01:39 |
<kaldari> |
Synchronized wmf-config/mobile.php: updating WikiGrok A/B test times (duration: 00m 03s) |
[production] |
| 01:04 |
<kaldari> |
Synchronized php-1.25wmf7/extensions/MobileFrontend: (no message) (duration: 00m 05s) |
[production] |
| 01:04 |
<kaldari> |
Synchronized php-1.25wmf7/extensions/WikiGrok: (no message) (duration: 00m 03s) |
[production] |
| 00:49 |
<jgage> |
logstash1003: migrating elasticsearch data to new raid volume |
[production] |
| 00:42 |
<kaldari> |
Synchronized wmf-config/mobile.php: Update WikiGrok A/B test times (duration: 00m 03s) |
[production] |
| 00:20 |
<demon> |
Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 06s) |
[production] |
| 00:15 |
<mutante> |
nickel - shutdown |
[production] |
| 00:15 |
<demon> |
Synchronized php-1.25wmf8/extensions/VisualEditor: (no message) (duration: 00m 04s) |
[production] |
| 00:14 |
<demon> |
Synchronized php-1.25wmf7/extensions/VisualEditor: (no message) (duration: 00m 05s) |
[production] |
| 00:08 |
<demon> |
Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 04s) |
[production] |
| 00:06 |
<demon> |
Synchronized wmf-config/: (no message) (duration: 00m 07s) |
[production] |
| 00:04 |
<demon> |
Synchronized php-1.25wmf8/extensions/Echo: (no message) (duration: 00m 04s) |
[production] |
| 00:04 |
<demon> |
Synchronized php-1.25wmf7/extensions/Echo: (no message) (duration: 00m 06s) |
[production] |
|
2014-11-13
§
|
| 23:55 |
<mutante> |
nickel - remove from puppet,salt,icinga,stop services... |
[production] |
| 23:52 |
<^d> |
restarted gitblit on antimony |
[production] |
| 23:39 |
<kaldari> |
Synchronized wmf-config/mobile.php: Adding WikiGrok A/B test start and end times (duration: 00m 03s) |
[production] |
| 22:19 |
<jgage> |
hadoop: analytics1010 is again active namenode |
[production] |
| 22:14 |
<awight> |
Synchronized php-1.25wmf8/extensions/CentralNotice: push CentralNotice updates (duration: 00m 04s) |
[production] |
| 22:13 |
<awight> |
Synchronized php-1.25wmf7/extensions/CentralNotice: push CentralNotice updates (duration: 00m 05s) |
[production] |
| 22:12 |
<qchris> |
restarted EventLogging jobs that write to disk, to pick up config changes |
[production] |
| 22:03 |
<jgage> |
failed over hadoop namenode to analytics1004 |
[production] |
| 21:42 |
<awight> |
Synchronized wmf-config: Enabling CentralNotice banner choice on testwiki, take 2 (duration: 00m 06s) |
[production] |
| 21:15 |
<cscott> |
updated Parsoid to version dabff010 |
[production] |
| 20:51 |
<cmjohnson> |
powering down logstash1001 to add disks |
[production] |
| 20:39 |
<cmjohnson> |
powering down logstash1002 to add disks |
[production] |
| 20:37 |
<awight> |
CentralNotice noops deployed to all wikis |
[production] |
| 20:36 |
<awight> |
Synchronized php-1.25wmf7/extensions/CentralNotice: push CentralNotice updates (duration: 00m 05s) |
[production] |
| 20:33 |
<awight> |
Synchronized wmf-config: Enabling CentralNotice banner choice on testwiki (duration: 00m 04s) |
[production] |
| 20:32 |
<bd808> |
Dropped replica count of all logstash indices except today to 0. Should make rolling restarts faster during hardware upgrade. |
[production] |
| 20:25 |
<awight> |
Synchronized php-1.25wmf8/extensions/CentralNotice: push CentralNotice updates (duration: 00m 05s) |
[production] |
| 20:19 |
<csteipp> |
patched bugs 71111 and 71394 in wmf7 and wmf8 |
[production] |
| 20:14 |
<cmjohnson> |
powering down logstash1003 for a few mins to add disks |
[production] |
| 19:52 |
<ottomata> |
starting upgrade to trusty on analytics1023 |
[production] |