|
2014-07-07
§
|
| 15:49 |
<bd808> |
Logstash event volume looks better after restart. Probably related to bug 63490. |
[production] |
| 15:33 |
<bd808> |
Restarted logstash on logstash1001 because log volume looked lower than I though it should be. |
[production] |
| 15:16 |
<cmjohnson1> |
reseating PEM2 cr1-eqiad |
[production] |
| 15:08 |
<godog> |
powercycled ms-be1007, unresponsive on console and remnants of a stack trace |
[production] |
| 14:49 |
<manybubbles> |
(Cirrus) Applying cache warmer configuration that went out last Thursday to all wikipedias. |
[production] |
| 12:11 |
<hashar> |
Jenkins job builder e1ddd23 fails for us :/ Moving back to parent commit |
[production] |
| 12:09 |
<hashar> |
Updated our Jenkins job builder fork 0972985..e1ddd23 |
[production] |
| 09:40 |
<godog> |
upgrade ms-be1003/1004/1012 (zone2) to swift icehouse |
[production] |
| 09:16 |
<_joe_> |
restarting rhenium, pings but no ssh since 2 days, serial console is blank and unresponsive |
[production] |
| 09:15 |
<godog> |
upgrade ms-be1002/1008 (zone1) to swift icehouse |
[production] |
| 02:53 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Mon Jul 7 02:52:10 UTC 2014 (duration 52m 9s) |
[production] |
| 02:24 |
<LocalisationUpdate> |
completed (1.24wmf12) at 2014-07-07 02:23:48+00:00 |
[production] |
| 02:13 |
<LocalisationUpdate> |
completed (1.24wmf11) at 2014-07-07 02:12:44+00:00 |
[production] |
|
2014-07-03
§
|
| 23:40 |
<mutante> |
osmium - libboost-dev : Depends: libboost1.54-dev but it is not going to be installed |
[production] |
| 23:33 |
<mutante> |
rhenium (pmacct / flow) Out of memory: Kill process 3123 (pmacctd) score 1 or sacrifice child |
[production] |
| 23:22 |
<K4-713> |
updated payments to c5689f385b2f0a7bdc55c5752010e9eb |
[production] |
| 23:17 |
<mwalker> |
Synchronized php-1.24wmf12/extensions/VisualEditor/: Updating VisualEditor for {{gerrit|144081}} (duration: 00m 12s) |
[production] |
| 21:07 |
<oblivian> |
gracefulled all apaches |
[production] |
| 20:45 |
<mutante> |
deleted analytics/kraken branch from ops/puppet via gerrit ui, ack'ed by ottomata |
[production] |
| 20:12 |
<bd808|deploy> |
Updated scap to ff04431 (restart-nutcracker script) |
[production] |
| 19:53 |
<reedy> |
Synchronized wmf-config/InitialiseSettings-labs.php: (no message) (duration: 00m 14s) |
[production] |
| 19:48 |
<reedy> |
Synchronized wmf-config/: (no message) (duration: 00m 14s) |
[production] |
| 19:48 |
<reedy> |
Synchronized docroot and w: (no message) (duration: 00m 30s) |
[production] |
| 19:47 |
<reedy> |
Synchronized database lists: (no message) (duration: 00m 20s) |
[production] |
| 19:36 |
<jgage> |
rebooting analytics1012 for bios change: cpufreq governor |
[production] |
| 19:27 |
<ottomata> |
disabling puppet on hadoop related analytics nodes, preparing for reinstall |
[production] |
| 19:21 |
<reedy> |
Synchronized wmf-config/: (no message) (duration: 00m 14s) |
[production] |
| 19:16 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.24wmf12 |
[production] |
| 19:12 |
<reedy> |
Synchronized php-1.24wmf11/languages/Language.php: I039547b867b2eab47692dcc018c95b89975bc65d (duration: 00m 40s) |
[production] |
| 18:49 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedia to 1.24wmf11 |
[production] |
| 18:41 |
<reedy> |
Finished scap: testwiki to 1.24wmf12 and build l10n cache (duration: 30m 47s) |
[production] |
| 18:18 |
<ottomata> |
doing rolling restarts of zookeeper servers and kafka brokers to load up new zk timeout changes |
[production] |
| 18:10 |
<reedy> |
Started scap: testwiki to 1.24wmf12 and build l10n cache |
[production] |
| 18:10 |
<reedy> |
scap aborted: testwiki to 1.24wmf12 and build l10n cache (duration: 27m 26s) |
[production] |
| 17:53 |
<godog> |
reloading librenms, semi-broke it with a syslog search (again) |
[production] |
| 17:46 |
<godog> |
reloading librenms, semi-broke it with a syslog search |
[production] |
| 17:42 |
<reedy> |
Started scap: testwiki to 1.24wmf12 and build l10n cache |
[production] |