2014-12-31
§
|
14:02 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool db1065, warm up (duration: 00m 06s) |
[production] |
09:47 |
<godog> |
updating precise-wikimedia from third-party repo (hwraid) |
[production] |
09:45 |
<godog> |
previous reprepro update also accidentally updated elasticsearch in trusty-wikimedia to 1.3.7 |
[production] |
09:43 |
<godog> |
updating trusty-wikimedia from third-party repo (hwraid) |
[production] |
02:22 |
<springle> |
upgrade db1065 trusty |
[production] |
02:16 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool db1065 (duration: 00m 05s) |
[production] |
02:03 |
<awight> |
updated payments from 78b72063e4e0cc76b7e168be1e626d5e10e34d4a to 62c81d4574e5e994ff8f3cac7115eff335bd5265 |
[production] |
00:52 |
<bd808> |
restarted elasticsearch on logstash1001 |
[production] |
00:49 |
<awight> |
updated payments from e81f473acc5b31b49dd27714c40f9b71c3462e26 to 78b72063e4e0cc76b7e168be1e626d5e10e34d4a |
[production] |
00:42 |
<bd808> |
log2udp events still not making it into logstash; possibly related to earlier elasticsearch cluster issues; I don't want to restart elasticsearch on logstash1001 while the cluster is still recovering form that. |
[production] |
00:33 |
<bd808> |
restarted logstash on logstash1001; log2udp events not being recorded in elasticsearch |
[production] |
2014-12-27
§
|
23:33 |
<paravoid> |
restarting puppetmasters |
[production] |
20:29 |
<gwicke> |
dropped old keyspaces titan{,2,3} on xenon to free space for titan4 |
[production] |
19:53 |
<ori> |
gallium: restarted jenkins |
[production] |
16:19 |
<Reedy> |
jenkins started again... |
[production] |
16:17 |
<Reedy> |
jenkins killed |
[production] |
16:12 |
<Reedy> |
attempting to kill jenkins |
[production] |
16:11 |
<Reedy> |
jenkins is hung with high cpu/memory usage |
[production] |
12:55 |
<springle> |
Synchronized wmf-config/db-eqiad.php: bump up s1 api load sent to db1066 (duration: 00m 06s) |
[production] |
12:11 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool db1066 and db1028, warm up (duration: 00m 06s) |
[production] |
05:49 |
<ori> |
cerium disk space critical, so moved /mnt/data/cassandra/java_{1418354329,1418533386,1418537719}.hprof to /tmp/hprof_files, freeing up ~17G of space. |
[production] |
2014-12-23
§
|
22:21 |
<hoo> |
Synchronized wmf-config/: Syncing Kaldari's beta-only change (duration: 00m 07s) |
[production] |
20:34 |
<K4-713> |
Re-enabled Thank You send job |
[production] |
19:53 |
<anomie> |
Synchronized wmf-config: Labs-only change (duration: 00m 06s) |
[production] |
19:43 |
<K4-713> |
disabled Thank You email send job |
[production] |
19:40 |
<chasemp> |
updated phab sprint app to 0.6.1.4 |
[production] |
18:32 |
<paravoid> |
restarting icing |
[production] |
15:06 |
<_joe_> |
gracefully reloading apache on palladium to clean up old puppet master instances |
[production] |
14:50 |
<_joe_> |
restarted apache on strontium to verify hiera is working |
[production] |
14:17 |
<godog> |
restart icinga on neon |
[production] |
08:30 |
<_joe_> |
restarting gitblit, stuck at 100% cpu on a thread |
[production] |
03:25 |
<andrewbogott> |
graceful'd apache2 on virt1000 (same intermittent passenger crash as always) |
[production] |