2015-01-03
§
|
22:38 |
<YuviPanda> |
restarted parsoid on wtp1010 |
[production] |
22:38 |
<YuviPanda> |
restarted parsoid on wtp1006 |
[production] |
22:37 |
<YuviPanda> |
restarted parsoid on wtp1004 |
[production] |
22:29 |
<subbu> |
hotfix synced to parsoid cores (to return 500 for urwiki:نام_مقامات_اے); restart coming next |
[production] |
22:15 |
<YuviPanda> |
restarted parsoid on wtp* hosts agian |
[production] |
21:19 |
<YuviPanda> |
restarting parsoid on wtp* hosts again |
[production] |
20:46 |
<YuviPanda> |
restarting parsoid on wtp* again |
[production] |
20:29 |
<YuviPanda> |
manually restarted parsoid on wtp1012 |
[production] |
20:12 |
<YuviPanda> |
restarting parsoid on all wtp* hosts |
[production] |
20:06 |
<YuviPanda> |
restarting parsoid on wtp1008 |
[production] |
17:13 |
<_joe_> |
restarting parsoid across the cluster |
[production] |
2014-12-31
§
|
23:45 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool db1061 (duration: 00m 05s) |
[production] |
14:02 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool db1065, warm up (duration: 00m 06s) |
[production] |
09:47 |
<godog> |
updating precise-wikimedia from third-party repo (hwraid) |
[production] |
09:45 |
<godog> |
previous reprepro update also accidentally updated elasticsearch in trusty-wikimedia to 1.3.7 |
[production] |
09:43 |
<godog> |
updating trusty-wikimedia from third-party repo (hwraid) |
[production] |
02:22 |
<springle> |
upgrade db1065 trusty |
[production] |
02:16 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool db1065 (duration: 00m 05s) |
[production] |
02:03 |
<awight> |
updated payments from 78b72063e4e0cc76b7e168be1e626d5e10e34d4a to 62c81d4574e5e994ff8f3cac7115eff335bd5265 |
[production] |
00:52 |
<bd808> |
restarted elasticsearch on logstash1001 |
[production] |
00:49 |
<awight> |
updated payments from e81f473acc5b31b49dd27714c40f9b71c3462e26 to 78b72063e4e0cc76b7e168be1e626d5e10e34d4a |
[production] |
00:42 |
<bd808> |
log2udp events still not making it into logstash; possibly related to earlier elasticsearch cluster issues; I don't want to restart elasticsearch on logstash1001 while the cluster is still recovering form that. |
[production] |
00:33 |
<bd808> |
restarted logstash on logstash1001; log2udp events not being recorded in elasticsearch |
[production] |
2014-12-27
§
|
23:33 |
<paravoid> |
restarting puppetmasters |
[production] |
20:29 |
<gwicke> |
dropped old keyspaces titan{,2,3} on xenon to free space for titan4 |
[production] |
19:53 |
<ori> |
gallium: restarted jenkins |
[production] |
16:19 |
<Reedy> |
jenkins started again... |
[production] |
16:17 |
<Reedy> |
jenkins killed |
[production] |
16:12 |
<Reedy> |
attempting to kill jenkins |
[production] |
16:11 |
<Reedy> |
jenkins is hung with high cpu/memory usage |
[production] |
12:55 |
<springle> |
Synchronized wmf-config/db-eqiad.php: bump up s1 api load sent to db1066 (duration: 00m 06s) |
[production] |
12:11 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool db1066 and db1028, warm up (duration: 00m 06s) |
[production] |
05:49 |
<ori> |
cerium disk space critical, so moved /mnt/data/cassandra/java_{1418354329,1418533386,1418537719}.hprof to /tmp/hprof_files, freeing up ~17G of space. |
[production] |