|
2009-10-23
§
|
| 22:36 |
<domas> |
db28 has multiple fan failures (LOM is finally able to do something :) - still needs datacenter ops |
[production] |
| 22:20 |
<domas> |
db28 is a toast, needs cold restart by datacenter ops, LOM not able to do anything |
[production] |
| 22:20 |
<midom> |
synchronized php-1.5/wmf-config/db.php 'db28 dead' |
[production] |
| 11:17 |
<domas> |
Fixed skip-list of cached query pages, was broken for past two months :) |
[production] |
| 10:54 |
<midom> |
synchronized php-1.5/thumb.php 'removing livehack' |
[production] |
| 10:52 |
<domas> |
rotating logs becomes difficult when they become too big, so they continue to grow indefinitely! db20 / nearly full, loooots of /var/log/remote ;-) |
[production] |
| 10:39 |
<domas> |
who watches the watchers? :) rrdtool process on spence was using 8G of memory. :-)))) |
[production] |
| 10:24 |
<domas> |
semaphore leaks made some apaches fail, failed apache in rendering farm was not depooled, thus having 404 handler serve plenty of "can't connect to host" broken thumbs. |
[production] |
| 10:13 |
<domas> |
apparently there're intermittent connection failures from ms4 to scalers |
[production] |
| 09:56 |
<midom> |
synchronized php-1.5/thumb.php 'error header livehack' |
[production] |
| 04:04 |
<domas> |
noticed intermittent network failure inside pmtpa |
[production] |
| 04:01 |
<domas> |
switched jobs table on db22 with an empty one, old one was having just few noop entries and five million invalidated rows... hit interesting (but probably easy to fix) performance problem at mtr_memo_release/mtr_commit code inside MySQL :) |
[production] |
| 03:17 |
<Fred> |
restarted powerdns on ns2 to kill some zombies with a double tap :p |
[production] |
|
2009-10-20
§
|
| 20:16 |
<RoanKattouw> |
Brion synced r57957 |
[production] |
| 19:59 |
<robh> |
synchronized php-1.5/wmf-config/InitialiseSettings.php '21023 Allow bureaucrats to remove sysop flag on Simple English WIkiquote' |
[production] |
| 15:54 |
<hcatlin> |
Deployed changes to S60. Stopping redirect for Nokia Series60, because it needs more work. |
[production] |
| 14:38 |
<andrew> |
synchronized php-1.5/extensions/LiquidThreads/api/ApiThreadAction.php 'Merge r57945' |
[production] |
| 14:32 |
<Andrew> |
Updating LiquidThreads again, scapping |
[production] |
| 14:03 |
<Andrew> |
Updating LiquidThreads, scapping |
[production] |
| 12:20 |
<hcatlin> |
Updated Common.js on en.wiki to add in redirects for the new batch of supported devices |
[production] |
| 11:57 |
<hcatlin> |
deploying updated software to mobile1 |
[production] |
| 04:39 |
<Tim> |
fixing static.wikipedia.org, broken due to lack of DumpHTML extension in wmf-deployment |
[production] |
| 00:42 |
<robh> |
synchronized php-1.5/wmf-config/InitialiseSettings.php '19865 en.Wikinews would like to make Vector the default skin' |
[production] |
| 00:37 |
<robh> |
synchronized php-1.5/wmf-config/CommonSettings.php 'Bug 20361' |
[production] |
|
2009-10-19
§
|
| 23:12 |
<andrew> |
synchronized php-1.5/extensions/LiquidThreads/api/ApiThreadAction.php 'Merge r57930' |
[production] |
| 23:05 |
<andrew> |
synchronized php-1.5/extensions/LiquidThreads/lqt.js 'Merge r57928' |
[production] |
| 23:05 |
<andrew> |
synchronized php-1.5/extensions/LiquidThreads/api/ApiThreadAction.php 'Merge r57928' |
[production] |
| 23:04 |
<andrew> |
synchronized php-1.5/extensions/LiquidThreads/classes/Thread.php 'Merge r57928' |
[production] |
| 22:52 |
<rainman-sr> |
starting complete rebuild of en.wp index |
[production] |
| 22:31 |
<atglenn> |
turned off replication on ms1 to ms5 (it was failing), running manually to catch up |
[production] |
| 22:27 |
<Fred> |
restarted apache on srv[105,117,173] |
[production] |
| 22:19 |
<andrew> |
synchronized php-1.5/extensions/LiquidThreads/lqt.js 'Merge r57924' |
[production] |
| 21:56 |
<Andrew> |
Scapping |
[production] |
| 21:44 |
<Andrew> |
Updating LiquidThreads to trunk state again for bugfixes and ajax improvements. |
[production] |
| 19:50 |
<rainman-sr> |
stopping search incremental updates to rebuild en.wp index for increased efficiency |
[production] |
| 19:39 |
<tstarling> |
synchronized php-1.5/wmf-config/InitialiseSettings.php |
[production] |
| 19:37 |
<tstarling> |
synchronized php-1.5/includes/User.php 'r57910' |
[production] |
| 19:36 |
<tstarling> |
synchronized php-1.5/includes/DefaultSettings.php 'r57910' |
[production] |
| 18:54 |
<tstarling> |
synchronized php-1.5/wmf-config/InitialiseSettings.php |
[production] |
| 18:47 |
<tstarling> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'rate limit exclusion for seminar' |
[production] |
| 14:50 |
<andrew> |
synchronized php-1.5/wmf-config/mc.php 'Swap out srv156' |
[production] |
| 14:29 |
<Andrew> |
Wikitech is insanely slow :) |
[production] |