|
2009-11-17
§
|
| 23:36 |
<Tim> |
deploying squid configuration (second attempt) |
[production] |
| 23:32 |
<Tim> |
squid conf had a syntax error in it, all upload squids crashed, mostly recovered now |
[production] |
| 23:21 |
<Tim> |
deploying an aggressive squid refresh_pattern (ignore-reload) for http://upload.wikimedia.org/centralnotice/images |
[production] |
| 20:50 |
<Andrew> |
Scapping to deploy LiquidThreads alpha updates, let us hope the site does not go downa gain :) |
[production] |
| 19:58 |
<domas> |
frontend squids occasionally have 100% cpu usage |
[production] |
| 19:57 |
<domas> |
http://p.defau.lt/?QsA_sTZ_ykf0R_t0dnNtuQ <--- ipvs state during the overload |
[production] |
| 19:55 |
<domas> |
hawthorn hitting %si capacity limits after +50% fundraiser image surge |
[production] |
| 18:01 |
<midom> |
synchronized php-1.5/wmf-config/db.php 're-adding db26, ixia' |
[production] |
| 02:13 |
<tfinc> |
synchronized php-1.5/wmf-config/CommonSettings.php |
[production] |
| 01:16 |
<tfinc> |
synchronized php-1.5/extensions/ContributionReporting/ContributionReporting.php 'picking up 2009 time range' |
[production] |
|
2009-11-16
§
|
| 23:27 |
<tfinc> |
synchronized php-1.5/wmf-config/CommonSettings.php 'updating for 2009 figures' |
[production] |
| 22:44 |
<midom> |
synchronized php-1.5/wmf-config/db.php |
[production] |
| 22:02 |
<Fred> |
enabled raw data gathering from the squids on Support pages (on Locke) |
[production] |
| 19:41 |
<midom> |
synchronized php-1.5/wmf-config/db.php 'taking out db18 and db27 for reimaging' |
[production] |
| 19:03 |
<Fred> |
root=fred above |
[production] |
| 19:03 |
<root> |
synchronized php-1.5/wmf-config/CommonSettings.php 'Added CH and NL landing page for geo-location' |
[production] |
| 17:24 |
<domas> |
increased file/connection limits on ms3 |
[production] |
| 14:39 |
<Andrew> |
To clarify, software update was NOT reverted, as the issue was due to the scap itself. |
[production] |
| 14:27 |
<Andrew> |
Site seems to be mostly back up by now. |
[production] |
| 14:24 |
<Andrew> |
Site seems to be up again, mostly. |
[production] |
| 14:18 |
<Andrew> |
Looks like the error was caused by the scap pushing 4-cpu apaches into swap, causing memcached nodes to fall over, resulting in higher than normal database traffic |
[production] |
| 14:12 |
<Andrew> |
Reverting software update |
[production] |
| 14:12 |
<Andrew> |
Reports of slowness and down-ness |
[production] |
| 14:07 |
<Andrew> |
Scapping to apply r59136, r59135, r59133, r59127 |
[production] |
| 12:24 |
<andrew> |
synchronized php-1.5/extensions/ProofreadPage/ProofreadPage.php 'Merge r58865 and r59070 from trunk -- fixes for deferred updates.' |
[production] |
| 12:12 |
<Andrew> |
scapping to update LiquidThreads alpha |
[production] |
| 07:35 |
<tomaszf> |
adding backup cron on db10 for civi and grosley for /srv |
[production] |
|
2009-11-13
§
|
| 23:21 |
<tfinc> |
synchronized php-1.5/extensions/GeoLite/GeoLite.php |
[production] |
| 22:55 |
<tfinc> |
synchronized php-1.5/extensions/GeoLite/GeoLite_body.php |
[production] |
| 21:45 |
<Andrew> |
Scapping to update LiquidThreads alpha to LiquidThreads trunk state |
[production] |
| 17:06 |
<mark> |
Deployed new sudoers file on the search servers |
[production] |
| 15:53 |
<andrew> |
synchronized php-1.5/extensions/LiquidThreads_alpha/classes/Dispatch.php 'Updating alpha version to trunk state' |
[production] |
| 15:52 |
<andrew> |
synchronized php-1.5/extensions/LiquidThreads_alpha/classes/ParserFunctions.php 'Updating alpha version to trunk state' |
[production] |
| 12:48 |
<mark> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'Use the newly setup proxy for upload-by-url' |
[production] |
| 00:31 |
<robh> |
synchronized php-1.5/cache/interwiki.cdb 'added outreach to the interwiki map' |
[production] |