|
2009-11-18
§
|
| 21:41 |
<Andrew> |
Scapping to deploy updates. |
[production] |
| 21:41 |
<Andrew> |
... and merging r59218 into LiquidThreads |
[production] |
| 21:40 |
<Andrew> |
Updating LiquidThreads_alpha to trunk state, and merging rrm: cannot remove directory `.svn/tmp/prop-base': Permission denied |
[production] |
| 21:14 |
<catrope> |
ran sync-common-all |
[production] |
| 21:12 |
<RoanKattouw> |
Deploying r59214, r59217 |
[production] |
| 20:57 |
<RoanKattouw> |
Deploying r59212 |
[production] |
| 20:33 |
<catrope> |
ran sync-common-all |
[production] |
| 20:31 |
<RoanKattouw> |
sync-common-all throwing lots of rsync errors "failed to set times on dirname/.svn : Operation not permitted" |
[production] |
| 20:30 |
<RoanKattouw> |
Deploying usability fixes for real this time |
[production] |
| 20:04 |
<RoanKattouw> |
Deploying usability fixes |
[production] |
| 20:03 |
<tfinc> |
synchronized php-1.5/extensions/ContributionReporting/FundraiserStatistics_body.php |
[production] |
| 20:02 |
<tfinc> |
synchronized php-1.5/extensions/ContributionReporting/FundraiserStatistics.css |
[production] |
| 19:50 |
<andrew> |
synchronized php-1.5/wmf-config/liquidthreads.php 'Remove debugging code' |
[production] |
| 19:31 |
<RoanKattouw> |
Non-roots can't compile texvc on at least srv124, which breaks sync-common on that server and scap for non-roots |
[production] |
| 19:28 |
<RoanKattouw> |
Syncing srv124 (test.wikipedia.org) |
[production] |
| 19:12 |
<RoanKattouw> |
Running svn up on test for real this time, in /home instead of /apache |
[production] |
| 18:56 |
<RoanKattouw> |
Running svn up on test for usability deployment |
[production] |
| 17:59 |
<andrew> |
synchronized php-1.5/wmf-config/liquidthreads.php 'Debugging' |
[production] |
| 17:58 |
<andrew> |
synchronized php-1.5/extensions/LiquidThreads_alpha/classes/View.php 'More debugging' |
[production] |
| 17:23 |
<andrew> |
synchronized php-1.5/extensions/LiquidThreads_alpha/classes/View.php '-m Debugging code' |
[production] |
| 06:24 |
<Tim> |
running logrotate -f on db20 |
[production] |
| 05:39 |
<Tim> |
logrotate on db20 has been broken since October 23, now the root partition is full. Fixing. |
[production] |
| 02:55 |
<tstarling> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'disabling the ClickTracking extension due to scary SQL escaping practices' |
[production] |
| 00:01 |
<Tim> |
on ms1, changed the max-age for centralnotice/images to 3600, max-age for the rest of centralnotice remains at 300. |
[production] |
| 00:01 |
<andrew> |
synchronized php-1.5/extensions/LiquidThreads/lqt.js 'Merged r59190' |
[production] |
|
2009-11-17
§
|
| 23:59 |
<andrew> |
synchronized php-1.5/extensions/LiquidThreads_alpha/lqt.js 'Merged r59189' |
[production] |
| 23:36 |
<Tim> |
deploying squid configuration (second attempt) |
[production] |
| 23:32 |
<Tim> |
squid conf had a syntax error in it, all upload squids crashed, mostly recovered now |
[production] |
| 23:21 |
<Tim> |
deploying an aggressive squid refresh_pattern (ignore-reload) for http://upload.wikimedia.org/centralnotice/images |
[production] |
| 20:50 |
<Andrew> |
Scapping to deploy LiquidThreads alpha updates, let us hope the site does not go downa gain :) |
[production] |
| 19:58 |
<domas> |
frontend squids occasionally have 100% cpu usage |
[production] |
| 19:57 |
<domas> |
http://p.defau.lt/?QsA_sTZ_ykf0R_t0dnNtuQ <--- ipvs state during the overload |
[production] |
| 19:55 |
<domas> |
hawthorn hitting %si capacity limits after +50% fundraiser image surge |
[production] |
| 18:01 |
<midom> |
synchronized php-1.5/wmf-config/db.php 're-adding db26, ixia' |
[production] |
| 02:13 |
<tfinc> |
synchronized php-1.5/wmf-config/CommonSettings.php |
[production] |
| 01:16 |
<tfinc> |
synchronized php-1.5/extensions/ContributionReporting/ContributionReporting.php 'picking up 2009 time range' |
[production] |
|
2009-11-16
§
|
| 23:27 |
<tfinc> |
synchronized php-1.5/wmf-config/CommonSettings.php 'updating for 2009 figures' |
[production] |
| 22:44 |
<midom> |
synchronized php-1.5/wmf-config/db.php |
[production] |
| 22:02 |
<Fred> |
enabled raw data gathering from the squids on Support pages (on Locke) |
[production] |
| 19:41 |
<midom> |
synchronized php-1.5/wmf-config/db.php 'taking out db18 and db27 for reimaging' |
[production] |
| 19:03 |
<Fred> |
root=fred above |
[production] |
| 19:03 |
<root> |
synchronized php-1.5/wmf-config/CommonSettings.php 'Added CH and NL landing page for geo-location' |
[production] |
| 17:24 |
<domas> |
increased file/connection limits on ms3 |
[production] |
| 14:39 |
<Andrew> |
To clarify, software update was NOT reverted, as the issue was due to the scap itself. |
[production] |
| 14:27 |
<Andrew> |
Site seems to be mostly back up by now. |
[production] |
| 14:24 |
<Andrew> |
Site seems to be up again, mostly. |
[production] |
| 14:18 |
<Andrew> |
Looks like the error was caused by the scap pushing 4-cpu apaches into swap, causing memcached nodes to fall over, resulting in higher than normal database traffic |
[production] |
| 14:12 |
<Andrew> |
Reverting software update |
[production] |
| 14:12 |
<Andrew> |
Reports of slowness and down-ness |
[production] |
| 14:07 |
<Andrew> |
Scapping to apply r59136, r59135, r59133, r59127 |
[production] |