|
2008-11-06
§
|
| 17:23 |
<brion> |
restarting apache on srv47, seem smysteriously stuck |
[production] |
| 17:15 |
<brion> |
setting $wgMaxAnimatedGifArea to 1 to prevent animated thumbnailing of GIFs for now, see if that helps |
[production] |
| 17:10 |
<brion> |
river complaining of image scaler issues -- load spikes, depooling? |
[production] |
| 01:42 |
<Tim> |
restarting squids |
[production] |
| 01:42 |
<mark> |
Setup lvs4 as temp LVS support for lvs2, balancing the load |
[production] |
| 01:07 |
<brion> |
updated ContributionReporting to add paging links to ContributionHistory (might be a little funky w/ caching, we'll work it out :) |
[production] |
| 00:45 |
<Tim> |
progressively clearing /a on the remaining image scalers |
[production] |
| 00:37 |
<Tim> |
wiping /a on srv44 |
[production] |
| 00:00 |
<brion> |
scapping for update to ContributionReporting |
[production] |
|
2008-11-05
§
|
| 23:38 |
<brion> |
set yongle to restart apache every hour since it still seems to bork up and get stuck sometimes |
[production] |
| 22:01 |
<RobH> |
srv100 rebooted, was down. |
[production] |
| 18:28 |
<mark> |
tech team is procrastinating |
[production] |
| 18:16 |
<atglenn> |
added dhelps to office@wikimedia.org alias, redirected office@wikipedia.org to him also |
[production] |
| 18:14 |
<brion> |
disabling centralnotice on private wikis, we don't need to be told to donate to ourselves ;) |
[production] |
| 18:03 |
<brion> |
poking sitenotices off wikibooks, on *.wikipedia |
[production] |
| 18:03 |
<brion> |
set up ariel on mchenry for mail admin |
[production] |
| 05:38 |
<brion_> |
opera users may rejoice ;) |
[production] |
| 05:38 |
<brion_> |
tweaked storage1 lighttpd config so centralnotice.js is served with utf-8 charset |
[production] |
| 05:17 |
<brion_> |
for reference -- load spikes are page rendering on enwiki and dewiki mostly :) |
[production] |
| 05:17 |
<brion_> |
bumping enwiki notice to 100% |
[production] |
| 05:07 |
<Tim> |
killed various mysqld_safe processes which were using 100% CPU on ES servers |
[production] |
| 04:50 |
<brion_> |
fixed morebots -- bots now allowed to edit again at wikitech |
[production] |
| 04:50 |
<brion_> |
enabling enwiki notice at about 10% sampling |
[production] |
| 03:27 |
<brion_> |
squids are... i think.... looking better :D |
[production] |
| 02:47 |
<brion_> |
seems happier after restart of front-end squids |
[production] |
| 02:43 |
<brion_> |
tim's doing hard restarts of more squids, we're kinda offline briefly |
[production] |
| 02:34 |
<brion_> |
disabling centralnotices on remaining sites just for good measure while we debug |
[production] |
| 02:29 |
<brion_> |
current status: the squids which borked are still kind of borked, but perhaps slightly better. mark is examining squid memory reports |
[production] |
| 02:14 |
<brion> |
tim's attempting to restart borked squids |
[production] |
| 02:01 |
<brion> |
disabling enwiki centralnotice while investigating hits dropoff |
[production] |
|
2008-11-04
§
|
| 21:36 |
<Tim> |
added nagios monitoring of HTTP on image backends |
[production] |
| 21:14 |
<Tim> |
installed NRPE stuff on db19 |
[production] |
| 19:37 |
<Tim> |
killed the broken NFS mount on db21:/mnt with umount -l. The processes that are waiting for it will probably hang until system restart |
[production] |
| 18:33 |
<brion_> |
enabling ja-wikipedia notice for testing :D |
[production] |
| 18:32 |
<Tim> |
installed nagios stuff on db21,db22,db23 |
[production] |
| 18:27 |
<Tim> |
srv104 done, cluster18 re-added to the write list |
[production] |
| 18:15 |
<Tim> |
installed NRPE on srv159,srv171,srv183 |
[production] |
| 17:25 |
<domas> |
bounced db16 after jfs deadlock |
[production] |
| 17:24 |
<brion> |
settin' centralnotice on wikibooks to test, should show up in a few minutes |
[production] |
|
2008-11-03
§
|
| 22:05 |
<brion> |
enabled extra file upload types for foundationwiki, since it's restricted-write-access |
[production] |
| 20:47 |
<brion> |
Special:ContributionTracking form submission intermediary live on foundationwiki |
[production] |
| 20:33 |
<brion> |
scapping for ContribtionTracking extension |
[production] |
| 19:59 |
<brion> |
enabled mp3 and aiff uploads for private wikis so jay can upload some radio PSAs for fundraiser |
[production] |
| 19:46 |
<brion> |
poking $wgSquidMaxage from 31 days to 1 hour on wikimediafoundation.org, since templates and funkypage URLs may do funky things and not get purged (extra parameters) |
[production] |
| 19:32 |
<brion> |
note there's no notice up yet ;) |
[production] |
| 19:32 |
<brion> |
enabling centralnotice loader on all wikis |
[production] |
| 11:00 |
<domas> |
''mount -o remount,nobarrier /a'' on db15, observed 20x more performance. I am an idiot. :) |
[production] |
| 02:36 |
<brion-away> |
got a test centralnotice notice running on test.wikipedia.org. rock on |
[production] |
| 02:23 |
<Krimpet> |
Hi, Mom!!! :D |
[production] |
| 02:18 |
<brion> |
set up every-10-minute cronjob on zwinger to regen the centralnotice template JS files |
[production] |