|
2008-11-05
§
|
| 23:38 |
<brion> |
set yongle to restart apache every hour since it still seems to bork up and get stuck sometimes |
[production] |
| 22:01 |
<RobH> |
srv100 rebooted, was down. |
[production] |
| 18:28 |
<mark> |
tech team is procrastinating |
[production] |
| 18:16 |
<atglenn> |
added dhelps to office@wikimedia.org alias, redirected office@wikipedia.org to him also |
[production] |
| 18:14 |
<brion> |
disabling centralnotice on private wikis, we don't need to be told to donate to ourselves ;) |
[production] |
| 18:03 |
<brion> |
poking sitenotices off wikibooks, on *.wikipedia |
[production] |
| 18:03 |
<brion> |
set up ariel on mchenry for mail admin |
[production] |
| 05:38 |
<brion_> |
opera users may rejoice ;) |
[production] |
| 05:38 |
<brion_> |
tweaked storage1 lighttpd config so centralnotice.js is served with utf-8 charset |
[production] |
| 05:17 |
<brion_> |
for reference -- load spikes are page rendering on enwiki and dewiki mostly :) |
[production] |
| 05:17 |
<brion_> |
bumping enwiki notice to 100% |
[production] |
| 05:07 |
<Tim> |
killed various mysqld_safe processes which were using 100% CPU on ES servers |
[production] |
| 04:50 |
<brion_> |
fixed morebots -- bots now allowed to edit again at wikitech |
[production] |
| 04:50 |
<brion_> |
enabling enwiki notice at about 10% sampling |
[production] |
| 03:27 |
<brion_> |
squids are... i think.... looking better :D |
[production] |
| 02:47 |
<brion_> |
seems happier after restart of front-end squids |
[production] |
| 02:43 |
<brion_> |
tim's doing hard restarts of more squids, we're kinda offline briefly |
[production] |
| 02:34 |
<brion_> |
disabling centralnotices on remaining sites just for good measure while we debug |
[production] |
| 02:29 |
<brion_> |
current status: the squids which borked are still kind of borked, but perhaps slightly better. mark is examining squid memory reports |
[production] |
| 02:14 |
<brion> |
tim's attempting to restart borked squids |
[production] |
| 02:01 |
<brion> |
disabling enwiki centralnotice while investigating hits dropoff |
[production] |
|
2008-11-04
§
|
| 21:36 |
<Tim> |
added nagios monitoring of HTTP on image backends |
[production] |
| 21:14 |
<Tim> |
installed NRPE stuff on db19 |
[production] |
| 19:37 |
<Tim> |
killed the broken NFS mount on db21:/mnt with umount -l. The processes that are waiting for it will probably hang until system restart |
[production] |
| 18:33 |
<brion_> |
enabling ja-wikipedia notice for testing :D |
[production] |
| 18:32 |
<Tim> |
installed nagios stuff on db21,db22,db23 |
[production] |
| 18:27 |
<Tim> |
srv104 done, cluster18 re-added to the write list |
[production] |
| 18:15 |
<Tim> |
installed NRPE on srv159,srv171,srv183 |
[production] |
| 17:25 |
<domas> |
bounced db16 after jfs deadlock |
[production] |
| 17:24 |
<brion> |
settin' centralnotice on wikibooks to test, should show up in a few minutes |
[production] |
|
2008-11-03
§
|
| 22:05 |
<brion> |
enabled extra file upload types for foundationwiki, since it's restricted-write-access |
[production] |
| 20:47 |
<brion> |
Special:ContributionTracking form submission intermediary live on foundationwiki |
[production] |
| 20:33 |
<brion> |
scapping for ContribtionTracking extension |
[production] |
| 19:59 |
<brion> |
enabled mp3 and aiff uploads for private wikis so jay can upload some radio PSAs for fundraiser |
[production] |
| 19:46 |
<brion> |
poking $wgSquidMaxage from 31 days to 1 hour on wikimediafoundation.org, since templates and funkypage URLs may do funky things and not get purged (extra parameters) |
[production] |
| 19:32 |
<brion> |
note there's no notice up yet ;) |
[production] |
| 19:32 |
<brion> |
enabling centralnotice loader on all wikis |
[production] |
| 11:00 |
<domas> |
''mount -o remount,nobarrier /a'' on db15, observed 20x more performance. I am an idiot. :) |
[production] |
| 02:36 |
<brion-away> |
got a test centralnotice notice running on test.wikipedia.org. rock on |
[production] |
| 02:23 |
<Krimpet> |
Hi, Mom!!! :D |
[production] |
| 02:18 |
<brion> |
set up every-10-minute cronjob on zwinger to regen the centralnotice template JS files |
[production] |
| 02:10 |
<brion> |
centralnotice .js file loader up on test and meta for poking at |
[production] |
| 01:12 |
<mark> |
level 3 blackholing of traffic disappeared, brought BGP sessions back up |
[production] |
| 00:59 |
<mark> |
shutdown BGP session to AS 30217, for blackholing of traffic behind it (L3?) |
[production] |
| 00:58 |
<brion> |
network problems at pmtpa |
[production] |