|
2009-04-05
§
|
| 19:45 |
<jeluf> |
synchronized php-1.5/mc-pmtpa.php 'replace srv92 b srv152' |
[production] |
| 15:07 |
<domas> |
Tim rocks, cluster back to normal |
[production] |
| 15:04 |
<Tim> |
deployed r49212 to fix infinite template recursion issue |
[production] |
| 11:24 |
<domas> |
restarted some fedora apaches, were stuck in write() after the previous hiccups |
[production] |
| 10:28 |
<mark> |
Restarted memcached on srv151 |
[production] |
| 08:59 |
<domas> |
I hate computers |
[production] |
| 08:28 |
<domas> |
hanging latex processes held :80, thus not allowing clean apache restarts on some nodes |
[production] |
| 08:18 |
<domas> |
restarted plentiful of crashlooping apaches, investigating resource consumption problem |
[production] |
| 05:53 |
<Andrew> |
srv183 back up and memcached running, moved it from DOWN to SPARE. Threads on db12 back below 1000 again (normal range). Somehow I just resolved my first site problems by myself :O |
[production] |
| 05:44 |
<andrew> |
synchronized php-1.5/mc-pmtpa.php |
[production] |
| 05:42 |
<Andrew> |
srv183 went down, and it's running memcached. Replaced it with srv61, the first one in the SPARE section of mc-pmtpa.php, and moved it out. Hoping I did this right, and that it helps with db12 overload. |
[production] |
| 05:30 |
<Andrew> |
db12 (enwiki master) overloaded. Nothing I can do about it. |
[production] |
| 01:52 |
<andrew> |
synchronized php-1.5/includes/specials/SpecialBlockip.php 'Live-merge r49191, fixes a bug that gets in the way of suppression' |
[production] |
|
2009-04-01
§
|
| 23:08 |
<Fred> |
restarted apache on srv99 |
[production] |
| 22:57 |
<mark> |
Restored session to AS 30217 as well |
[production] |
| 22:33 |
<mark> |
Brought session to AS 13680 back up |
[production] |
| 21:30 |
<mark> |
Shut down BGP sessions to AS 13680 and 30217 for what appears to be problems to/within Level 3 Tampa |
[production] |
| 16:10 |
<JeLuF> |
Image import of about 5000 images done. 245000 left to do... |
[production] |
| 15:53 |
<Fred> |
rebooting srv217 since it is wedged. |
[production] |
| 15:47 |
<Fred> |
restarted apache on srv137 |
[production] |
| 13:47 |
<andrew> |
synchronized php-1.5/InitialiseSettings.php 'Activating AbuseFilter on tpiwiki and hewiki, bugs 18299, 18300' |
[production] |
| 13:45 |
<mark> |
Rebooting srv217 |
[production] |
| 13:45 |
<andrew> |
synchronized php-1.5/CommonSettings.php 'AbuseFilter custom settings, hewiki' |
[production] |
| 13:40 |
<JeLuF> |
batch importing images from the Deutsche Fotothek, commons.wikimedia.org/wiki/Commons:Deutsche_Fotothek |
[production] |
| 13:11 |
<Andrew> |
Reports that Common.css/Common.js weren't working on hsbwiki. Manually purging http://hsb.wikipedia.org/w/index.php?title=-&action=raw&smaxage=0&gen=js&useskin=monobook on the command-line fixed the issue. |
[production] |
| 08:42 |
<andrew> |
synchronized php-1.5/InitialiseSettings.php 'Similar fix for plwiki wgRemoveGroups -- added "bot"' |
[production] |
| 08:41 |
<andrew> |
synchronized php-1.5/InitialiseSettings.php 'Fix for plwiki wgAddGroups, overriding with array( "abusefilter" ) stopped plwiki bureaucrats from adding other groups' |
[production] |
| 04:05 |
<Andrew> |
Works again |
[production] |
| 04:04 |
<Andrew> |
Testing re-enabling of identi.ca bridge for morebots |
[production] |