|
2009-04-07
§
|
| 18:34 |
<Rob> |
restarted memcached on srv116 |
[production] |
| 18:26 |
<robh> |
synchronized php-1.5/InitialiseSettings.php 'Bug 18307 Autopatrolled permission on enwikisource.' |
[production] |
| 16:36 |
<Rob> |
rmeoved the outdated fundraising blog out of planet |
[production] |
| 16:27 |
<Rob> |
singer is back to normal. whygive is fubar, but no one cares ;] The rest of the services are online and functional. |
[production] |
| 08:28 |
<domas> |
reset srv217 - where did I hear that again. it was hanging on image NFS and had segfaulting apache too. 7 incidents with it in past two months - needs hardware diagnostics |
[production] |
| 01:48 |
<Rob> |
blog.wikimedia.org is now up. singer is kinda mostly fixed, i will finish it in the morning. all sites on it are up. |
[production] |
| 00:16 |
<Rob> |
pushed blog.wikimedia.org out of squid via dns |
[production] |
|
2009-04-06
§
|
| 21:49 |
<Rob> |
singer is near returned to normal, however the primary corporate blog is doing funny things with caching and apache direction on the server. |
[production] |
| 21:34 |
<Rob> |
random insanity with apache on singer, affected corporate blogs, ocs, wm09scholarships, communicate portal, and ...well... thats enough. Still working on resolution. |
[production] |
| 14:26 |
<robh> |
synchronized php-1.5/InitialiseSettings.php 'Bug 16178 Activate Collection Extension for generating PDF on the French Wikiversity' |
[production] |
| 14:18 |
<robh> |
synchronized php-1.5/InitialiseSettings.php 'Bug 17861 logo update at german wikibooks' |
[production] |
| 13:52 |
<kate> |
synchronized php-1.5/extensions/CodeReview/CodeReview.php |
[production] |
| 12:44 |
<domas> |
reset power for srv187 |
[production] |
| 12:44 |
<domas> |
restarted hanging apaches on srv90, srv97 and crashlooping ones srv189, srv206 |
[production] |
| 12:35 |
<domas> |
restarted busylooping memcached on srv143, new bug! |
[production] |
| 11:47 |
<kate> |
synchronized php-1.5/extensions/CodeReview/codereview.css |
[production] |
| 04:57 |
<andrew> |
synchronized php-1.5/mc-pmtpa.php 'srv129 down, swapped it for srv182' |
[production] |
| 04:54 |
<andrew> |
synchronized php-1.5/includes/specials/SpecialBlockip.php 'Live-merging r49222 -- fix for hiding of logging data in recentchanges. Tests okay on testwiki.' |
[production] |
|
2009-04-05
§
|
| 19:45 |
<jeluf> |
synchronized php-1.5/mc-pmtpa.php 'replace srv92 b srv152' |
[production] |
| 15:07 |
<domas> |
Tim rocks, cluster back to normal |
[production] |
| 15:04 |
<Tim> |
deployed r49212 to fix infinite template recursion issue |
[production] |
| 11:24 |
<domas> |
restarted some fedora apaches, were stuck in write() after the previous hiccups |
[production] |
| 10:28 |
<mark> |
Restarted memcached on srv151 |
[production] |
| 08:59 |
<domas> |
I hate computers |
[production] |
| 08:28 |
<domas> |
hanging latex processes held :80, thus not allowing clean apache restarts on some nodes |
[production] |
| 08:18 |
<domas> |
restarted plentiful of crashlooping apaches, investigating resource consumption problem |
[production] |
| 05:53 |
<Andrew> |
srv183 back up and memcached running, moved it from DOWN to SPARE. Threads on db12 back below 1000 again (normal range). Somehow I just resolved my first site problems by myself :O |
[production] |
| 05:44 |
<andrew> |
synchronized php-1.5/mc-pmtpa.php |
[production] |
| 05:42 |
<Andrew> |
srv183 went down, and it's running memcached. Replaced it with srv61, the first one in the SPARE section of mc-pmtpa.php, and moved it out. Hoping I did this right, and that it helps with db12 overload. |
[production] |
| 05:30 |
<Andrew> |
db12 (enwiki master) overloaded. Nothing I can do about it. |
[production] |
| 01:52 |
<andrew> |
synchronized php-1.5/includes/specials/SpecialBlockip.php 'Live-merge r49191, fixes a bug that gets in the way of suppression' |
[production] |