5801-5850 of 7452 results (17ms)
2009-04-07 §
08:28 <domas> reset srv217 - where did I hear that again. it was hanging on image NFS and had segfaulting apache too. 7 incidents with it in past two months - needs hardware diagnostics [production]
01:48 <Rob> blog.wikimedia.org is now up. singer is kinda mostly fixed, i will finish it in the morning. all sites on it are up. [production]
00:16 <Rob> pushed blog.wikimedia.org out of squid via dns [production]
2009-04-06 §
21:49 <Rob> singer is near returned to normal, however the primary corporate blog is doing funny things with caching and apache direction on the server. [production]
21:34 <Rob> random insanity with apache on singer, affected corporate blogs, ocs, wm09scholarships, communicate portal, and ...well... thats enough. Still working on resolution. [production]
14:26 <robh> synchronized php-1.5/InitialiseSettings.php 'Bug 16178 Activate Collection Extension for generating PDF on the French Wikiversity' [production]
14:18 <robh> synchronized php-1.5/InitialiseSettings.php 'Bug 17861 logo update at german wikibooks' [production]
13:52 <kate> synchronized php-1.5/extensions/CodeReview/CodeReview.php [production]
12:44 <domas> reset power for srv187 [production]
12:44 <domas> restarted hanging apaches on srv90, srv97 and crashlooping ones srv189, srv206 [production]
12:35 <domas> restarted busylooping memcached on srv143, new bug! [production]
11:47 <kate> synchronized php-1.5/extensions/CodeReview/codereview.css [production]
04:57 <andrew> synchronized php-1.5/mc-pmtpa.php 'srv129 down, swapped it for srv182' [production]
04:54 <andrew> synchronized php-1.5/includes/specials/SpecialBlockip.php 'Live-merging r49222 -- fix for hiding of logging data in recentchanges. Tests okay on testwiki.' [production]
2009-04-05 §
19:45 <jeluf> synchronized php-1.5/mc-pmtpa.php 'replace srv92 b srv152' [production]
15:07 <domas> Tim rocks, cluster back to normal [production]
15:04 <Tim> deployed r49212 to fix infinite template recursion issue [production]
11:24 <domas> restarted some fedora apaches, were stuck in write() after the previous hiccups [production]
10:28 <mark> Restarted memcached on srv151 [production]
08:59 <domas> I hate computers [production]
08:28 <domas> hanging latex processes held :80, thus not allowing clean apache restarts on some nodes [production]
08:18 <domas> restarted plentiful of crashlooping apaches, investigating resource consumption problem [production]
05:53 <Andrew> srv183 back up and memcached running, moved it from DOWN to SPARE. Threads on db12 back below 1000 again (normal range). Somehow I just resolved my first site problems by myself :O [production]
05:44 <andrew> synchronized php-1.5/mc-pmtpa.php [production]
05:42 <Andrew> srv183 went down, and it's running memcached. Replaced it with srv61, the first one in the SPARE section of mc-pmtpa.php, and moved it out. Hoping I did this right, and that it helps with db12 overload. [production]
05:30 <Andrew> db12 (enwiki master) overloaded. Nothing I can do about it. [production]
01:52 <andrew> synchronized php-1.5/includes/specials/SpecialBlockip.php 'Live-merge r49191, fixes a bug that gets in the way of suppression' [production]
2009-04-04 §
21:01 <azafred> bounced apache on srv82 [production]
20:14 <azafred> bounced apache on srv50 [production]
18:16 <azafred> bounced apache on srv72 [production]
18:15 <azafred> bounced apache on srv115 [production]
18:01 <azafred> bounced apache on srv121 [production]
17:55 <azafred> bounced apache on srv71 [production]
14:32 <azafred> bounced apache on srv147 [production]
14:25 <domas> restarted memcacheds on srv67, srv112 and srv143 - they seem to have hit reference leak condition (that was probably resolved in memcached 1.2.7) [production]
13:44 <brion_> deployed update to wikibugs [production]
13:03 <Tim> restarting trackBlobs.php, probably died during db2 crash [production]
09:17 <domas> reset-mysql-slave on db23, purged 1-100 logs on db13 [production]
05:06 <azafred> bounced apache on srv89 [production]
04:46 <azafred> bounced apache on srv124 [production]
2009-04-03 §
20:36 <robh> synchronized php-1.5/InitialiseSettings.php 'Turned on collections on srwiki, srwikibooks, and srwikisource' [production]
09:16 <mark> Power cycled sq34 [production]
00:15 <Andrew> installed renameuser, updated mediawiki to r48811 on usability [production]
2009-04-01 §
23:08 <Fred> restarted apache on srv99 [production]
22:57 <mark> Restored session to AS 30217 as well [production]
22:33 <mark> Brought session to AS 13680 back up [production]
21:30 <mark> Shut down BGP sessions to AS 13680 and 30217 for what appears to be problems to/within Level 3 Tampa [production]
16:10 <JeLuF> Image import of about 5000 images done. 245000 left to do... [production]
15:53 <Fred> rebooting srv217 since it is wedged. [production]
15:47 <Fred> restarted apache on srv137 [production]