2801-2850 of 4479 results (13ms)
2009-04-12 §
09:05 <Tim> loading license update messages into SecurePoll jump side with sp-msgs-reduced.sql [production]
2009-04-11 §
12:05 <domas> restarting all ubuntu memcacheds, rolling 1.2.8-4 live [production]
08:31 <domas> rebooted srv187 with all the new kernels and such [production]
2009-04-10 §
08:51 <domas> few memcacheds were hitting OOMs, I really have to upgrade them :) [production]
2009-04-09 §
20:15 <JeLuF> started "maintenance/importImages.php" upload of the second batch of Fotothek images to commons [production]
16:37 <tstarling> synchronized php-1.5/extensions/SecurePoll/includes/Auth.php [production]
14:43 <Tim> php_admin_flag engine on in the SecurePoll directory [production]
14:37 <tstarling> synchronized php-1.5/extensions/SecurePoll/includes/VotePage.php [production]
13:55 <robh> synchronized php-1.5/InitialiseSettings.php 'enabling subpages on fiwikiversity' [production]
12:59 <tstarling> synchronized php-1.5/CommonSettings.php 're-enabled SecurePoll' [production]
12:59 <tstarling> synchronized php-1.5/extensions/SecurePoll/includes/Entity.php [production]
12:33 <tstarling> synchronized php-1.5/CommonSettings.php 'enabled SecurePoll' [production]
12:32 <tstarling> synchronized php-1.5/CommonSettings.php 'enabled SecurePoll' [production]
09:04 <mark> Restored Amsterdam traffic; problem resolved [production]
08:07 <mark> Moving Amsterdam traffic to pmtpa while a power problem at esams is being investigated [production]
05:27 <JeLuF> many hosts in esams not reachable any more. Switch outage? [production]
2009-04-08 §
21:31 <Tim> running voterList.php for all wikis on hume, to construct license update voter lists [production]
21:05 <Rob> pushed blog.wikimedia.org dns back into the squid cluster [production]
12:13 <Tim> installing SecurePoll on WM including IP.php r49117 [production]
04:33 <andrew> synchronized php-1.5/mc-pmtpa.php 'Comment for DOWN nodes that seem to be up' [production]
04:19 <andrew> synchronized php-1.5/mc-pmtpa.php 'Memcached on srv143 died, replaced with srv192 (slot 12)' [production]
00:34 <andrew> synchronized php-1.5/InitialiseSettings.php [production]
00:33 <Andrew> deploying AbuseFilter to zhwiki [production]
2009-04-07 §
18:34 <Rob> restarted memcached on srv116 [production]
18:26 <robh> synchronized php-1.5/InitialiseSettings.php 'Bug 18307 Autopatrolled permission on enwikisource.' [production]
16:36 <Rob> rmeoved the outdated fundraising blog out of planet [production]
16:27 <Rob> singer is back to normal. whygive is fubar, but no one cares ;] The rest of the services are online and functional. [production]
08:28 <domas> reset srv217 - where did I hear that again. it was hanging on image NFS and had segfaulting apache too. 7 incidents with it in past two months - needs hardware diagnostics [production]
01:48 <Rob> blog.wikimedia.org is now up. singer is kinda mostly fixed, i will finish it in the morning. all sites on it are up. [production]
00:16 <Rob> pushed blog.wikimedia.org out of squid via dns [production]
2009-04-06 §
21:49 <Rob> singer is near returned to normal, however the primary corporate blog is doing funny things with caching and apache direction on the server. [production]
21:34 <Rob> random insanity with apache on singer, affected corporate blogs, ocs, wm09scholarships, communicate portal, and ...well... thats enough. Still working on resolution. [production]
14:26 <robh> synchronized php-1.5/InitialiseSettings.php 'Bug 16178 Activate Collection Extension for generating PDF on the French Wikiversity' [production]
14:18 <robh> synchronized php-1.5/InitialiseSettings.php 'Bug 17861 logo update at german wikibooks' [production]
13:52 <kate> synchronized php-1.5/extensions/CodeReview/CodeReview.php [production]
12:44 <domas> reset power for srv187 [production]
12:44 <domas> restarted hanging apaches on srv90, srv97 and crashlooping ones srv189, srv206 [production]
12:35 <domas> restarted busylooping memcached on srv143, new bug! [production]
11:47 <kate> synchronized php-1.5/extensions/CodeReview/codereview.css [production]
04:57 <andrew> synchronized php-1.5/mc-pmtpa.php 'srv129 down, swapped it for srv182' [production]
04:54 <andrew> synchronized php-1.5/includes/specials/SpecialBlockip.php 'Live-merging r49222 -- fix for hiding of logging data in recentchanges. Tests okay on testwiki.' [production]
2009-04-05 §
19:45 <jeluf> synchronized php-1.5/mc-pmtpa.php 'replace srv92 b srv152' [production]
15:07 <domas> Tim rocks, cluster back to normal [production]
15:04 <Tim> deployed r49212 to fix infinite template recursion issue [production]
11:24 <domas> restarted some fedora apaches, were stuck in write() after the previous hiccups [production]
10:28 <mark> Restarted memcached on srv151 [production]
08:59 <domas> I hate computers [production]
08:28 <domas> hanging latex processes held :80, thus not allowing clean apache restarts on some nodes [production]
08:18 <domas> restarted plentiful of crashlooping apaches, investigating resource consumption problem [production]
05:53 <Andrew> srv183 back up and memcached running, moved it from DOWN to SPARE. Threads on db12 back below 1000 again (normal range). Somehow I just resolved my first site problems by myself :O [production]