301-350 of 2001 results (6ms)
2009-04-14 §
19:01 <Rob> replaced dead drive in ms4 [production]
18:41 <Rob> srv78 back online [production]
18:37 <Rob> srv78 was wonky and such, reinstalled to fix. [production]
18:21 <Rob> srv90 reinstalled and redeployed [production]
18:21 <Rob> memcached had stopped on srv89, restarted. [production]
18:16 <Rob> all fans are good on srv86, bringing back online. [production]
18:13 <Rob> srv86 has temp warnings, shutting down to check fans and such [production]
17:59 <Rob> reinstalling srv90 from FC to ubuntu [production]
17:52 <Rob> replaced bad fan in srv90 [production]
17:40 <Rob> pulling srv90, overheating warnings. [production]
17:38 <Rob> srv85 overheating due to dead fans. server is old and out of warranty, decommissioned but kept on site for parting out. [production]
17:10 <Rob> bringing back up sq1, no memory on hand for upgrading these (They are ddr pc3200, all the spare memory we have is ddr2 or ddr 2700) [production]
17:02 <Rob> pulling sq1 for memory upgrade. [production]
16:55 <Rob> replaced bad patch for search9, LOM functions properly. [production]
16:38 <Rob> Upgraded memory in search1 and search2 to a total of 16GB each (previously 8). [production]
16:14 <Rob> Had to restart wikitech due to OOM issues, again. Perhaps it is time to up the memory in the machine or tweak settings. [production]
05:05 <Andrew> testwiki problem seems to be a squid problem, can get srv123, srv84 to serve the main page with no problems by sending a request through netcat. Trying to connect to rr just gets no response [production]
05:01 <Andrew> testwiki seems fubar, timing out on all pageviews. [production]
03:47 <andrew> synchronized php-1.5/InitialiseSettings.php [production]
03:45 <Andrew> Installing AbuseFilter on alswiki [production]
2009-04-12 §
15:22 <tstarling> synchronized php-1.5/extensions/SecurePoll/SecurePoll.i18n.php [production]
12:25 <Tim> updated CentralNotice templates manually to get the license vote in the header [production]
09:05 <Tim> loading license update messages into SecurePoll jump side with sp-msgs-reduced.sql [production]
2009-04-11 §
12:05 <domas> restarting all ubuntu memcacheds, rolling 1.2.8-4 live [production]
08:31 <domas> rebooted srv187 with all the new kernels and such [production]
2009-04-10 §
08:51 <domas> few memcacheds were hitting OOMs, I really have to upgrade them :) [production]
2009-04-09 §
20:15 <JeLuF> started "maintenance/importImages.php" upload of the second batch of Fotothek images to commons [production]
16:37 <tstarling> synchronized php-1.5/extensions/SecurePoll/includes/Auth.php [production]
14:43 <Tim> php_admin_flag engine on in the SecurePoll directory [production]
14:37 <tstarling> synchronized php-1.5/extensions/SecurePoll/includes/VotePage.php [production]
13:55 <robh> synchronized php-1.5/InitialiseSettings.php 'enabling subpages on fiwikiversity' [production]
12:59 <tstarling> synchronized php-1.5/CommonSettings.php 're-enabled SecurePoll' [production]
12:59 <tstarling> synchronized php-1.5/extensions/SecurePoll/includes/Entity.php [production]
12:33 <tstarling> synchronized php-1.5/CommonSettings.php 'enabled SecurePoll' [production]
12:32 <tstarling> synchronized php-1.5/CommonSettings.php 'enabled SecurePoll' [production]
09:04 <mark> Restored Amsterdam traffic; problem resolved [production]
08:07 <mark> Moving Amsterdam traffic to pmtpa while a power problem at esams is being investigated [production]
05:27 <JeLuF> many hosts in esams not reachable any more. Switch outage? [production]
2009-04-08 §
21:31 <Tim> running voterList.php for all wikis on hume, to construct license update voter lists [production]
21:05 <Rob> pushed blog.wikimedia.org dns back into the squid cluster [production]
12:13 <Tim> installing SecurePoll on WM including IP.php r49117 [production]
04:33 <andrew> synchronized php-1.5/mc-pmtpa.php 'Comment for DOWN nodes that seem to be up' [production]
04:19 <andrew> synchronized php-1.5/mc-pmtpa.php 'Memcached on srv143 died, replaced with srv192 (slot 12)' [production]
00:34 <andrew> synchronized php-1.5/InitialiseSettings.php [production]
00:33 <Andrew> deploying AbuseFilter to zhwiki [production]
2009-04-07 §
18:34 <Rob> restarted memcached on srv116 [production]
18:26 <robh> synchronized php-1.5/InitialiseSettings.php 'Bug 18307 Autopatrolled permission on enwikisource.' [production]
16:36 <Rob> rmeoved the outdated fundraising blog out of planet [production]
16:27 <Rob> singer is back to normal. whygive is fubar, but no one cares ;] The rest of the services are online and functional. [production]
08:28 <domas> reset srv217 - where did I hear that again. it was hanging on image NFS and had segfaulting apache too. 7 incidents with it in past two months - needs hardware diagnostics [production]