201-250 of 2001 results (3ms)
2009-04-21 §
05:04 <Andrew> Live-merged r49685, fix for unsuppression of usernames on unblock -- some usernames were left stuck suppressed if they were unblocked when the block suppressed their username [production]
05:03 <andrew> synchronized php-1.5/includes/specials/SpecialBlockip.php [production]
05:03 <andrew> synchronized php-1.5/includes/specials/SpecialIpblocklist.php [production]
01:34 <azafred> Made some improvments on Spam handling. Bayes is in play and can learn from everybody what is spam and what is ham. Documentation to follow. [production]
2009-04-20 §
19:59 <Rob> Powering down srv67, srv85, srv88, srv90 due to temp warnings and bad fans. [production]
19:36 <Rob> updated mc-pmtpa.php to reflect the status of down or spare for the memcached servers. (lots more spares now) [production]
17:35 <azafred> restarted apache on srv217 [production]
17:34 <azafred> srv125 reinstall completed. [production]
17:24 <Rob> srv146 back online [production]
17:10 <Rob> srv131 back up, updated and synced. [production]
16:52 <azafred> srv118 reinstall completed. [production]
16:52 <Rob> srv127 back online and synced. [production]
16:41 <Rob> srv125 reinstalled, passing off to fred [production]
16:40 <Rob> replaced dead disk in sq26 [production]
16:31 <Rob> shutting down sq26 to replace bad hdd [production]
16:27 <Rob> reinstalling srv125 [production]
16:13 <azafred> finished re-install of srv63. [production]
16:11 <Rob> reinstalled srv118, handed off to fred for completion [production]
16:02 <Rob> restarted srv118 and reinstalled it [production]
15:57 <Rob> restarted a locked up srv110 and synced it. [production]
15:49 <Rob> srv81 lacked up, fixed, synced and online [production]
15:29 <Rob> replaced fan and drive in srv63, reinstalling [production]
14:36 <Rob> memory replaced in srv203, back online. [production]
14:11 <Rob> shutting down srv203 to swap out bad memory [production]
05:12 <Tim> fixed memcached on srv75, stopped old ES slave on srv102, srv106, srv107, srv159, srv171 [production]
2009-04-18 §
14:05 <Tim> unblocked 80legs, they promised to be nice [production]
13:56 <tstarling> synchronized robots.txt [production]
05:26 <azafred> rebooted db20 after / ran out of space and started causing all kind of issues. [production]
2009-04-17 §
22:49 <brion> regenerated centralnotice output again... this time ok [production]
22:48 <brion> srv93 and srv107 memcached nodes are running but broken. restarting them... [production]
22:43 <brion> restarted srv82 memcache node. attempting to rebuild centralnotices... [production]
22:41 <brion> bad memcached node srv82 [production]
22:05 <mark> Set up 3 new pywikipedia mailing lists, redirected svn commit output to one of them [production]
19:38 <robh> synchronized php-1.5/InitialiseSettings.php 'Bug 18494 Logo for ln.wiki' [production]
17:22 <Rob> removed wikimedia.se from our nameservers as they are using their own. [production]
16:48 <azafred> updated spamassassin rules on lily to include the SARE rules and mirror the settings on McHenry. [production]
10:25 <tstarling> synchronized robots.txt [production]
08:19 <tstarling> synchronized php-1.5/InitialiseSettings.php [production]
07:13 <Tim> temporarily killed apache on overloaded ES masters [production]
07:11 <tstarling> synchronized php-1.5/db.php 'zeroing read load on ES masters' [production]
06:04 <Tim> brief site-wide outage while it rebooted, reason unknown. All good now. Resuming logrotate. [production]
05:55 <Tim> db20 h/w reboot [production]
05:48 <Tim> shutting down daemons on db20 for pre-emptive reboot. Serial console shows "BUG: soft lockup - CPU#4 stuck for 11s! [rsync:27854]" etc. [production]
05:10 <Tim> on db20: killed logrotate -f half done due to alarming kswapd CPU (linked to deadlocked rsync processes). May need a reboot. [production]
05:00 <Tim> fixed logrotate on db20, broken since March 10 due to broken status file, most likely due to non-ASCII filenames generated by demux.py. Patched demux.py. Removed everything.log. [production]
02:14 <river> set up ms6.esams, copying /export/upload from ms1 [production]
00:24 <Tim> blocked lots of uci.edu IPs that were collectively doing 20 req/s of expensive API queries, overloading ES [production]
00:15 <brion> techblog post on Phorm opt-out is linked from slashdot; load on singer seems fairly stable. [production]
2009-04-16 §
23:06 <tfinc> synchronized php-1.5/extensions/ContributionReporting/ContributionHistory_body.php [production]
22:48 <azafred> bounced apache on srv217. All threads were DED - dead [production]