451-500 of 2080 results (5ms)
2009-04-05 §
05:53 <Andrew> srv183 back up and memcached running, moved it from DOWN to SPARE. Threads on db12 back below 1000 again (normal range). Somehow I just resolved my first site problems by myself :O [production]
05:44 <andrew> synchronized php-1.5/mc-pmtpa.php [production]
05:42 <Andrew> srv183 went down, and it's running memcached. Replaced it with srv61, the first one in the SPARE section of mc-pmtpa.php, and moved it out. Hoping I did this right, and that it helps with db12 overload. [production]
05:30 <Andrew> db12 (enwiki master) overloaded. Nothing I can do about it. [production]
01:52 <andrew> synchronized php-1.5/includes/specials/SpecialBlockip.php 'Live-merge r49191, fixes a bug that gets in the way of suppression' [production]
2009-04-04 §
21:01 <azafred> bounced apache on srv82 [production]
20:14 <azafred> bounced apache on srv50 [production]
18:16 <azafred> bounced apache on srv72 [production]
18:15 <azafred> bounced apache on srv115 [production]
18:01 <azafred> bounced apache on srv121 [production]
17:55 <azafred> bounced apache on srv71 [production]
14:32 <azafred> bounced apache on srv147 [production]
14:25 <domas> restarted memcacheds on srv67, srv112 and srv143 - they seem to have hit reference leak condition (that was probably resolved in memcached 1.2.7) [production]
13:44 <brion_> deployed update to wikibugs [production]
13:03 <Tim> restarting trackBlobs.php, probably died during db2 crash [production]
09:17 <domas> reset-mysql-slave on db23, purged 1-100 logs on db13 [production]
05:06 <azafred> bounced apache on srv89 [production]
04:46 <azafred> bounced apache on srv124 [production]
2009-04-03 §
20:36 <robh> synchronized php-1.5/InitialiseSettings.php 'Turned on collections on srwiki, srwikibooks, and srwikisource' [production]
09:16 <mark> Power cycled sq34 [production]
00:15 <Andrew> installed renameuser, updated mediawiki to r48811 on usability [production]
2009-04-01 §
23:08 <Fred> restarted apache on srv99 [production]
22:57 <mark> Restored session to AS 30217 as well [production]
22:33 <mark> Brought session to AS 13680 back up [production]
21:30 <mark> Shut down BGP sessions to AS 13680 and 30217 for what appears to be problems to/within Level 3 Tampa [production]
16:10 <JeLuF> Image import of about 5000 images done. 245000 left to do... [production]
15:53 <Fred> rebooting srv217 since it is wedged. [production]
15:47 <Fred> restarted apache on srv137 [production]
13:47 <andrew> synchronized php-1.5/InitialiseSettings.php 'Activating AbuseFilter on tpiwiki and hewiki, bugs 18299, 18300' [production]
13:45 <mark> Rebooting srv217 [production]
13:45 <andrew> synchronized php-1.5/CommonSettings.php 'AbuseFilter custom settings, hewiki' [production]
13:40 <JeLuF> batch importing images from the Deutsche Fotothek, commons.wikimedia.org/wiki/Commons:Deutsche_Fotothek [production]
13:11 <Andrew> Reports that Common.css/Common.js weren't working on hsbwiki. Manually purging http://hsb.wikipedia.org/w/index.php?title=-&action=raw&smaxage=0&gen=js&useskin=monobook on the command-line fixed the issue. [production]
08:42 <andrew> synchronized php-1.5/InitialiseSettings.php 'Similar fix for plwiki wgRemoveGroups -- added "bot"' [production]
08:41 <andrew> synchronized php-1.5/InitialiseSettings.php 'Fix for plwiki wgAddGroups, overriding with array( "abusefilter" ) stopped plwiki bureaucrats from adding other groups' [production]
04:05 <Andrew> Works again [production]
04:04 <Andrew> Testing re-enabling of identi.ca bridge for morebots [production]
2009-03-31 §
21:19 <aZaFred> restarted apache on srv99 [production]
05:04 <JeLuF> removed 30 GB of binlogs on db17 [production]
2009-03-30 §
23:50 <domas> interrupted crash recovery for db2, will do crime scene investigation afterwards :) [production]
23:37 <midom> synchronized php-1.5/CommonSettings.php 'enwiki rw with new master' [production]
23:36 <midom> synchronized php-1.5/db.php 'welcome new enwiki master, db12 - db12-bin.016 79' [production]
23:27 <domas> too many open transactions (?) on enwiki/db2 caused it to go OOM or so... [production]
23:25 <midom> synchronized php-1.5/CommonSettings.php [production]
23:23 <river> Out of Memory: Killed process 5374 (mysqld). [production]
23:21 <Andrew> mysqld on db2 crashed. [production]
15:58 <aZaFred> restarted Apache on srv126 and srv103 [production]
08:22 <domas> memcached on srv60 segfaulted.. [production]
05:55 <andrew> synchronized php-1.5/includes/specials/SpecialUserrights.php 'Live-merging r48993, fix for global group membership form (regression)' [production]
05:55 <andrew> synchronized php-1.5/extensions/CentralAuth/SpecialGlobalGroupMembership.php 'Live-merging r48993, fix for global group membership form (regression)' [production]