6651-6700 of 10000 results (23ms)
2009-08-27 §
17:44 <Tim> running recompressTracked.sh on hume [production]
17:40 <mark> Shutdown MySQL on ms2 for reboot [production]
17:28 <mark> Replication on ms2 caught up with master ms3 [production]
17:08 <mark> Recovered data on ms2 by rebuilding raid-arrays, started mysql, started replication [production]
16:44 <midom> synchronized php-1.5/wmf-config/InitialiseSettings.php 'remove optin for performance reasons' [production]
16:43 <midom> synchronized php-1.5/wmf-config/InitialiseSettings.php 'remove optin for performance reasons' [production]
16:28 <mark> Installed gmond on mayflower [production]
16:20 <mark> Power cycled mayflower again [production]
16:06 <mark> Recovering RAID arrays on ms2 [production]
15:43 <brion> updating to wmf-deployment r55629 -- fixes to api, (un)delete storage, and collection [production]
15:33 <tstarling> synchronized php-1.5/includes/Revision.php 'r55628' [production]
15:26 <midom> synchronized php-1.5/includes/AutoLoader.php 'profiling' [production]
15:10 <domas> bunch of servers in srv190+ range have intermittent network problems, leading to connection failures, etc [production]
14:58 <tstarling> synchronized php-1.5/includes/Revision.php 'testing archive issue pre-commit' [production]
10:03 <mark> Power cycled mayflower (svn) [production]
2009-08-26 §
21:56 <Tim> observing overload on s4 slaves due to long-running ApiQueryBacklinks queries, will kill [production]
21:29 <brion> update to r55616 -- fixlets and api limit tweak [production]
18:11 <midom> synchronized php-1.5/includes/AutoLoader.php 'profiling hook' [production]
17:08 <brion> rerunning parser tests for r55603; temporary failure on the test box [production]
2009-08-25 §
15:52 <midom> synchronized php-1.5/wmf-config/db.php 'mwahahaha' [production]
15:47 <midom> synchronized php-1.5/wmf-config/db.php [production]
15:44 <midom> synchronized php-1.5/wmf-config/db.php 'load testing, muhaha' [production]
15:40 <midom> synchronized php-1.5/wmf-config/db.php 'putting storage1 to cluster22' [production]
14:21 <brion> synchronized php-1.5/wmf-config/InitialiseSettings.php 'enable PdfHandler sitewide; forgot to do this earlier :D' [production]
2009-08-24 §
00:45 <domas> CPU 100% on apache cluster for few minutes, reason unknown :) (thats what 'unbalanced load' probably means, hehe) [production]
00:12 <brion> unbalanced load; db16 reporting too many connected threads [production]
2009-08-23 §
23:32 <brion> synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 20363: namespace fixes' [production]
23:14 <Fred> HD in db18 Enclosure 0 Slot 21 failed and needs to be replaced. [production]
23:12 <Fred> HD in db21 - Enclosure0 Slot 7 needs to be replaced. [production]
22:55 <Fred> restarted apache on srv140 [production]
22:55 <Fred> restarted memcached on srv197 [production]
16:43 <Fred> restarted memcached on srv211 [production]
00:50 <brion> synchronized php-1.5/wmf-config/InitialiseSettings.php '20338 Enable Collection on Bengali Wikipedia' [production]
00:18 <brion> synchronized php-1.5/wmf-config/InitialiseSettings.php 'fix eswikibooks english import source (bug 20352)' [production]
2009-08-22 §
00:48 <atglenn> next approach: zfs send (ms1) -> cat > filename (ms5). running in screen. [production]
00:43 <atglenn> stopped zfs send/receive from ms1 -> ms5: 3 GB in two hours = fail :-P [production]
2009-08-21 §
22:16 <atglenn> starting copy of image data to ms5 (consider this timing tests, not the real thing): zfs send running in screen session [production]
20:51 <aZaFred_> restarted memcached on srv212 [production]
20:36 <fvassard> synchronized php-1.5/wmf-config/db.php 'commenting out the rc2 cluster since it doesnt seem to be used anymore...' [production]
20:01 <fvassard> synchronized php-1.5/wmf-config/mc.php 'Removed srv52 and added srv219 in its place since srv52 is not behaving...' [production]
17:48 <aZaFred_> fixed nrpe config on adler as it was set not to accept any connection not coming form 127.0.0.1 [production]
17:37 <aZaFred_> switching Nagios from bart to spence (its new home) and decomissioning Nagios on Bart. [production]
16:34 <aZaFred_> changed settings for sq43 since /dev/sdd disappeared. [production]
16:15 <aZaFred_> restarted memcached on srv89 [production]
15:09 <hcatlin> Mobile1 just had af, az, bn, ca, bg, and drb languages deployed. [production]
2009-08-20 §
23:02 <domas> opensearch is causing most of 'revision' SQL load (probably text too). search overall accounts for more than 25% of database::query time :-) [production]
19:48 <midom> synchronized php-1.5/includes/Revision.php 'removing profiling hooks' [production]
19:07 <midom> synchronized php-1.5/includes/Revision.php 'profiling hooks' [production]
18:42 <brion> running featured article imports on readerfeedback & flaggedrevs test wikis [production]
18:25 <brion> pulling enwiki featured articles to copy into readerfeedback & flaggrevs test wikis [production]