601-650 of 2349 results (6ms)
2009-04-16 §
17:41 <domas> fantastic. I start _looking_ at stuff and it fixes itself. [production]
17:35 <midom> synchronized php-1.5/includes/Revision.php 'live profiling hook' [production]
17:28 <domas> db20 has kswapd deadlock, needs reboot soonish [production]
17:18 <midom> synchronized php-1.5/InitialiseSettings.php 'disabled stats' [production]
17:15 <midom> synchronized php-1.5/InitialiseSettings.php 'enabling udp stats' [production]
16:18 <azafred> bounced apache on srv217 (no pid file so previous restart did not include this one) [production]
15:57 <brion> network borkage between Florida and Amsterdam. Visitors through AMS proxies can't reach sites. [production]
15:55 <azafred> bounced apache on srv[73,86,88,93,108,114,139,141,154,181,194,204,213,99] [production]
15:52 <Tim-away> started mysqld on srv98,srv122,srv124,srv142,srv106,srv107: done with them for now. srv102 still going. [production]
15:30 <mark> Set up ms6 with SP management at ms6.ipmi.esams.wikimedia.org [production]
14:13 <mark> Restoring traffic to Amsterdam cluster [production]
14:06 <mark> Reloading csw1-esams [production]
13:55 <mark> Reloading csw1-esams [production]
13:53 <JeLuF> ms1 NFS issues again. Might be load related [production]
13:49 <Tim> copying fedora ES data from ms3 to ms2 [production]
13:44 <JeLuF> ms1 is reachable, no errors logged, NFS daemons running fine. After some minutes, NFS clients were able to access the server again. Root cause unknown. [production]
13:38 <JeLuF> ms1 issues. On NFS slaves: "ls: cannot access /mnt/upload5/: Input/output error" [production]
13:24 <mark> DNS scenario knams-down for upcoming core switch reboot [production]
08:23 <river> pdns on bayle crashed, bindbackend parser seems rather fragile [production]
03:01 <andrew> synchronized php-1.5/InitialiseSettings.php 'Deployed AbuseFilter to ptwiki' [production]
2009-04-15 §
22:42 <tomaszf> adding ramdisk to db9 to speed up create tmp tables [production]
22:34 <mark> PowerDNS got confused by a commented DNS entry and broke zone wikimedia.org, fixed [production]
22:32 <brion-codereview> DNS broken. mark's poking it [production]
22:24 <mark> Temporarily removed AAAA record from mayflower in DNS [production]
22:14 <brion-codereview> db9 tmpfs full, breaking anything using that db [production]
22:00 <brion-codereview> ipv6 connectivity broken between isidore & mayflower, breaking codereview SVN updates [production]
20:59 <brion> civicrm queries bogging down db9 affecting otrs performance. tom's looking into it [production]
18:24 <robh> synchronized php-1.5/InitialiseSettings.php 'for subpages on ukwikimedia' [production]
17:32 <robh> synchronized php-1.5/InitialiseSettings.php 'Bug 17898 Wiktionary is a bad interwiki prefix on ukwiktionary and mlwiktionary' [production]
17:25 <robh> synchronized php-1.5/InitialiseSettings.php 'per bug 17773 Install Labeled Section Transclusion for dewikiversity' [production]
14:33 <robh> synchronized php-1.5/InitialiseSettings.php 'Bug 17718 Disable CentralNotice on private/fishbowl wikis' [production]
14:29 <robh> synchronized php-1.5/InitialiseSettings.php '18434 Enable the rollback feature on Commons' [production]
14:19 <robh> synchronized php-1.5/InitialiseSettings.php '18307 Add autopatrolled group to English Wikisource' [production]
14:12 <robh> synchronized php-1.5/InitialiseSettings.php 'Bug 17717 Enable subpages on main namespace of UK chapter website' [production]
13:55 <robh> synchronized php-1.5/InitialiseSettings.php 'Bug 18428 cswikisource settings updates' [production]
12:38 <Tim> restarting copy to ms3 [production]
12:25 <Tim> rebooting ms3 with 2.6.28 kernel [production]
12:18 <Tim> running xfs_check on ms3 [production]
12:14 <Tim> restarting ms2 with domas's 2.6.28 kernel [production]
12:06 <midom> synchronized php-1.5/db.php 'removing db25 - apparently it was down for more than a day' [production]
11:58 <domas> db25 went down, resetting [production]
11:08 <Tim> ms3 went down, no response on serial console, rebooting [production]
11:05 <tstarling> synchronized php-1.5/db.php [production]
08:32 <Tim> copy in progress, rsync over ssh controlled via screen on tstarling@zwinger [production]
08:23 <Tim> shutting down mysqld on srv98,srv122,srv124,srv142,srv102,srv106,srv107 for data directory copy to ms3 [production]
2009-04-14 §
23:48 <tfinc> synchronized php-1.5/extensions/ContributionTracking/ContributionTracking_body.php [production]
23:39 <tfinc> synchronized php-1.5/extensions/ContributionTracking/ContributionTracking_body.php [production]
23:37 <tfinc> synchronized php-1.5/reporting-setup.php [production]
19:01 <Rob> replaced dead drive in ms4 [production]
18:41 <Rob> srv78 back online [production]