2301-2350 of 10000 results (19ms)
2010-11-01 §
23:54 <tfinc> synchronized php-1.5/extensions/CentralNotice/CentralNotice.php 'Picking up fixes for r25564' [production]
20:43 <domas> ms4 mildly loaded (disks go to >100i/s each) throwing nfs timeouts, I bumped up NFSD_SERVERS to 2048 [production]
19:05 <Ryan_Lane> powercycling srv207 [production]
16:18 <RoanKattouw> Something weird's going on with srv207: Nagios says its SSH is up but it times out on SSH from fenari [production]
16:15 <catrope> synchronized php-1.5/includes/api/ApiBase.php 'r75798' [production]
2010-10-31 §
17:21 <catrope> synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 25719 - Add missing slash in timezone' [production]
2010-10-30 §
23:05 <apergos> test of logging (sorry) [production]
21:22 <mark> Deploying a sudoers file for NRPE using Puppet [production]
20:48 <mark> Running apt-get upgrade on db17 [production]
20:48 <mark> Pushed updated wikimedia-raid-utils package into the APT repository, with a newer arcconf that should work on Lucid [production]
15:53 <atglenn> powercycled mobile2, it was unresponsive to ssh and pings, ganaglia showed no activity [production]
03:05 <domas> ms1 can't snapshot either, I suspect kernel bugs. we either have to roll back to 2.6.28 or move forward, or actually try rebuilding filesystems from scratch with new kernels... [production]
2010-10-29 §
23:21 <domas> lol repaired myisam tables on db9, call if data has been lost, hehe [production]
22:58 <domas> resynced srv154, was running with months old configuration/code. [production]
22:58 <domas> was db22 disabled silently by someone? or not reenabled? :) reenabled now... [production]
22:55 <midom> synchronized php-1.5/wmf-config/db.php [production]
18:33 <apergos> restarted torrus on streber, after reports that it was not responding [production]
17:46 <apergos> domas ran "reset-mysql-slave db18" (from fenari) which clears out *all* old relay logs, and restarts the slaves. [production]
17:34 <apergos> removed some old relay logs from /a/sqldata on db18 to get space back, it was at 95% [production]
15:22 <RoanKattouw> Followers on Twitter: view missing entries between Sep 2 and today at http://identi.ca/wikimediatech [production]
15:22 <RoanKattouw> Re-established identi.ca->Twitter bridge for wikimediatech, broken since September 2 [production]
15:21 <RobH> repaired the sessions table, rt is now happy [production]
15:09 <RobH> rt is being odd, looking into it [production]
14:43 <phuzion> test [production]
2010-10-28 §
21:34 <RobH> powercycled sq69, ran puppet, its back online [production]
21:24 <RobH> sq69 is borked, powercycling [production]
17:51 <Ryan_Lane> running checksetup.pl on kaulen for bugzilla [production]
17:50 <Ryan_Lane> running mysqlcheck --autorepair on bugzilla database on db9 for the bug_fulltext table [production]
15:23 <atglenn> reenabled logging for fundraising on locke [production]
14:50 <atglenn> I see a lot of lot of ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO) after reboot of db9... not awake enough to try to look at it; services seem to be running ok [production]
14:46 <atglenn> powercycled db9, it was unreachable by ssh, ganglia showed load and wait_cpu through the roof [production]
14:46 <RoanKattouw> db9 back after having been powercycled by Ariel [production]
14:18 <RoanKattouw> db9 down. Responds to ping but doesn't respond to anything else [production]
2010-10-27 §
18:32 <mark> Shutdown yongle for decommissioning [production]
17:49 <jeluf> synchronized php-1.5/wmf-config/InitialiseSettings.php '25668 - Allow bureaucrats to de-sysop and de-bureaucrat on fiwikimedia' [production]
17:43 <jeluf> synchronized php-1.5/wmf-config/InitialiseSettings.php '25659 - Activation of [[Special' [production]
17:41 <jeluf> synchronized php-1.5/wmf-config/InitialiseSettings.php '25659 - Activation of [[Special' [production]
17:29 <jeluf> synchronized php-1.5/wmf-config/InitialiseSettings.php '25652 - Allow importing from frwiki on arzwiki' [production]
11:06 <catrope> synchronized php-1.5/wmf-config/InitialiseSettings.php 'Set $wgOldChangeTagsIndex=false on fawikinews' [production]
2010-10-26 §
22:34 <RobH> removed borked tenwiki deployments entirely from cluster, time to start over [production]
18:33 <midom> synchronized php-1.5/wmf-config/db.php 'readding ms1' [production]
18:04 <RobH> dropping out all the tenwikipedia stuff again, internet hates this wiki [production]
18:00 <robh> synchronized php-1.5/wmf-config/InitialiseSettings.php 'rob makes typos some more' [production]
17:59 <robh> synchronized php-1.5/wmf-config/InitialiseSettings.php 'rob makes typos' [production]
17:30 <RoanKattouw> Removed 10wikipedia from all.dblist and ran sync-dblist [production]
17:08 <robh> synchronized php-1.5/wmf-config/mc.php 'sync cuz i said so' [production]
17:04 <RobH> kicking srv183 around [production]
16:56 <robh> synchronized php-1.5/wmf-config/mc.php 'srv183 fell over' [production]
16:51 <robh> ran sync-common-all [production]
16:26 <RobH> sync-docroot run, nothing blew up, yay [production]