1-50 of 10000 results (5ms)
2011-12-23 §
16:34 <mutante> kswapd crashed, there is a Call Trace, and there was a load spike before, guess it is https://bugs.launchpad.net/ubuntu/+bug/721896 or similar [production]
16:27 <mutante> first interesting syslog line when it started: formey kernel: [39665413.570024] INFO: task kswapd0:36 blocked for more than 120 seconds. [production]
16:10 <mutante> gerrit and svn back up [production]
16:07 <mutante> gerrit stopped working and formey would still ping but no ssh connect and no mgmt output, powercycling formey [production]
02:07 <binasher> db9 maintenance completed [production]
02:02 <LocalisationUpdate> completed (1.18) at Fri Dec 23 02:05:19 UTC 2011 [production]
01:57 <binasher> starting db9 maintenance - services will be unavailable for approx 15 minutes [production]
00:08 <hashar> synchronized php-1.18/extensions/WikimediaIncubator/IncubatorTest.php 'deploy [[rev:107120|r107120]] - ([[bugzilla:32772|bug 32772]]) fix up MoodBar and WikiLove on WIkimedia Incubator' [production]
2011-12-22 §
23:43 <maplebed> put owa1-3 in as container servers, took ms1-3 out for pmtpa test swift cluster [production]
20:56 <neilk> synchronized p/resources/mediawiki/mediawiki.feedback.js [production]
20:45 <Ryan_Lane> *date [production]
20:45 <Ryan_Lane> bringing mediawiki on virt1 up to daye [production]
20:24 <catrope> synchronized wmf-config/InitialiseSettings.php 'Re-enable Contest extension' [production]
20:10 <mark> Turned off puppet dashboard reporting [production]
15:07 <apergos> er, on srv219, that is. [production]
15:07 <apergos> cleaned out tmp but... see, there really was only today's stuff in there so it's making me nervous [production]
14:15 <31NAAAO2O> hashar: testswarm: deleted mobile jobs requests since they are now disabled [production]
12:30 <mark> Made swift thumb seeder fetch from the squids instead of ms5, as a performance test [production]
05:10 <Ryan_Lane> finished creating all puppet configuration groups, classes, and variables. It's safe to configure and create instances again. [production]
04:40 <Ryan_Lane> upping version of OpenStackManager on virt1 to match development. configuration and creation of instances should be avoided. [production]
02:04 <binasher> db9 is writable again [production]
02:01 <LocalisationUpdate> completed (1.18) at Thu Dec 22 02:04:05 UTC 2011 [production]
01:59 <binasher> started db9 maintenance phase 1 (getting it replicated to db10 again) [production]
00:15 <K4-713> synchronized payments cluster to [[rev:107018|r107018]] [production]
00:07 <awjrichards> synchronizing Wikimedia installation... : [[rev:107015|r107015]] [production]
2011-12-21 §
22:44 <mark> proxy worker processes increased from 8 to 24 on owa1-2, 48 on owa3 [production]
22:28 <mark> Unmounted /srv/swift-storage/sdab1 on ms2 (borken filesystem) [production]
22:16 <mark> Flushed all iptables rules down the drain on ms1-3 (live hack, puppet will restore) [production]
21:54 <mark> Ran swift-init all restart on ms3 [production]
21:41 <mark> Ran swift-init all restart on ms2 [production]
21:33 <mark> Running ben's swift thumb loader script in a screen on hume [production]
21:05 <apergos> three more bin logs tossed from ds9 [production]
20:31 <mark> Restarted swift-container on ms1 with higher worker count (4 instead of 2) [production]
20:31 <Jeff_Green> power cycled kaulen because it's deathswapped and unresponsive [production]
19:41 <mark> Ended oprofile run on ms1 [production]
19:33 <catrope> synchronized wmf-config/missing.php 'Update missing.php from trunk, see [[bugzilla:30206|bug 30206]]' [production]
19:24 <mark> Started oprofile run on ms1 [production]
19:20 <mark> Migrated DRBD sync between nfs1 and nfs2 from protocol C (sync) to A (async) [production]
17:48 <RoanKattouw> srv224 has a full disk [production]
17:48 <catrope> synchronized php-1.18/extensions/ArticleFeedbackv5/modules/jquery.articleFeedbackv5/jquery.articleFeedbackv5.js '[[rev:106959|r106959]]' [production]
17:28 <maplebed> ran apt-get clean on hume to clear out ~600M space on the / partition [production]
16:18 <apergos> so that was fast. barf from scp, nice call trace etc, shot the process on ds2, will email the vendor [production]
15:27 <apergos> and starting another huge copy from ds2 to ds1, let's see what happens... [production]
15:18 <apergos> reboot dataset1 with new kernel [production]
15:14 <apergos> installing 2.6.38 from natty backports on ds1 for further testing [production]
13:55 <apergos> powering on and off ds1 the hard way via the pdu. [production]
11:15 <apergos> rebooting ds1 as it's got the one cpu tied up with a hung scp process and continual spewing to syslog... [production]
10:23 <apergos> s/lgo/log/ as in syslog. saving a copy of the bad log in fenari:/home/ariel/dataset1-syslog-dec-20-2012 [production]
10:09 <apergos> dataset1 kernel panics in lgo during copy :-( :-( [production]
09:27 <apergos> a few more binlogs deleted on db9... [production]