151-200 of 10000 results (8ms)
2011-11-30 ยง
17:46 <mutante> applied hexmode patch to bugzilla [production]
17:12 <RoanKattouw> NFS is hung, expect it to stay that way for at least 15 mins [production]
17:02 <reedy> synchronized wmf-config/InitialiseSettings.php 'Adding basic wmgUseApiSandbox' [production]
17:01 <catrope> synchronized php-1.18/includes/LinksUpdate.php '[[rev:104701|r104701]]' [production]
17:01 <mutante> hume - installed mysql client 5.1, sym link of wikidiff2.so to php_wikidiff2.so in /usr/lib/php5/20090626, changed # to // in /etc/php5/cli/conf.d/fss.ini, changed variables_order in /etc/php5/cli/php.ini to be EGPCS [production]
16:57 <mutante> applied live hacks apergos did on fenari also on hume [production]
16:13 <RobH> db1025 hdd replaced [production]
16:08 <RobH> swapping failed drive on db1025 per rt2047 [production]
16:03 <reedy> synchronizing Wikimedia installation... : Pushing ApiSandbox out to the cluster, not enabled anywhere yet [production]
15:56 <reedy> synchronized wmf-config/InitialiseSettings.php 'Bug 32500 - Defining transwiki sources for sa.wikipedia.org' [production]
15:49 <catrope> synchronized wmf-config/liquidthreads.php 'Fix notice about wmgLQTUserControlNamespaces being undefined' [production]
15:46 <reedy> synchronized wmf-config/InitialiseSettings.php 'Group tweak for [[bugzilla:32637|bug 32637]]' [production]
15:37 <reedy> synchronized wmf-config/CommonSettings.php 'Adding wmgBabelUseUserLanguage and making wmgBabelDefaultLevel from InitialiseSettings be respected' [production]
15:35 <reedy> synchronized wmf-config/InitialiseSettings.php 'Creating wmgBabelUseUserLanguage' [production]
14:24 <reedy> synchronized php-1.18/extensions/VipsScaler/modules/ext.vipsScaler/ext.vipsScaler.js '[[rev:104663|r104663]]' [production]
14:15 <cmjohnson1> shutting down ms3 [production]
13:48 <RobH> software raid rebuilding on singer [production]
13:45 <RobH> singer back up, racktables, outreachcivi and other services back online [production]
13:36 <cmjohnson1> shutdown singer to replace hard drive [production]
13:33 <mark> Fixed puppet on the karmic search servers (search13..20) [production]
13:27 <mark> Imported puppet 2.7.6+2.7.7rc2-1wm1~karmic1 packages into the karmic-wikimedia APT repository [production]
13:26 <RobH> singer coming down for hard disk replacement. racktables, outreachcivi, and other services will be offline for a short time. [production]
12:37 <mutante> magnesium - rebooted to finish upgrade, started swift [production]
12:22 <mutante> torrus threw 500 errors - fixed it by following "deadlock problem" instructions on wikitech [production]
11:21 <mutante> copper - rebooted to finish upgraded, started swift [production]
11:17 <mark> Imported puppet 2.7.6+2.7.7rc2-1wm1~hardy1 packages into the hardy-wikimedia APT repository [production]
11:16 <mutante> copper - installing new kernel and apt,apache2,bind9-host,.. upgrades [production]
10:47 <mutante> zinc - back up, running swift-init all start [production]
10:45 <mutante> zinc - installed new kernel,rebooting.. [production]
10:02 <mutante> done, bugzilla back up [production]
09:56 <mutante> installing apache2,php-wikidiff2,kernel,.. upgrades on kaulen [production]
09:50 <tstarling> synchronized wmf-config/InitialiseSettings.php [production]
09:43 <tstarling> synchronized wmf-config/InitialiseSettings.php [production]
02:07 <tstarling> ran sync-common-all [production]
01:57 <LocalisationUpdate> failed [production]
01:21 <binasher> added prod eqiad bits caches to dsh varnish group [production]
01:17 <Tim> running sync-common on srv260, srv280, srv284 [production]
01:16 <Tim> rebooting srv284, ssh not working [production]
01:13 <binasher> deployed varnish changes for bits+mobile - serve slightly stale content if backend down + retry backend 503's 4 times [production]
01:05 <Tim> rebooting srv162, ssh not working [production]
01:01 <Tim> rebooting srv280, no working ssh [production]
00:58 <Tim> rebooting srv260, hangs on ssh connections [production]
00:43 <tstarling> synchronized php-1.18/extensions/VipsScaler/SpecialVipsTest.php [production]
00:42 <tstarling> synchronized php-1.18/extensions/VipsScaler/SpecialVipsTest.php [production]
00:39 <tstarling> synchronized php-1.18/extensions/VipsScaler/VipsTest.php [production]
00:28 <tstarling> synchronized php-1.18/includes/StreamFile.php 'debugging hack' [production]
00:21 <tstarling> synchronized php-1.18/extensions/VipsScaler/SpecialVipsTest.php [production]
00:19 <tstarling> synchronized php-1.18/extensions/VipsScaler/VipsScaler_body.php [production]
00:10 <tstarling> synchronized php-1.18/extensions/VipsScaler/SpecialVipsTest.php [production]
00:09 <tstarling> synchronized php-1.18/extensions/VipsScaler/VipsTest.php [production]