|
2011-11-30
§
|
| 17:12 |
<RoanKattouw> |
NFS is hung, expect it to stay that way for at least 15 mins |
[production] |
| 17:02 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Adding basic wmgUseApiSandbox' |
[production] |
| 17:01 |
<catrope> |
synchronized php-1.18/includes/LinksUpdate.php '[[rev:104701|r104701]]' |
[production] |
| 17:01 |
<mutante> |
hume - installed mysql client 5.1, sym link of wikidiff2.so to php_wikidiff2.so in /usr/lib/php5/20090626, changed # to // in /etc/php5/cli/conf.d/fss.ini, changed variables_order in /etc/php5/cli/php.ini to be EGPCS |
[production] |
| 16:57 |
<mutante> |
applied live hacks apergos did on fenari also on hume |
[production] |
| 16:13 |
<RobH> |
db1025 hdd replaced |
[production] |
| 16:08 |
<RobH> |
swapping failed drive on db1025 per rt2047 |
[production] |
| 16:03 |
<reedy> |
synchronizing Wikimedia installation... : Pushing ApiSandbox out to the cluster, not enabled anywhere yet |
[production] |
| 15:56 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Bug 32500 - Defining transwiki sources for sa.wikipedia.org' |
[production] |
| 15:49 |
<catrope> |
synchronized wmf-config/liquidthreads.php 'Fix notice about wmgLQTUserControlNamespaces being undefined' |
[production] |
| 15:46 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Group tweak for [[bugzilla:32637|bug 32637]]' |
[production] |
| 15:37 |
<reedy> |
synchronized wmf-config/CommonSettings.php 'Adding wmgBabelUseUserLanguage and making wmgBabelDefaultLevel from InitialiseSettings be respected' |
[production] |
| 15:35 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Creating wmgBabelUseUserLanguage' |
[production] |
| 14:24 |
<reedy> |
synchronized php-1.18/extensions/VipsScaler/modules/ext.vipsScaler/ext.vipsScaler.js '[[rev:104663|r104663]]' |
[production] |
| 14:15 |
<cmjohnson1> |
shutting down ms3 |
[production] |
| 13:48 |
<RobH> |
software raid rebuilding on singer |
[production] |
| 13:45 |
<RobH> |
singer back up, racktables, outreachcivi and other services back online |
[production] |
| 13:36 |
<cmjohnson1> |
shutdown singer to replace hard drive |
[production] |
| 13:33 |
<mark> |
Fixed puppet on the karmic search servers (search13..20) |
[production] |
| 13:27 |
<mark> |
Imported puppet 2.7.6+2.7.7rc2-1wm1~karmic1 packages into the karmic-wikimedia APT repository |
[production] |
| 13:26 |
<RobH> |
singer coming down for hard disk replacement. racktables, outreachcivi, and other services will be offline for a short time. |
[production] |
| 12:37 |
<mutante> |
magnesium - rebooted to finish upgrade, started swift |
[production] |
| 12:22 |
<mutante> |
torrus threw 500 errors - fixed it by following "deadlock problem" instructions on wikitech |
[production] |
| 11:21 |
<mutante> |
copper - rebooted to finish upgraded, started swift |
[production] |
| 11:17 |
<mark> |
Imported puppet 2.7.6+2.7.7rc2-1wm1~hardy1 packages into the hardy-wikimedia APT repository |
[production] |
| 11:16 |
<mutante> |
copper - installing new kernel and apt,apache2,bind9-host,.. upgrades |
[production] |
| 10:47 |
<mutante> |
zinc - back up, running swift-init all start |
[production] |
| 10:45 |
<mutante> |
zinc - installed new kernel,rebooting.. |
[production] |
| 10:02 |
<mutante> |
done, bugzilla back up |
[production] |
| 09:56 |
<mutante> |
installing apache2,php-wikidiff2,kernel,.. upgrades on kaulen |
[production] |
| 09:50 |
<tstarling> |
synchronized wmf-config/InitialiseSettings.php |
[production] |
| 09:43 |
<tstarling> |
synchronized wmf-config/InitialiseSettings.php |
[production] |
| 02:07 |
<tstarling> |
ran sync-common-all |
[production] |
| 01:57 |
<LocalisationUpdate> |
failed |
[production] |
| 01:21 |
<binasher> |
added prod eqiad bits caches to dsh varnish group |
[production] |
| 01:17 |
<Tim> |
running sync-common on srv260, srv280, srv284 |
[production] |
| 01:16 |
<Tim> |
rebooting srv284, ssh not working |
[production] |
| 01:13 |
<binasher> |
deployed varnish changes for bits+mobile - serve slightly stale content if backend down + retry backend 503's 4 times |
[production] |
| 01:05 |
<Tim> |
rebooting srv162, ssh not working |
[production] |
| 01:01 |
<Tim> |
rebooting srv280, no working ssh |
[production] |
| 00:58 |
<Tim> |
rebooting srv260, hangs on ssh connections |
[production] |
| 00:43 |
<tstarling> |
synchronized php-1.18/extensions/VipsScaler/SpecialVipsTest.php |
[production] |
| 00:42 |
<tstarling> |
synchronized php-1.18/extensions/VipsScaler/SpecialVipsTest.php |
[production] |
| 00:39 |
<tstarling> |
synchronized php-1.18/extensions/VipsScaler/VipsTest.php |
[production] |
| 00:28 |
<tstarling> |
synchronized php-1.18/includes/StreamFile.php 'debugging hack' |
[production] |
| 00:21 |
<tstarling> |
synchronized php-1.18/extensions/VipsScaler/SpecialVipsTest.php |
[production] |
| 00:19 |
<tstarling> |
synchronized php-1.18/extensions/VipsScaler/VipsScaler_body.php |
[production] |
| 00:10 |
<tstarling> |
synchronized php-1.18/extensions/VipsScaler/SpecialVipsTest.php |
[production] |
| 00:09 |
<tstarling> |
synchronized php-1.18/extensions/VipsScaler/VipsTest.php |
[production] |