|
2013-03-26
§
|
| 23:54 |
<awight> |
payments cluster rolled back from e758102 to fa2e240b6e9 |
[production] |
| 23:53 |
<awight> |
payments cluster updated from fa2e240b6e9 to e758102 |
[production] |
| 23:53 |
<Tim> |
on all apaches: replacing /usr/local/apache/common-local/live-1.5 with a symlink to w |
[production] |
| 23:44 |
<tstarling> |
synchronized w |
[production] |
| 23:35 |
<maxsem> |
synchronized wmf-config 'https://gerrit.wikimedia.org/r/#/c/56019/' |
[production] |
| 23:33 |
<awight> |
payment cluster rollback, from 4862fb8 to fa2e240b |
[production] |
| 23:31 |
<mutante> |
restarting nrpe on professor |
[production] |
| 23:28 |
<olivneh> |
synchronized php-1.21wmf12/extensions/NavigationTiming/modules/ext.navigationTiming.js 'Workaround for buggy NavTiming implementation in IE9 (bug 46474)' |
[production] |
| 23:20 |
<awight> |
payments cluster updated from fa2e240b to 4862fb8 |
[production] |
| 23:10 |
<mutante> |
db11 - disabling all notifications |
[production] |
| 23:07 |
<mutante> |
mw13 - killing puppet and re-running |
[production] |
| 23:05 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php |
[production] |
| 23:00 |
<mutante> |
removing mysql packages from db1036 (broken dpkg, status "pi") |
[production] |
| 22:55 |
<mutante> |
re-installing ganglia-monitor on mw1135, fix dpkg |
[production] |
| 22:53 |
<reedy> |
synchronized docroot |
[production] |
| 22:51 |
<mutante> |
starting udpprofile (carbon-cache.py) on professor |
[production] |
| 22:50 |
<reedy> |
synchronized wmf-config/ |
[production] |
| 22:49 |
<mutante> |
search21 - remove sun-j2sdk , replaced by oracle-jdsdk |
[production] |
| 22:47 |
<mutante> |
search21 - broken DPKG: iF sun-j2sdk1.6 |
[production] |
| 22:45 |
<mutante> |
mw2 - broken DPKG: pi librsvg2-2 , pi librsvg2-bin |
[production] |
| 22:39 |
<LeslieCarr> |
did a service apache2 start on mw108* |
[production] |
| 22:28 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php |
[production] |
| 22:24 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php |
[production] |
| 22:10 |
<notpeter> |
icinga mostly trustworthy again :) |
[production] |
| 22:08 |
<andrewbogott> |
restarted nova-api on virt0 because it appeared to have crashed. |
[production] |
| 21:31 |
<andrewbogott> |
restarted memcached on virt0 in a futile attempt to address https://bugzilla.wikimedia.org/show_bug.cgi?id=46583 |
[production] |
| 21:29 |
<mutante> |
killed stuck snmptt procs on neon to fix the others.. apparently :p |
[production] |
| 21:28 |
<mutante> |
stopping gmetad on neon |
[production] |
| 20:51 |
<kaldari> |
synchronized wmf-config/CommonSettings.php 'syncing CommonSettings.php for Echo event tracking' |
[production] |
| 20:41 |
<mutante> |
more degraded RAIDs: db1001->RT-4832, db1028->4834 |
[production] |
| 20:35 |
<mutante> |
broken disks/degraded RAID on db servers: db11->RT-4828, db29->RT-4829, db45->RT-4831 |
[production] |
| 20:34 |
<kaldari> |
synchronized php-1.21wmf12/extensions/ContributionReporting 'syncing ContributionReporting extension' |
[production] |
| 20:07 |
<asher> |
synchronized wmf-config/db-eqiad.php 'raising db1051 to full weight' |
[production] |
| 19:49 |
<khorn> |
synchronized php-1.21wmf12/extensions/CentralNotice 'Fixing CentralNotice bug in which it was possible to initialize multiple times' |
[production] |
| 19:45 |
<khorn> |
synchronized php-1.21wmf11/extensions/CentralNotice 'Fixing CentralNotice bug in which it was possible to initialize multiple times' |
[production] |
| 19:40 |
<kaldari> |
synchronized php-1.21wmf12/resources/startup.js 'syncing php-1.21wmf12/resources/startup.js' |
[production] |
| 19:32 |
<reedy> |
synchronized php-1.21wmf12/extensions/Wikibase/client |
[production] |
| 19:27 |
<cmjohnson1> |
mw1209 re-imaging |
[production] |
| 19:20 |
<kaldari> |
synchronized php-1.21wmf12/extensions/Echo/modules/icons 're-syncing Echo icons dir' |
[production] |
| 19:19 |
<mlitn> |
synchronized wmf-config/InitialiseSettings.php 'Include AFTv5 on frwiki' |
[production] |
| 19:12 |
<mlitn> |
synchronized wmf-config 'Enable AFTv5 on frwiki' |
[production] |
| 19:05 |
<reedy> |
synchronized php-1.21wmf12/extensions/Collection/ |
[production] |
| 19:04 |
<mlitn> |
synchronized php-1.21wmf12/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master' |
[production] |
| 19:03 |
<mlitn> |
synchronized php-1.21wmf11/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master' |
[production] |
| 19:02 |
<kaldari> |
synchronized php-1.21wmf12/extensions/Echo/modules/icons 'syncing Echo icons dir' |
[production] |
| 18:58 |
<mutante> |
starting gmetad on neon - we failed to resolve data source name ms-fe1001.eqiad.wmnet |
[production] |