|
2013-03-27
§
|
| 18:23 |
<mutante> |
installing package upgrades on neon (Icinga), incl. ganglia-monitor,gmeta |
[production] |
| 18:08 |
<mutante> |
bunch of mw* hosts: fix puppet runs |
[production] |
| 18:08 |
<kaldari> |
synchronized php-1.21wmf12/extensions/Echo 'syncing Echo for wmf12' |
[production] |
| 18:05 |
<mutante> |
mw1130 - kill and fix puppet runs |
[production] |
| 17:41 |
<Jeff_Green> |
authdns-update, pdns on ns1 croaked, restarted |
[production] |
| 15:05 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: be_x_oldwiki to 1.21wmf12 |
[production] |
| 14:09 |
<hashar> |
jenkins has been restarted by mistake :-( Job building will be unavailable for up to half an hour. |
[production] |
| 13:58 |
<hashar> |
Jenkins: disabled the old Gerrit Trigger Plugin {{bug|46415}} |
[production] |
| 09:56 |
<Nikerabbit> |
Testing Translate bug fixes on test.wikipedia.org |
[production] |
| 05:20 |
<Tim> |
on fluorine: running mw-log-cleanup once to test Id2196e7b |
[production] |
| 05:02 |
<Ryan_Lane> |
disabling Bayes-collector cron on sanger |
[production] |
| 04:55 |
<Ryan_Lane> |
purging BayesLearning files older than 70 days on mchenry |
[production] |
| 02:45 |
<LocalisationUpdate> |
completed (1.21wmf11) at Wed Mar 27 02:44:56 UTC 2013 |
[production] |
| 02:19 |
<LocalisationUpdate> |
completed (1.21wmf12) at Wed Mar 27 02:19:11 UTC 2013 |
[production] |
| 00:55 |
<LeslieCarr> |
ms-be3 was crashed with stack traces for swift-container |
[production] |
| 00:54 |
<LeslieCarr> |
rebooting ms-be3 |
[production] |
| 00:37 |
<awight> |
payments cluster updated from 4862fb8eef to f295d851c |
[production] |
| 00:34 |
<asher> |
synchronized wmf-config/db-eqiad.php 'db1052 to full weight' |
[production] |
| 00:31 |
<asher> |
synchronized wmf-config/db-eqiad.php 'pulling db1001, adding db1052 at a warmup weight' |
[production] |
| 00:10 |
<awight> |
payments cluster updated from fa2e2 to a06a03f05 wheee! |
[production] |
|
2013-03-26
§
|
| 23:54 |
<awight> |
payments cluster rolled back from e758102 to fa2e240b6e9 |
[production] |
| 23:53 |
<awight> |
payments cluster updated from fa2e240b6e9 to e758102 |
[production] |
| 23:53 |
<Tim> |
on all apaches: replacing /usr/local/apache/common-local/live-1.5 with a symlink to w |
[production] |
| 23:44 |
<tstarling> |
synchronized w |
[production] |
| 23:35 |
<maxsem> |
synchronized wmf-config 'https://gerrit.wikimedia.org/r/#/c/56019/' |
[production] |
| 23:33 |
<awight> |
payment cluster rollback, from 4862fb8 to fa2e240b |
[production] |
| 23:31 |
<mutante> |
restarting nrpe on professor |
[production] |
| 23:28 |
<olivneh> |
synchronized php-1.21wmf12/extensions/NavigationTiming/modules/ext.navigationTiming.js 'Workaround for buggy NavTiming implementation in IE9 (bug 46474)' |
[production] |
| 23:20 |
<awight> |
payments cluster updated from fa2e240b to 4862fb8 |
[production] |
| 23:10 |
<mutante> |
db11 - disabling all notifications |
[production] |
| 23:07 |
<mutante> |
mw13 - killing puppet and re-running |
[production] |
| 23:05 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php |
[production] |
| 23:00 |
<mutante> |
removing mysql packages from db1036 (broken dpkg, status "pi") |
[production] |
| 22:55 |
<mutante> |
re-installing ganglia-monitor on mw1135, fix dpkg |
[production] |
| 22:53 |
<reedy> |
synchronized docroot |
[production] |
| 22:51 |
<mutante> |
starting udpprofile (carbon-cache.py) on professor |
[production] |
| 22:50 |
<reedy> |
synchronized wmf-config/ |
[production] |
| 22:49 |
<mutante> |
search21 - remove sun-j2sdk , replaced by oracle-jdsdk |
[production] |
| 22:47 |
<mutante> |
search21 - broken DPKG: iF sun-j2sdk1.6 |
[production] |
| 22:45 |
<mutante> |
mw2 - broken DPKG: pi librsvg2-2 , pi librsvg2-bin |
[production] |
| 22:39 |
<LeslieCarr> |
did a service apache2 start on mw108* |
[production] |
| 22:28 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php |
[production] |
| 22:24 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php |
[production] |
| 22:10 |
<notpeter> |
icinga mostly trustworthy again :) |
[production] |
| 22:08 |
<andrewbogott> |
restarted nova-api on virt0 because it appeared to have crashed. |
[production] |
| 21:31 |
<andrewbogott> |
restarted memcached on virt0 in a futile attempt to address https://bugzilla.wikimedia.org/show_bug.cgi?id=46583 |
[production] |
| 21:29 |
<mutante> |
killed stuck snmptt procs on neon to fix the others.. apparently :p |
[production] |
| 21:28 |
<mutante> |
stopping gmetad on neon |
[production] |
| 20:51 |
<kaldari> |
synchronized wmf-config/CommonSettings.php 'syncing CommonSettings.php for Echo event tracking' |
[production] |
| 20:41 |
<mutante> |
more degraded RAIDs: db1001->RT-4832, db1028->4834 |
[production] |