|
2011-03-31
§
|
| 14:01 |
<mark> |
Outage resolved by disabling ehcache |
[production] |
| 13:57 |
<midom> |
synchronized php-1.17/wmf-config/InitialiseSettings.php |
[production] |
| 13:51 |
<root> |
synchronized php-1.17/wmf-config/InitialiseSettings.php |
[production] |
| 13:37 |
<mark> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'Disable ehcache' |
[production] |
| 11:43 |
<mark> |
set max_stale 1 month on pmtpa.upload squids |
[production] |
| 11:21 |
<mark> |
Set refresh_pattern on pmtpa upload squids |
[production] |
| 10:47 |
<mark> |
Set refresh_pattern on sq85 |
[production] |
| 00:21 |
<Ryan_Lane> |
added drac management script and dependencies to fenari, via puppet |
[production] |
| 00:06 |
<Ryan_Lane> |
make that python-paramiko |
[production] |
| 00:06 |
<Ryan_Lane> |
installing paramiko on fenari |
[production] |
|
2011-03-30
§
|
| 23:15 |
<Ryan_Lane> |
adding puppet configuration for the collector and adding that class to spence |
[production] |
| 23:09 |
<Ryan_Lane> |
updating udpprofile package again on spence |
[production] |
| 23:02 |
<Ryan_Lane> |
upgrading udpprofile package on spence to fix init script |
[production] |
| 22:48 |
<Ryan_Lane> |
killing the collector on spence, and trying to run via the package |
[production] |
| 22:48 |
<Ryan_Lane> |
installing udpprofile package on spence |
[production] |
| 22:44 |
<Ryan_Lane> |
created a package for the udpprofile code in svn; added to the main lucid-wikimedia repo |
[production] |
| 19:33 |
<Ryan_Lane> |
fixed sanger certificate check for ldaps. warning will go away on next puppet run |
[production] |
| 17:05 |
<Ryan_Lane1> |
enabled a nagios certificate check on sanger for ldaps |
[production] |
| 16:57 |
<thedj> |
Maintenance ongoing on image server. Thumbnails on #Wikimedia sites may appear broken; we expect to restore full service shortly. (by Danese) |
[production] |
| 15:23 |
<apergos> |
s/fine/find/ |
[production] |
| 15:22 |
<ariel> |
synchronized php-1.17/wmf-config/CommonSettings.php 'increase account creation throttle value for el wiki for editing workshop (maybe we should fine a better solution?)' |
[production] |
| 13:03 |
<mark> |
Started 8 parallel rsyncs of ms4 /export/thumbs/wikipedia/commons/thumb/{8..f} to ms5 |
[production] |
| 10:50 |
<Tim> |
restarted ehcache with increased cache size limits |
[production] |
| 10:49 |
<tstarling> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 're-enabling ehcache' |
[production] |
| 10:45 |
<tstarling> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'disabling ehcache temporarily' |
[production] |
| 10:15 |
<apergos> |
rebooting ms5, trying to fix up raid config |
[production] |
| 08:52 |
<catrope> |
ran sync-common-all |
[production] |
| 08:52 |
<RoanKattouw> |
Running sync-common-all to push out UploadWizard changes |
[production] |
| 08:30 |
<RoanKattouw> |
Deploying UploadWizard cahnges to testwiki |
[production] |
| 05:47 |
<Tim> |
restarted profile-stats-logger on fenari, it died when spence went down |
[production] |
| 05:39 |
<tstarling> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabled ehcache on all wikis' |
[production] |
| 05:29 |
<tstarling> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 're-enabled ehcache' |
[production] |
| 05:27 |
<Tim> |
got OOM. Restarted tomcat with substantially more than the default heap maximum of 128MB |
[production] |
| 05:19 |
<tstarling> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'disabling ehcache, something is wrong' |
[production] |
| 05:05 |
<tstarling> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'enable ehcache on enwiki' |
[production] |
| 05:01 |
<tstarling> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'enable ehcache on enwikiquote' |
[production] |
| 04:59 |
<tstarling> |
synchronized php-1.17/includes/GlobalFunctions.php 'live hack: send site notice cache to memcached not ehcache' |
[production] |
| 04:58 |
<tstarling> |
synchronized php-1.17/includes/MessageCache.php 'another one' |
[production] |
| 04:55 |
<tstarling> |
synchronized php-1.17/includes/Skin.php 'live hack to send sidebar cache to memcached not ehcache' |
[production] |
| 04:50 |
<tstarling> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabling ehcache client on testwiki and test2wiki' |
[production] |
| 04:46 |
<Tim> |
installed ehcache under tomcat on db40. Documented at [[Ehcache]]. |
[production] |
| 03:49 |
<Tim> |
did apt-get upgrade on db40, now rebooting to get new kernel |
[production] |
| 01:11 |
<mark> |
Reinstalled ms5 with Ubuntu 10.04, created software RAID10 array /dev/md1 out of 48 drives |
[production] |
|
2011-03-29
§
|
| 20:44 |
<mark> |
Moved ms5 to internal vlan 2 |
[production] |
| 20:34 |
<mark> |
Pushed DNS for ms5.pmtpa.wmnet |
[production] |
| 18:56 |
<mark> |
Changed NFS client params to rsize=8192,wsize=8192 on srv228:/mnt/thumbs |
[production] |
| 15:37 |
<ariel> |
synchronized php-1.17/wmf-config/CommonSettings.php 'increase account creation throttle value for el wiki for editing workshop' |
[production] |
| 10:41 |
<apergos> |
rsync on dataset2 for White Cat one last time, please shoot in three days (check with user first) |
[production] |
| 00:42 |
<Ryan_Lane> |
enabled ssl on RT, all non-http now forces to https for RT |
[production] |
| 00:42 |
<Ryan_Lane> |
fixed puppet issue on erzurumi by adding the wikidev group. |
[production] |