|
2011-03-30
§
|
| 13:03 |
<mark> |
Started 8 parallel rsyncs of ms4 /export/thumbs/wikipedia/commons/thumb/{8..f} to ms5 |
[production] |
| 10:50 |
<Tim> |
restarted ehcache with increased cache size limits |
[production] |
| 10:49 |
<tstarling> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 're-enabling ehcache' |
[production] |
| 10:45 |
<tstarling> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'disabling ehcache temporarily' |
[production] |
| 10:15 |
<apergos> |
rebooting ms5, trying to fix up raid config |
[production] |
| 08:52 |
<catrope> |
ran sync-common-all |
[production] |
| 08:52 |
<RoanKattouw> |
Running sync-common-all to push out UploadWizard changes |
[production] |
| 08:30 |
<RoanKattouw> |
Deploying UploadWizard cahnges to testwiki |
[production] |
| 05:47 |
<Tim> |
restarted profile-stats-logger on fenari, it died when spence went down |
[production] |
| 05:39 |
<tstarling> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabled ehcache on all wikis' |
[production] |
| 05:29 |
<tstarling> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 're-enabled ehcache' |
[production] |
| 05:27 |
<Tim> |
got OOM. Restarted tomcat with substantially more than the default heap maximum of 128MB |
[production] |
| 05:19 |
<tstarling> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'disabling ehcache, something is wrong' |
[production] |
| 05:05 |
<tstarling> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'enable ehcache on enwiki' |
[production] |
| 05:01 |
<tstarling> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'enable ehcache on enwikiquote' |
[production] |
| 04:59 |
<tstarling> |
synchronized php-1.17/includes/GlobalFunctions.php 'live hack: send site notice cache to memcached not ehcache' |
[production] |
| 04:58 |
<tstarling> |
synchronized php-1.17/includes/MessageCache.php 'another one' |
[production] |
| 04:55 |
<tstarling> |
synchronized php-1.17/includes/Skin.php 'live hack to send sidebar cache to memcached not ehcache' |
[production] |
| 04:50 |
<tstarling> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabling ehcache client on testwiki and test2wiki' |
[production] |
| 04:46 |
<Tim> |
installed ehcache under tomcat on db40. Documented at [[Ehcache]]. |
[production] |
| 03:49 |
<Tim> |
did apt-get upgrade on db40, now rebooting to get new kernel |
[production] |
| 01:11 |
<mark> |
Reinstalled ms5 with Ubuntu 10.04, created software RAID10 array /dev/md1 out of 48 drives |
[production] |
|
2011-03-29
§
|
| 20:44 |
<mark> |
Moved ms5 to internal vlan 2 |
[production] |
| 20:34 |
<mark> |
Pushed DNS for ms5.pmtpa.wmnet |
[production] |
| 18:56 |
<mark> |
Changed NFS client params to rsize=8192,wsize=8192 on srv228:/mnt/thumbs |
[production] |
| 15:37 |
<ariel> |
synchronized php-1.17/wmf-config/CommonSettings.php 'increase account creation throttle value for el wiki for editing workshop' |
[production] |
| 10:41 |
<apergos> |
rsync on dataset2 for White Cat one last time, please shoot in three days (check with user first) |
[production] |
| 00:42 |
<Ryan_Lane> |
enabled ssl on RT, all non-http now forces to https for RT |
[production] |
| 00:42 |
<Ryan_Lane> |
fixed puppet issue on erzurumi by adding the wikidev group. |
[production] |
|
2011-03-28
§
|
| 23:37 |
<Ryan_Lane> |
spence died; powercycling it |
[production] |
| 23:06 |
<awjrichards> |
synchronized php-1.17/extensions/UserDailyContribs/api/ApiUserDailyContribs.php 'Deploying bug fixes to UserDailyContribs API (r84935)' |
[production] |
| 22:41 |
<Ryan_Lane> |
added a certificate check for ldaps on nfs1/2 |
[production] |
| 22:36 |
<Ryan_Lane> |
pushed the internal CA certificate to spence, so that we can monitor the LDAP certificates |
[production] |
| 21:43 |
<awjr> |
running sync-comon all to deploy api updates to extension UserDailyContrib |
[production] |
| 21:43 |
<awjrichards> |
ran sync-common-all |
[production] |
| 21:38 |
<awjrichards> |
synchronized php-1.17/extensions/UserDailyContribs/api 'Attempting to create UserDailyContribs/api dir using sync-file' |
[production] |
| 21:07 |
<Ryan_Lane> |
pushed out a cert check for secure.wikimedia.org |
[production] |
| 19:17 |
<Ryan_Lane> |
changing weight of mobile1 to 10 |
[production] |
| 17:11 |
<catrope> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'Disable Vector/SectionEditLinks on enwiki' |
[production] |
| 16:58 |
<catrope> |
synchronized php-1.17/includes/api/ApiQueryCategoryMembers.php 'r84908' |
[production] |
| 14:58 |
<apergos> |
was seeing sluggish response from scalers, ms4, running out of swap space |
[production] |
| 14:54 |
<apergos> |
restarting web server on ms4... |
[production] |
| 05:58 |
<tstarling> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'disabling ehcache' |
[production] |
| 05:57 |
<Tim> |
ehcache broke, disabling client |
[production] |
| 05:39 |
<tstarling> |
synchronized php-1.17/wmf-config/InitialiseSettings.php |
[production] |
| 05:35 |
<Tim> |
db40 CPU usage is 5% of all cores, disk util <1%, network ~1%, request rate ~58% of weekly peak. Enabling ehcache on all wikis. |
[production] |
| 05:32 |
<Tim> |
fixing torrus per wikitech instructions |
[production] |
| 05:08 |
<tstarling> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabling ehcache on enwiki' |
[production] |
| 05:07 |
<Tim> |
resource utilisation on db40 is a nothingth of a percent, moving on to stage 4 |
[production] |
| 05:04 |
<tstarling> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabling ehcache for enwikiquote' |
[production] |