|
2010-02-13
§
|
| 17:02 |
<mark> |
Throwing all API requests at the new API app server cluster |
[production] |
| 16:49 |
<mark> |
Converted srv187-srv189 and srv250-srv253 into API app servers |
[production] |
| 15:46 |
<mark> |
Created new Ganglia group "API application servers", deploying with Puppet |
[production] |
| 15:31 |
<mark> |
Created API app servers group in Puppet, deploying on srv254-257 to start with |
[production] |
| 15:24 |
<mark> |
Created API LVS service on lvs3 and restarted PyBal |
[production] |
| 15:10 |
<mark> |
Added DNS entry for api.svc.pmpta.wmnet (LVS ip) |
[production] |
| 14:18 |
<mark> |
Enabled more verbose logging on all squids: debug message level 1 for all message categories |
[production] |
| 14:18 |
<mark> |
Disabled cache digests on all squids |
[production] |
| 13:58 |
<mark> |
Removing sq30 and lower from the Text CARP pool |
[production] |
| 13:51 |
<mark> |
Reducing persistent_connection_timeout and request_timeout from 1 min to 20s on all frontend squids, to reduce FD usage |
[production] |
| 12:50 |
<mark> |
Restarting knsq* frontend squids with FD limit raised from 32k to 64k |
[production] |
| 12:23 |
<mark> |
Grown filesystem /dev/data/ES (/a) on ms3 by 90G, leaving another 98G free |
[production] |
| 00:17 |
<catrope> |
synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php 'Sync CSS style versions' |
[production] |
| 00:16 |
<catrope> |
synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix' |
[production] |
| 00:10 |
<catrope> |
synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r62403' |
[production] |
| 00:02 |
<catrope> |
synchronized php-1.5/extensions/UsabilityInitiative/WikiEditor/WikiEditor.combined.min.js 'r62400' |
[production] |
| 00:01 |
<catrope> |
synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r62400' |
[production] |
|
2010-02-12
§
|
| 23:55 |
<catrope> |
synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'Try deploying r62397 again' |
[production] |
| 23:46 |
<catrope> |
synchronized php-1.5/extensions/UsabilityInitiative/css/combined.min.css 'r62397' |
[production] |
| 23:46 |
<catrope> |
synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r62397' |
[production] |
| 23:22 |
<mark> |
Deployed sq86 |
[production] |
| 22:31 |
<catrope> |
synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix' |
[production] |
| 22:30 |
<catrope> |
synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r62388' |
[production] |
| 21:12 |
<mark> |
Implemented new routing policy for AS43821 |
[production] |
| 21:01 |
<catrope> |
synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix' |
[production] |
| 20:59 |
<catrope> |
synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r62382' |
[production] |
| 19:15 |
<Rob> |
sq67 has both disks installed now (had to get one replaced) and has no OS installation yet. holding off on install, not sure if mark wants this up as a squid or for varnish development |
[production] |
| 18:22 |
<atglenn> |
resstarting transfer of data (one chunk) from ms8->ms5->ms6, running in screen as root on all three hosts using nc |
[production] |
| 16:18 |
<Rob> |
setup project1 and project2 boxes for flaggedrevsdevelopment and project mangement suite testing |
[production] |
| 15:12 |
<mark> |
Upgraded our AMS-IX connection from 2x 1G to 10G along the way |
[production] |
| 15:12 |
<mark> |
Reloaded br1-knams twice to fix a CAM partitioning problem; not sufficient IP next hops for 8-path routes |
[production] |
| 11:34 |
<mark> |
Filtering all prefixes to/from AMS-IX peers |
[production] |
| 02:55 |
<tstarling> |
synchronized php-1.5/wmf-config/CommonSettings.php 're-enabling WikiEditor' |
[production] |
| 02:04 |
<Tim> |
installed openssh-server on bayes, Erik Z apparently uninstalled it with aptitude |
[production] |
| 01:37 |
<tstarling> |
synchronized php-1.5/wmf-config/CommonSettings.php 'disabled WikiEditor due to complaints about bug 22428' |
[production] |
| 01:06 |
<atglenn> |
shoveling over some data from ms8 to ms6 via ms5 :-/ |
[production] |
| 00:48 |
<andrew> |
synchronized php-1.5/extensions/LiquidThreads/lqt.js 'Disable live preview entirely, broken by CentralNotice and DismissableSiteNotice' |
[production] |
| 00:37 |
<andrew> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disable DismissableSiteNotice for lqt.labs, breaks LiquidThreads live preview' |
[production] |
| 00:17 |
<andrew> |
synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version' |
[production] |
| 00:07 |
<Andrew> |
Running scap |
[production] |
| 00:02 |
<Andrew> |
Updating LiquidThreads alpha to trunk state, deploying with scap in a few minutes |
[production] |
|
2010-02-11
§
|
| 22:06 |
<Fred> |
implemented disk space check for tridge |
[production] |
| 21:10 |
<mark> |
Pooled sq71-85 with full CARP weight (30) |
[production] |
| 21:06 |
<mark> |
Pooled sq79-85 frontends in LVS with full load (30) |
[production] |
| 20:59 |
<mark> |
Pooled sq79-85 frontends in LVS with load 1 |
[production] |
| 20:51 |
<mark> |
Pooling sq79-85 in CARP on all frontends, with low CARP weight 10 instead of 30 |
[production] |
| 20:30 |
<mark> |
Pooling sq79-85 in CARP on frontend sq51 to seed the cache |
[production] |
| 19:39 |
<atglenn> |
replication every 15 minutes enabled from ms7 to ms8 |
[production] |
| 19:31 |
<mark> |
Removed sq16-30 from the text frontend squid pool in LVS - these servers will soon be decommissioned |
[production] |
| 19:28 |
<mark> |
Pooled sq71-78 frontend squids in LVS with full load (30) |
[production] |