|
2010-02-12
§
|
| 02:04 |
<Tim> |
installed openssh-server on bayes, Erik Z apparently uninstalled it with aptitude |
[production] |
| 01:37 |
<tstarling> |
synchronized php-1.5/wmf-config/CommonSettings.php 'disabled WikiEditor due to complaints about bug 22428' |
[production] |
| 01:06 |
<atglenn> |
shoveling over some data from ms8 to ms6 via ms5 :-/ |
[production] |
| 00:48 |
<andrew> |
synchronized php-1.5/extensions/LiquidThreads/lqt.js 'Disable live preview entirely, broken by CentralNotice and DismissableSiteNotice' |
[production] |
| 00:37 |
<andrew> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disable DismissableSiteNotice for lqt.labs, breaks LiquidThreads live preview' |
[production] |
| 00:17 |
<andrew> |
synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version' |
[production] |
| 00:07 |
<Andrew> |
Running scap |
[production] |
| 00:02 |
<Andrew> |
Updating LiquidThreads alpha to trunk state, deploying with scap in a few minutes |
[production] |
|
2010-02-11
§
|
| 22:06 |
<Fred> |
implemented disk space check for tridge |
[production] |
| 21:10 |
<mark> |
Pooled sq71-85 with full CARP weight (30) |
[production] |
| 21:06 |
<mark> |
Pooled sq79-85 frontends in LVS with full load (30) |
[production] |
| 20:59 |
<mark> |
Pooled sq79-85 frontends in LVS with load 1 |
[production] |
| 20:51 |
<mark> |
Pooling sq79-85 in CARP on all frontends, with low CARP weight 10 instead of 30 |
[production] |
| 20:30 |
<mark> |
Pooling sq79-85 in CARP on frontend sq51 to seed the cache |
[production] |
| 19:39 |
<atglenn> |
replication every 15 minutes enabled from ms7 to ms8 |
[production] |
| 19:31 |
<mark> |
Removed sq16-30 from the text frontend squid pool in LVS - these servers will soon be decommissioned |
[production] |
| 19:28 |
<mark> |
Pooled sq71-78 frontend squids in LVS with full load (30) |
[production] |
| 19:21 |
<mark> |
Pooled sq71-78 frontend squids in LVS with low load (1) to seed caches |
[production] |
| 18:30 |
<mark> |
Pooling sq71-78 backend squids on all frontends, with lower CARP weight (10 instead of 30) to seed the caches |
[production] |
| 18:04 |
<mark> |
Deployed new squid config to sq58-66 frontend squid, to seed the caches of backend squids sq71-78 |
[production] |
| 17:27 |
<mark> |
Deployed new squid config to sq66 frontend squid, to seed the caches of backend squids sq71-78 |
[production] |
| 17:24 |
<mark> |
Deployed Squid with correct configuration on sq71-78 |
[production] |
| 11:08 |
<RoanKattouw> |
Load spike on 4CPU Apaches ended, rr.knams back up, downtime seems to be over |
[production] |
| 11:05 |
<RoanKattouw> |
Another load spike on the 8 CPU Apaches, approx 11:02-11:-05 UTC |
[production] |
| 11:00 |
<RoanKattouw> |
Apache load spike started around 10:42 UTC and coincides with a load dip followed by a monitoring blackout on the Kennisnet Squids and a load spike on the Tampa Squids |
[production] |
| 10:58 |
<RoanKattouw> |
8CPU Apaches stopped going bonkers all of a sudden according to Ganglia, 4CPU ones still have high CPU usage |
[production] |
| 10:56 |
<RoanKattouw> |
Site went down, all Apaches have high CPU usage |
[production] |
|
2010-02-10
§
|
| 22:31 |
<catrope> |
synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix to deploy usability changes sitewide' |
[production] |
| 22:22 |
<catrope> |
synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'Deploy r62275 to test' |
[production] |
| 21:28 |
<catrope> |
synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'Deploy r62264 for real. Running svn up first helps' |
[production] |
| 21:19 |
<catrope> |
synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'Deploy r62264' |
[production] |
| 21:03 |
<andrew> |
synchronized php-1.5/wmf-config/InitialiseSettings.php |
[production] |
| 21:02 |
<Andrew> |
Switching strategywiki to opt-out LiquidThreads, rather than opt-in |
[production] |
| 20:46 |
<catrope> |
synchronized php-1.5/extensions/UsabilityInitiative/WikiEditor/WikiEditor.combined.min.js 'Deploy r62262' |
[production] |
| 20:46 |
<catrope> |
synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'Deploy r62262' |
[production] |
| 19:22 |
<catrope> |
synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix' |
[production] |
| 19:21 |
<catrope> |
synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'Deploying r62257' |
[production] |
| 19:05 |
<Rob> |
storage1 disks replaced, except for the newly dead one, rma placed. |
[production] |
| 18:39 |
<Rob> |
shutting down storage1 to replace its bad OS mirror disk |
[production] |
| 16:52 |
<mark> |
Fixed MySQL permissions on srv186 as well |
[production] |
| 16:15 |
<mark> |
Rigged puppet to deploy a PHP mail.ini file to the apaches, which sets it to call sendmail with -f <> (sending mail with empty envelope sender, like bounces) |
[production] |
|
2010-02-09
§
|
| 22:06 |
<Andrew> |
LiquidThreads production software updates completed successfully. |
[production] |
| 21:58 |
<andrew> |
synchronized php-1.5/extensions/LiquidThreads/classes/Dispatch.php |
[production] |
| 21:57 |
<mark> |
Firewalled one ip on singer which was overloading Apache, looked like a Wordpress attack |
[production] |
| 21:50 |
<mark> |
Lowered MaxClients setting to 250 on singer |
[production] |
| 21:47 |
<Andrew> |
Running scap to deploy LiquidThreads updates |
[production] |
| 21:45 |
<Andrew> |
Updating LiquidThreads production deployments to the alpha version in the next few minutes. |
[production] |
| 21:44 |
<mark> |
Power cycled singer |
[production] |
| 21:19 |
<mark> |
Fixed MySQL system user & permissions on srv151-srv185 |
[production] |
| 21:17 |
<mark> |
Fixed MySQL instance on srv183 |
[production] |