|
2010-02-03
§
|
| 17:32 |
<mark> |
Set root password on all servers using Puppet |
[production] |
| 17:26 |
<Rob> |
loaded sq68 full with 6 ssd, had to 'borrow' drives from the unracked squids in pmtpa awaiting the rail issue resolution. |
[production] |
| 17:26 |
<Rob> |
connected sq67 and sq68 to secondary network ports as requested |
[production] |
| 16:43 |
<Rob> |
had to update gmetad and add sq14 and sq17 into puppet as ganglia aggregator hosts. Currently not showing in ganglia, but should when puppet updates their files shortly. |
[production] |
| 16:37 |
<Rob> |
updated gmetad_pmtpa in NFS store, as well as on zwinger and spence. |
[production] |
| 16:20 |
<Rob> |
!log cleaned sq1-sq10 certificates off sockpuppet |
[production] |
| 16:10 |
<mark> |
Rebooting sockpuppet for security upgrades |
[production] |
| 16:06 |
<Rob> |
updating DNS to remove lvs3.wikimedia.org, as its technically lvs3.pmtpa.wmnet |
[production] |
| 16:03 |
<Rob> |
removed sq1-sq10 in dsh groups, server roles, pybal (already done, just doublechecked), and now starting wipe and removing their network connections. |
[production] |
| 15:37 |
<Rob> |
updated video plugin on techblog, had to comment out sumotv support line due to errors. |
[production] |
| 13:30 |
<mark> |
Removed sq1-10 from the upload backend Squid pool, preparing for decommissioning |
[production] |
| 12:27 |
<mark> |
Removed eiximenis from the text backend pool |
[production] |
| 12:21 |
<mark> |
Restarted sq51 with hyperthreading disabled |
[production] |
| 11:31 |
<Tim-away> |
exim was flooding its mainlog with errors like "User 'exim' has exceeded the 'max_user_connections' resource (current value: 150)". OTRS is apparently broken as a result. Set the limit to infinity to fix it. |
[production] |
| 11:10 |
<Tim-away> |
debugging mysql connection errors from mchenry to db9 |
[production] |
| 11:02 |
<Tim-away> |
restarting exim4 on mchenry |
[production] |
| 10:57 |
<catrope> |
synchronized php-1.5/extensions/UsabilityInitiative_alpha/Vector/Vector.hooks.php 'r61922' |
[production] |
| 10:47 |
<RoanKattouw> |
Mail delivery reported broken again; last mail delivery to mediawiki-cvs was 3h22m ago |
[production] |
| 10:14 |
<catrope> |
synchronized php-1.5/wmf-config/CommonSettings.php 'Bump UsabilityInitiative_alpha style version' |
[production] |
| 10:13 |
<catrope> |
synchronized php-1.5/extensions/UsabilityInitiative_alpha/js/plugins/jquery.wikiEditor.html |
[production] |
| 10:13 |
<catrope> |
synchronized php-1.5/extensions/UsabilityInitiative_alpha/UsabilityInitiative.hooks.php |
[production] |
| 10:12 |
<catrope> |
synchronized php-1.5/extensions/UsabilityInitiative_alpha/js/plugins.combined.min.js |
[production] |
| 10:12 |
<catrope> |
synchronized php-1.5/extensions/UsabilityInitiative_alpha/Vector/Vector.hooks.php |
[production] |
| 10:12 |
<catrope> |
synchronized php-1.5/extensions/UsabilityInitiative_alpha/Vector/Vector.combined.min.js |
[production] |
| 10:12 |
<catrope> |
synchronized php-1.5/extensions/UsabilityInitiative_alpha/WikiEditor/WikiEditor.hooks.php |
[production] |
| 10:12 |
<catrope> |
synchronized php-1.5/extensions/UsabilityInitiative_alpha/WikiEditor/WikiEditor.combined.min.js |
[production] |
| 10:09 |
<RoanKattouw> |
Deploying r61919 with individual sync-files |
[production] |
| 07:45 |
<Tim> |
killed eximstats on mchenry, was sending the machine into swap |
[production] |
| 06:02 |
<Andrew> |
Running scap to deploy LiquidThreads alpha updates |
[production] |
| 05:09 |
<Andrew> |
Planning to update LiquidThreads alpha to trunk state in the next few minutes. |
[production] |
| 03:31 |
<Tim> |
removing refreshLinks2 jobs from the enwiki job queue with namespace=0, 100k of them put there by a nasty biography template |
[production] |
| 00:34 |
<domas> |
restarted varnish on db19 |
[production] |
|
2010-02-02
§
|
| 23:26 |
<mark> |
PyBal got into a confused state due to a duplicate LVS realserver entry. Set up LVS manually/statically for 30 mins to debug the problem. PyBal is now active again. |
[production] |
| 23:04 |
<Tim> |
pmtpa text squids down |
[production] |
| 22:33 |
<mark> |
Pooled sq59-68 in LVS (Text) |
[production] |
| 22:27 |
<apergos> |
restarted apache on srv170 (corrupted apc cache) |
[production] |
| 22:25 |
<mark> |
Increased CARP weight of the new Upload squids sq51-58 from 10 to 30 |
[production] |
| 22:17 |
<mark> |
Added sq59-66 to the Text pool of Squids, with low CARP weight (10) to seed the caches |
[production] |
| 21:44 |
<andrew> |
synchronized php-1.5/wmf-config/liquidthreads.php |
[production] |
| 21:42 |
<Andrew> |
temporarily disabling $wgLqtEnotif to mitigate mail server DoS impact on LiquidThreads post/reply speed. |
[production] |
| 21:42 |
<andrew> |
synchronized php-1.5/extensions/LiquidThreads_alpha/classes/NewMessagesController.php 'Deploy r61879' |
[production] |
| 21:42 |
<andrew> |
synchronized php-1.5/extensions/LiquidThreads/classes/NewMessagesController.php 'Deploy r61879' |
[production] |
| 21:04 |
<mark> |
Removed sq2-10 from the LVS pool |
[production] |
| 20:46 |
<catrope> |
synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version addition for usability alpha' |
[production] |
| 20:45 |
<catrope> |
synchronized php-1.5/skins/common/jquery.min.js 'Deploing r61700' |
[production] |
| 20:38 |
<mark> |
Added sq52-58 to the backend Squid pool with low CARP weight, to seed the caches |
[production] |
| 20:37 |
<catrope> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 22351: Enable Vector and toolbar on srwikinews' |
[production] |
| 20:36 |
<mark> |
Increased Exim load average threshold for delivery on mchenry |
[production] |
| 19:09 |
<Fred> |
rebooted singer (went down for unknown- at this time- reasons). |
[production] |
| 18:25 |
<Fred> |
OS-installed sq52-ssq58 |
[production] |