|
2010-01-28
§
|
| 20:15 |
<Rob> |
racked sq51-sq68, all drac accessible, wired, ready for install |
[production] |
| 19:05 |
<Rob> |
updated dns for management ranges on new squid servers |
[production] |
| 13:56 |
<mark> |
Added DNS entries for 36 new Squid servers |
[production] |
| 13:28 |
<mark> |
Storage1 went down due to I/O failure |
[production] |
| 13:00 |
<catrope> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'Remove aliases made obsolete by r61632' |
[production] |
| 12:55 |
<catrope> |
synchronized php-1.5/languages/messages/MessagesMl.php 'Deploying r61632' |
[production] |
| 10:57 |
<aaron> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'supressredirect for rollbackers on ruwikisource' |
[production] |
| 09:30 |
<aaron> |
synchronized php-1.5/wmf-config/flaggedrevs.php 'Removed redundant huwiki settings' |
[production] |
|
2010-01-27
§
|
| 18:04 |
<mark> |
Fixed ExtensionDistributor, which broke due to the image server migration |
[production] |
| 18:02 |
<mark> |
synchronized php-1.5/extensions/ExtensionDistributor/svn-invoker.php 's/upload5/upload6/' |
[production] |
| 16:10 |
<mark> |
Added a timeout to puppet's apt-get update execution; hopefully this will prevent things from getting stuck for weeks |
[production] |
| 16:08 |
<mark> |
Shutdown sq20 once again, please decommission this broken box |
[production] |
| 16:05 |
<mark> |
Fixed puppet on db7 |
[production] |
| 16:05 |
<mark> |
Killed stuck apt-get update processes on all servers |
[production] |
| 15:58 |
<mark> |
Made puppet manage NFS mounts /mnt/upload6, /mnt/thumbs and the absence of /mnt/upload5 on all application servers and image scalers |
[production] |
| 13:37 |
<catrope> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'Fix Category talk alias on mlwiki' |
[production] |
| 11:57 |
<catrope> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'Also add a Unicode 5.1 alias for Category talk on mlwiki' |
[production] |
| 11:52 |
<catrope> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 22286: Unicode 5.1 namespace alias for Category on mlwiki' |
[production] |
| 11:46 |
<catrope> |
synchronized php-1.5/includes/api/ApiQueryBacklinks.php 'Deploying r61571' |
[production] |
| 08:12 |
<andrew> |
synchronized php-1.5/extensions/LiquidThreads/lqt.js |
[production] |
| 08:11 |
<andrew> |
synchronized php-1.5/extensions/LiquidThreads_alpha/lqt.js |
[production] |
|
2010-01-26
§
|
| 23:44 |
<fvassard> |
synchronized php-1.5/wmf-config/CommonSettings.php 'Enable file uploads.' |
[production] |
| 23:34 |
<Fred> |
restarted morebot |
[production] |
| 21:10 |
<atglenn> |
copying uploaded data since last replication, from ms1 to ms7 |
[production] |
| 21:05 |
<atglenn> |
turned off file uploads temporarily |
[production] |
| 21:04 |
<ariel> |
synchronized wmf-deployment/wmf-config/CommonSettings.php |
[production] |
| 20:50 |
<Rob> |
rebooted srv125 per dc task entry by fred |
[production] |
| 20:35 |
<tomaszf> |
enabling global file uploads notice |
[production] |
| 20:29 |
<catrope> |
synchronized php-1.5/extensions/GlobalUsage/GlobalUsage.php 'Deploying r61534' |
[production] |
| 20:29 |
<catrope> |
synchronized php-1.5/extensions/GlobalUsage/GlobalUsageHooks.php 'Deploying r61534' |
[production] |
| 18:38 |
<Rob> |
shutting down pdf2 to relocate its rack position. |
[production] |
| 18:37 |
<Rob> |
!log mobile2 port not moved yet, as its port assignment on wikitech is invalid |
[production] |
| 18:08 |
<Rob> |
shutting down mobile2 to relocate its rack location |
[production] |
| 18:07 |
<Rob> |
bayes back online |
[production] |
| 18:00 |
<Rob> |
bayes management lom had ip issues, resolved, still working on host |
[production] |
| 17:52 |
<Rob> |
bayes is not coming back online properly, checking. |
[production] |
| 17:47 |
<Rob> |
bayes rack location moved and updated in racktables, powering back up |
[production] |
| 17:34 |
<Rob> |
shutting down bayes to move its rack location from b2-pmtpa to a2-pmtpa |
[production] |
| 14:42 |
<catrope> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 22274: Set $wgAutoConfirmCount to 10 on itwiktionary' |
[production] |
| 00:56 |
<Tim> |
on hume: increased RCT proc count, reduced max lag to 1s, increased lag check frequency |
[production] |
|
2010-01-25
§
|
| 22:36 |
<Tim> |
ms3 is reporting "table full" for rc1 enwiki.blobs. Running ALTER TABLE. |
[production] |
| 19:33 |
<Fred> |
stopped apache on srv225 |
[production] |
| 19:01 |
<Fred> |
adjusted apache.conf on srv225 to make it work again... |
[production] |
| 16:07 |
<Rob> |
powering srv224 & srv225 back up |
[production] |
| 16:01 |
<Rob> |
shutting down srv225 & srv224 to swap their power cables from single to Y split. |
[production] |
| 15:45 |
<Rob> |
srv196 memory replaced, back online and in service |
[production] |
| 15:45 |
<root> |
synchronized php-1.5/wmf-config/mc.php 'pushing out change to two hosts that were down a few minutes ago' |
[production] |
| 15:40 |
<Rob> |
shutting down db28 to swap memory around in diagnostics (bad memory in system) |
[production] |
| 15:25 |
<root> |
synchronized php-1.5/wmf-config/mc.php 'took down srv196 for hardware replacement' |
[production] |
| 15:20 |
<Rob> |
shutting down srv196 to replace faulty memory dimm |
[production] |