|
2010-01-11
§
|
| 23:38 |
<domas> |
logging the fact that we had cache layer meltdown at some point in time during the day |
[production] |
| 22:30 |
<domas> |
leaving bits.pmtpa on db19's varnish, in case of troubles - uncomment bits.pmtpa .2 record in /etc/powerdns/templates/wikimedia.org and run authdns-update |
[production] |
| 19:43 |
<fvassard> |
synchronized php-1.5/wmf-config/mc.php 'Swapped memcached from srv125 to srv232' |
[production] |
| 19:06 |
<Rob> |
new apaches srv255, srv257 deployed. Updated node groups and synced nagios |
[production] |
| 19:03 |
<Rob> |
new apache server srv254 deployed |
[production] |
| 18:24 |
<atglenn> |
copy backlog of image data from ms1 to ms7 (running in screen as root on both boxes) |
[production] |
| 14:43 |
<mark> |
Rebooting fuchsia, locked up again |
[production] |
| 14:24 |
<mark> |
Increased load on knsq16-22 by upping lvs weight from 10 to 15 |
[production] |
|
2010-01-10
§
|
| 23:02 |
<midom> |
synchronized php-1.5/wmf-config/lucene.php 'rainman asked, rainman guilty, hehehe' |
[production] |
| 23:01 |
<midom> |
synchronized php-1.5/wmf-config/secure.php |
[production] |
| 17:36 |
<rainman-sr> |
search limit raised to 500 again, interwiki search re-enabled for "other" wikis |
[production] |
| 16:07 |
<kate> |
synchronized php-1.5/wmf-config/db.php 'take ixia back out' |
[production] |
| 16:06 |
<kate> |
synchronized php-1.5/wmf-config/db.php 'put ixia back' |
[production] |
| 16:06 |
<rainman-sr> |
restarting search cluster to deploy search13-19 |
[production] |
| 15:58 |
<domas> |
all bits serving switched back to text cluster, we have problems with all threads blocking on write(): http://p.defau.lt/?dOBxveiHj_ukjzupEBX3rA |
[production] |
| 15:19 |
<rainman-sr> |
configuring search13-19, will leave search20 as spare |
[production] |
| 14:22 |
<domas> |
apparently varnish worker threads are blocking on network output, ... :) |
[production] |
| 12:40 |
<domas> |
full bits pmtpa load sent to sq1 |
[production] |
| 12:04 |
<domas> |
sending half of bits load to sq1 |
[production] |
| 12:02 |
<domas> |
set up separate geo balancing for bits via bits-geo.wikimedia.org |
[production] |
| 10:45 |
<midom> |
synchronized php-1.5/wmf-config/CommonSettings.php 'setting extension asset path to bits.wm' |
[production] |
| 10:42 |
<midom> |
synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php |
[production] |
| 10:29 |
<midom> |
synchronized php-1.5/includes/Setup.php |
[production] |
| 10:29 |
<midom> |
synchronized php-1.5/includes/DefaultSettings.php |
[production] |
| 09:36 |
<domas> |
moving over static assets to 'bits.wikimedia.org' |
[production] |
| 09:11 |
<midom> |
synchronized php-1.5/wmf-config/CommonSettings.php |
[production] |
| 09:09 |
<midom> |
synchronized php-1.5/wmf-config/secure.php |
[production] |
| 08:26 |
<kate> |
synchronized php-1.5/wmf-config/db.php |
[production] |
| 08:25 |
<river> |
taking ixia out of rotation to dump commons |
[production] |
|
2010-01-07
§
|
| 20:22 |
<catrope> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 22038: Enable Collection extension on skwiki' |
[production] |
| 20:21 |
<catrope> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 19418: Fix $wgUploadNavigationUrl on bnwiki' |
[production] |
| 20:16 |
<catrope> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21517: Enable patrolling on frwikiversity' |
[production] |
| 19:54 |
<Rob> |
base install done on mobile2, pdf2, transcode1, transcode2 |
[production] |
| 16:55 |
<Rob> |
srv254-srv257 racked, drac online, network attached. needs installs |
[production] |
| 15:46 |
<Rob> |
racked srv254-srv257. DRAC setup, cables run. Network not plugged in until mark provisions the ports. |
[production] |
| 02:51 |
<Tim> |
restarted apache2 on wikitech to fix swapping |
[production] |
| 02:28 |
<Fred> |
rebooted pascal as it was once again hung |
[production] |
| 02:07 |
<mark> |
Prepended AS 43821 once on incoming prefixes from AS 1299 |
[production] |