|
2012-02-17
§
|
| 21:30 |
<notpeter> |
remounted nfs mounts on searchidx2. to protect our house of cards |
[production] |
| 21:03 |
<notpeter> |
unmounting ms nfs mounts from searchidx2 |
[production] |
| 20:44 |
<notpeter> |
restarting puppet on brewster |
[production] |
| 20:33 |
<nikerabbit> |
synchronized wmf-config/InitialiseSettings.php 'Bug 34235 - Enable WebFonts on am.wikipedia' |
[production] |
| 20:13 |
<maplebed> |
sending thumbnail traffic back to swift; head bug is fixed. |
[production] |
| 20:10 |
<RobH> |
reinstalling search1001 and searchidx1001 |
[production] |
| 19:53 |
<RobH> |
search1001 down for reinstall |
[production] |
| 19:37 |
<RobH> |
ran dsh command to remove all the /tmp/mw-cache-1.17 |
[production] |
| 19:27 |
<LeslieCarr> |
manually cleaned up tmp on mw41 |
[production] |
| 19:27 |
<RobH> |
manually cleaned up tmp on mw21 |
[production] |
| 19:01 |
<LeslieCarr> |
changed sockpuppet's post-merge hook so that you need to have ssh keys forwarded (though you really would anyways due to brokenness) |
[production] |
| 18:51 |
<aaron> |
synchronized php-1.18/includes/api/ApiQueryAllUsers.php |
[production] |
| 18:50 |
<reedy> |
synchronized wmf-config/ 'Sync ExtensionMessages' |
[production] |
| 18:44 |
<reedy> |
synchronized php-1.19/extensions/WikimediaMessages/ |
[production] |
| 18:43 |
<reedy> |
synchronized php-1.18/extensions/WikimediaMessages/ |
[production] |
| 18:22 |
<aaron> |
synchronized php-1.18/includes/api/ApiQueryAllUsers.php |
[production] |
| 18:19 |
<aaron> |
synchronized php-1.18/includes/api/ApiQueryAllUsers.php 'force index' |
[production] |
| 18:06 |
<maplebed> |
sending thumbnail traffic back to ms5, taking swift out of production |
[production] |
| 18:00 |
<notpeter> |
temporarily stopping puppet on brewster. please let me know if you need to turn it back on |
[production] |
| 17:52 |
<maplebed> |
changing squids to send 100% of thumbnail traffic to swift |
[production] |
| 16:58 |
<maplebed> |
turned swift live for 50% of all thumbnail requests |
[production] |
| 15:02 |
<hashar> |
synchronized php-1.19/LocalSettings.php 'make jobrunners for testwiki to use the apache CommonSettings file instead of the non existant /home/wikipedia one' |
[production] |
| 14:59 |
<reedy> |
synchronized php-1.19/includes/specials/SpecialDeletedContributions.php '[[rev:111752|r111752]]' |
[production] |
| 14:58 |
<hashar> |
synchronized php-1.19/LocalSettings.php |
[production] |
| 14:40 |
<hashar> |
synchronized php-1.19/LocalSettings.php 'debug statement with argv join' |
[production] |
| 14:37 |
<hashar> |
synchronized php-1.19/LocalSettings.php 'syslog debug statement to investigate TESTWIKI issue' |
[production] |
| 13:40 |
<^demon> |
srv233: removed /tmp/mw-cache-1.17 to give it a little more space for now |
[production] |
| 09:58 |
<mark> |
Shutdown ragweed for decommissioning |
[production] |
| 07:42 |
<binasher> |
upgraded mysql on db40 to 5.1.53-facebook-[[rev:3753|r3753]], enabled innodb_use_purge_thread |
[production] |
| 05:39 |
<tstarling> |
synchronized wmf-config/InitialiseSettings.php |
[production] |
| 05:38 |
<tstarling> |
synchronized wmf-config/CommonSettings.php |
[production] |
| 05:35 |
<Tim> |
on db40: reduced to 10M, should be causing massive delays, but the site's not down and the purge rate is lower if anything. Going to disable the mysql parser cache entirely. |
[production] |
| 05:26 |
<Tim> |
on db40: purge lag is still increasing at 108 per second, so reducing innodb_max_purge_lag to 50M |
[production] |
| 05:21 |
<Tim> |
on db40: giving the innodb manual the benefit of the doubt and following its advice, setting innodb_max_purge_lag to 100M, which should give a delay of 4.5ms |
[production] |
| 05:13 |
<Tim> |
killing purgeParserCache.php since it is probably doing more harm than good |
[production] |
| 02:43 |
<maplebed> |
deployed updated thumb_handler.php to ms5 to include Content-Length in generated images |
[production] |
| 02:34 |
<LocalisationUpdate> |
completed (1.19) at Fri Feb 17 02:34:32 UTC 2012 |
[production] |
| 02:29 |
<Ryan_Lane> |
installed labstore1-4 |
[production] |
| 02:17 |
<LocalisationUpdate> |
completed (1.18) at Fri Feb 17 02:17:35 UTC 2012 |
[production] |
| 02:16 |
<Tim> |
on db40: truncating pc008 - 15 |
[production] |
| 02:01 |
<binasher> |
db1035 is replicating again |
[production] |
| 01:12 |
<Ryan_Lane> |
re-enabled the mobile plugin for the blogs, seems w3 total cache supports varying |
[production] |
| 01:03 |
<Ryan_Lane> |
disabling mobile skin for the blogs - we need to fix varnish support first |
[production] |
| 00:38 |
<Ryan_Lane> |
fixed singer by adding in ssl configuration to the planet configuration |
[production] |