|
2012-10-15
§
|
| 20:21 |
<mwalker> |
Attempting to purge out another 500k pages with CentralNotice skin problem (this time not crashing the cluster by filtering out media files and rate limiting to no more than 50hps). Estimated run time is 3 hours. |
[production] |
| 20:20 |
<logmsgbot_> |
awjrichards synchronized php-1.21wmf1/extensions/MobileFrontend/ 'touch files' |
[production] |
| 20:10 |
<logmsgbot_> |
asher synchronized wmf-config/db.php 'pulling es10, third crash in three weeks' |
[production] |
| 20:07 |
<logmsgbot_> |
reedy synchronized php-1.21wmf1/extensions/Collection/ 'Revert back to 9f37005e78a068821c45d9d313394c6728f55b4a' |
[production] |
| 20:05 |
<logmsgbot_> |
reedy synchronized php-1.21wmf2/extensions/Collection/ 'Revert back to 9f37005e78a068821c45d9d313394c6728f55b4a' |
[production] |
| 19:57 |
<cmjohnson1> |
powering down srv266 troubleshooting per rt2896 |
[production] |
| 19:04 |
<binasher> |
restarted udp2log on oxygen with correct --recv-queue |
[production] |
| 18:58 |
<logmsgbot_> |
reedy synchronized wmf-config/ |
[production] |
| 18:55 |
<logmsgbot_> |
reedy synchronized php-1.21wmf1/includes/resourceloader/ResourceLoaderFileModule.php |
[production] |
| 18:49 |
<RoanKattouw> |
Created fluorine:/a/mw-log/udp2log/resourceloader.log chmod 644 and owned by udp2log:udp2log as root per Reedy's request |
[production] |
| 18:19 |
<logmsgbot_> |
reedy synchronized php-1.21wmf2/extensions/CodeReview |
[production] |
| 18:11 |
<logmsgbot_> |
reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.21wmf2 |
[production] |
| 18:07 |
<logmsgbot_> |
reedy synchronized php-1.21wmf1/extensions/MobileFrontend |
[production] |
| 17:32 |
<logmsgbot_> |
reedy synchronized wmf-config/CommonSettings.php 'wgDebugLogGroups for resourceloader' |
[production] |
| 17:30 |
<logmsgbot_> |
reedy synchronized php-1.21wmf1/includes/resourceloader/ResourceLoaderFileModule.php |
[production] |
| 16:28 |
<logmsgbot_> |
reedy synchronized php-1.21wmf2/extensions/MobileFrontend/ |
[production] |
| 16:26 |
<logmsgbot_> |
reedy synchronized php-1.21wmf1/extensions/MobileFrontend/ |
[production] |
| 16:16 |
<logmsgbot_> |
reedy synchronized wmf-config/InitialiseSettings.php 'Re-enable education program on enwiki' |
[production] |
| 16:06 |
<logmsgbot_> |
reedy Finished syncing Wikimedia installation... : Build localisation cache for 1.21wmf2 |
[production] |
| 15:43 |
<ottomata> |
stopping puppet on oxygen, looking for cause of recent packet loss |
[production] |
| 15:19 |
<logmsgbot_> |
reedy synchronized php-1.21wmf2/extensions/MobileFrontend |
[production] |
| 15:17 |
<logmsgbot_> |
reedy synchronized php-1.21wmf1/extensions/MobileFrontend |
[production] |
| 15:12 |
<logmsgbot_> |
reedy Started syncing Wikimedia installation... : Build localisation cache for 1.21wmf2 |
[production] |
| 14:56 |
<logmsgbot_> |
reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki to 1.21wmf2 |
[production] |
| 14:55 |
<logmsgbot_> |
reedy synchronized wmf-config/ |
[production] |
| 14:52 |
<logmsgbot_> |
reedy synchronized php-1.21wmf2/ 'Initial sync' |
[production] |
| 13:35 |
<Reedy> |
Killing php-1.20wmf11 l10n cache from apaches |
[production] |
| 13:34 |
<Reedy> |
Killing php-1.20wmf11 l10n cache from apaches |
[production] |
| 13:31 |
<Reedy> |
Pointing php to php-1.21wmf1 |
[production] |
| 13:29 |
<logmsgbot_> |
reedy synchronized wmf-config/throttle.php 'Updated from gitrepo' |
[production] |
| 13:28 |
<logmsgbot_> |
reedy synchronized wmf-config/CommonSettings.php 'Remove extdist conditional for mount path' |
[production] |
| 11:49 |
<mark> |
Deployed range_offset_limit 64 MB change (from 0) to all upload backend squids |
[production] |
| 11:24 |
<paravoid> |
restarting pdns @ ns2 |
[production] |
| 11:16 |
<paravoid> |
switching European bits back to esams |
[production] |
| 09:49 |
<paravoid> |
restarting all imagescaler apaches, did not recover after swift outage |
[production] |
| 09:24 |
<paravoid> |
live-hacking swift on ms-be10, ms-be12, ms-be1 to remove fadvise calls |
[production] |
| 07:53 |
<Tim> |
increasing worker count to 100 on all swift backends, via puppet |
[production] |
| 07:37 |
<Tim> |
ms-be11 showed an immediate improvement in bandwidth out, but wchan still indicates that 48 is not enough, increasing to 100 |
[production] |
| 07:33 |
<Tim> |
experimentally doubling the worker count on ms-be11 since wchan indicates that the worker pool is exhausted |
[production] |
| 05:49 |
<apergos> |
ms-be12 reporting errors on disk /dev/sdi, I've rt'ed it an unmounted it in the meantime |
[production] |
| 04:34 |
<Tim> |
on ms-fe2: restarted swift-proxy due to mem leak |
[production] |
| 04:28 |
<Tim> |
on ms-fe3: restarting swift-proxy due to swap |
[production] |
| 04:26 |
<LeslieCarr> |
restarted swift proxy on ms-fe4 - it was in swap doom |
[production] |
| 02:25 |
<logmsgbot_> |
LocalisationUpdate completed (1.21wmf1) at Mon Oct 15 02:25:12 UTC 2012 |
[production] |