|
2011-05-10
§
|
| 21:02 |
<Ryan_Lane> |
powercycling srv281 since it's dead |
[production] |
| 21:01 |
<Ryan_Lane> |
powercycling srv247 since it's dead |
[production] |
| 20:56 |
<Ryan_Lane> |
mobile has stablized, moving weights back to normal |
[production] |
| 20:50 |
<Ryan_Lane> |
lowering weight on mobile1 and mobile2 |
[production] |
| 20:45 |
<Ryan_Lane> |
restarted apache on mobile2. |
[production] |
| 19:16 |
<mark> |
Moved all inter-subnet routing from csw5-pmtpa to csw1-pmtpa |
[production] |
| 19:16 |
<mark> |
Reconfigured pybal on lvs3 to BGP connect to csw1-sdtpa instead of csw5-pmtpa, and restarted it |
[production] |
| 17:53 |
<mark> |
Moved vlan 2 routing from csw5-pmtpa:ve2 to csw1-pmtpa:ve2 |
[production] |
| 16:27 |
<apergos> |
or at least srv254. |
[production] |
| 16:25 |
<apergos> |
nagios is whining about some services that are actually in operation (eg srv250) |
[production] |
| 16:07 |
<Ryan_Lane> |
or not, mgmt interface isn't letting me in |
[production] |
| 16:06 |
<Ryan_Lane> |
powercycling srv250 |
[production] |
| 13:56 |
<Tim> |
removed myself from SMS notification in nagios, since the configuration is so broken that it sends me a text every 10 minutes day or night |
[production] |
| 13:49 |
<RobH> |
nagios sms pages seem to be delayed from the actual service outage by a long, long delay |
[production] |
| 13:10 |
<Tim> |
did the above in both puppet and nagios directly since puppet seems to be still broken |
[production] |
| 13:07 |
<Tim> |
increased the number of retries of payments.wikimedia.org before sending an SMS from 1 to 20 |
[production] |
| 02:02 |
<notpeter> |
accidentally deleted sockpuppet:/var/lib/puppet/ssl am regnerating keys and will begin signing soon |
[production] |
|
2011-05-09
§
|
| 23:36 |
<notpeter> |
so annoying. doubling puppet freshness check timing to 4 hours. |
[production] |
| 22:01 |
<awjrichards> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'Disabling dashboard on enwiki pending fixes to data collection script' |
[production] |
| 21:20 |
<catrope> |
synchronizing Wikimedia installation... Revision: 87774: |
[production] |
| 21:20 |
<RoanKattouw> |
Running scap to deploy UploadWizard changes |
[production] |
| 20:54 |
<RoanKattouw> |
Running extensions/ArticleFeedback/populateAFStatistics.php on enwiki |
[production] |
| 20:49 |
<catrope> |
synchronized php-1.17/extensions/ArticleFeedback/populateAFStatistics.php 'r87770' |
[production] |
| 20:49 |
<catrope> |
synchronized php-1.17/extensions/ArticleFeedback/modules/ext.articleFeedback/ext.articleFeedback.js 'r87770' |
[production] |
| 20:30 |
<awjrichards> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enabling ArticleFeedback dashboard on testwiki' |
[production] |
| 20:28 |
<RoanKattouw> |
Running populateAFStatistics.php on testwiki |
[production] |
| 19:39 |
<catrope> |
synchronized php-1.17/resources/startup.js 'Dummy sync to push config change through in ResourceLoader' |
[production] |
| 19:39 |
<catrope> |
synchronized php-1.17/wmf-config/CommonSettings.php 'Re-enable AFT click tracking at 2%, and bump tracking version' |
[production] |
| 19:32 |
<catrope> |
synchronized php-1.17/resources/startup.js 'Force config changes to update in startup module' |
[production] |
| 19:21 |
<catrope> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'Raise AFT lottery odds to the target value of 2.7%' |
[production] |
| 19:03 |
<catrope> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'Raise AFT lottery odds to 2%' |
[production] |
| 18:41 |
<catrope> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'Raise AFT lottery odds to 1%' |
[production] |
| 18:39 |
<RobH> |
upgraded Reedy's cluster access to allow him to deploy and do shell requests, will take awhile to push out to the cluster |
[production] |
| 18:25 |
<catrope> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'Raise AFT lottery odds to 0.5% for real this time' |
[production] |
| 18:21 |
<catrope> |
synchronized php-1.17/wmf-config/CommonSettings.php 'Bump AFT tracking version, needed to actually change the click tracking percentage' |
[production] |
| 18:16 |
<demon> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 28495' |
[production] |
| 18:14 |
<Ryan_Lane> |
adding a shell account for asher (Asher Feldman), and adding him to roots |
[production] |
| 18:13 |
<catrope> |
synchronized php-1.17/wmf-config/CommonSettings.php 'Temporarily disable click tracking for AFT' |
[production] |
| 18:11 |
<catrope> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bump AFT lottery odds to 1%' |
[production] |
| 18:10 |
<catrope> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable AFT dashboard on enwiki' |
[production] |
| 18:09 |
<catrope> |
synchronized php-1.17/wmf-config/CommonSettings.php 'Bump ArticleFeedback tracking version' |
[production] |
| 18:01 |
<RoanKattouw> |
Running schema change (adding index of aa_timestamp field on enwiki.article_feedback table) on s1 by directly running it on the master. Table size is ~275k rows |
[production] |
| 17:56 |
<catrope> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'Set $wmgArticleFeedbackLotteryOdds to 0.005 on enwiki. This enables AFT on 0.5% of all enwiki main namespace pages' |
[production] |
| 17:44 |
<catrope> |
synchronizing Wikimedia installation... Revision: 87742: |
[production] |
| 17:43 |
<RoanKattouw> |
Running scap to deploy ArticleFeedback changes |
[production] |
| 16:49 |
<RoanKattouw> |
Deploying ArticleFeedback changes to testwiki |
[production] |
| 14:21 |
<apergos> |
restarting amssq39 squid back-end (memory hog) |
[production] |
| 13:41 |
<apergos> |
restarting squid back-end on amssq40 (memory hog) |
[production] |
| 13:27 |
<catrope> |
synchronized php-1.17/resources/mediawiki/mediawiki.js 'r87712' |
[production] |
| 13:27 |
<catrope> |
synchronized php-1.17/includes/OutputPage.php 'r87712' |
[production] |