|
2011-05-10
§
|
| 23:36 |
<Ryan_Lane> |
fixing torrus. why oh why must you corrupt yourself constantly, torrus? |
[production] |
| 23:06 |
<notpeter> |
distributing new puppet certs to all boxes managed by puppet |
[production] |
| 22:51 |
<tstarling> |
synchronized php-1.17/wmf-config/CommonSettings.php 'ED 1.17' |
[production] |
| 21:34 |
<awjrichards> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enabling Article Feedback dashboard on enwiki' |
[production] |
| 21:30 |
<awjrichards> |
synchronized php-1.17/extensions/ArticleFeedback/ArticleFeedback.php 'r87844' |
[production] |
| 21:29 |
<awjrichards> |
synchronized php-1.17/extensions/ArticleFeedback/ArticleFeedback.i18n.php 'r87844' |
[production] |
| 21:28 |
<awjrichards> |
synchronized php-1.17/extensions/ArticleFeedback/populateAFStatistics.php 'r87844' |
[production] |
| 21:28 |
<awjrichards> |
synchronized php-1.17/extensions/ArticleFeedback/SpecialArticleFeedback.php 'r87844' |
[production] |
| 21:02 |
<Ryan_Lane> |
powercycling srv281 since it's dead |
[production] |
| 21:01 |
<Ryan_Lane> |
powercycling srv247 since it's dead |
[production] |
| 20:56 |
<Ryan_Lane> |
mobile has stablized, moving weights back to normal |
[production] |
| 20:50 |
<Ryan_Lane> |
lowering weight on mobile1 and mobile2 |
[production] |
| 20:45 |
<Ryan_Lane> |
restarted apache on mobile2. |
[production] |
| 19:16 |
<mark> |
Moved all inter-subnet routing from csw5-pmtpa to csw1-pmtpa |
[production] |
| 19:16 |
<mark> |
Reconfigured pybal on lvs3 to BGP connect to csw1-sdtpa instead of csw5-pmtpa, and restarted it |
[production] |
| 17:53 |
<mark> |
Moved vlan 2 routing from csw5-pmtpa:ve2 to csw1-pmtpa:ve2 |
[production] |
| 16:27 |
<apergos> |
or at least srv254. |
[production] |
| 16:25 |
<apergos> |
nagios is whining about some services that are actually in operation (eg srv250) |
[production] |
| 16:07 |
<Ryan_Lane> |
or not, mgmt interface isn't letting me in |
[production] |
| 16:06 |
<Ryan_Lane> |
powercycling srv250 |
[production] |
| 13:56 |
<Tim> |
removed myself from SMS notification in nagios, since the configuration is so broken that it sends me a text every 10 minutes day or night |
[production] |
| 13:49 |
<RobH> |
nagios sms pages seem to be delayed from the actual service outage by a long, long delay |
[production] |
| 13:10 |
<Tim> |
did the above in both puppet and nagios directly since puppet seems to be still broken |
[production] |
| 13:07 |
<Tim> |
increased the number of retries of payments.wikimedia.org before sending an SMS from 1 to 20 |
[production] |
| 02:02 |
<notpeter> |
accidentally deleted sockpuppet:/var/lib/puppet/ssl am regnerating keys and will begin signing soon |
[production] |
|
2011-05-09
§
|
| 23:36 |
<notpeter> |
so annoying. doubling puppet freshness check timing to 4 hours. |
[production] |
| 22:01 |
<awjrichards> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'Disabling dashboard on enwiki pending fixes to data collection script' |
[production] |
| 21:20 |
<catrope> |
synchronizing Wikimedia installation... Revision: 87774: |
[production] |
| 21:20 |
<RoanKattouw> |
Running scap to deploy UploadWizard changes |
[production] |
| 20:54 |
<RoanKattouw> |
Running extensions/ArticleFeedback/populateAFStatistics.php on enwiki |
[production] |
| 20:49 |
<catrope> |
synchronized php-1.17/extensions/ArticleFeedback/populateAFStatistics.php 'r87770' |
[production] |
| 20:49 |
<catrope> |
synchronized php-1.17/extensions/ArticleFeedback/modules/ext.articleFeedback/ext.articleFeedback.js 'r87770' |
[production] |
| 20:30 |
<awjrichards> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enabling ArticleFeedback dashboard on testwiki' |
[production] |
| 20:28 |
<RoanKattouw> |
Running populateAFStatistics.php on testwiki |
[production] |
| 19:39 |
<catrope> |
synchronized php-1.17/resources/startup.js 'Dummy sync to push config change through in ResourceLoader' |
[production] |
| 19:39 |
<catrope> |
synchronized php-1.17/wmf-config/CommonSettings.php 'Re-enable AFT click tracking at 2%, and bump tracking version' |
[production] |
| 19:32 |
<catrope> |
synchronized php-1.17/resources/startup.js 'Force config changes to update in startup module' |
[production] |
| 19:21 |
<catrope> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'Raise AFT lottery odds to the target value of 2.7%' |
[production] |
| 19:03 |
<catrope> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'Raise AFT lottery odds to 2%' |
[production] |
| 18:41 |
<catrope> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'Raise AFT lottery odds to 1%' |
[production] |
| 18:39 |
<RobH> |
upgraded Reedy's cluster access to allow him to deploy and do shell requests, will take awhile to push out to the cluster |
[production] |
| 18:25 |
<catrope> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'Raise AFT lottery odds to 0.5% for real this time' |
[production] |