2011-05-10
§
|
23:36 |
<Ryan_Lane> |
fixing torrus. why oh why must you corrupt yourself constantly, torrus? |
[production] |
23:06 |
<notpeter> |
distributing new puppet certs to all boxes managed by puppet |
[production] |
22:51 |
<tstarling> |
synchronized php-1.17/wmf-config/CommonSettings.php 'ED 1.17' |
[production] |
21:34 |
<awjrichards> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enabling Article Feedback dashboard on enwiki' |
[production] |
21:30 |
<awjrichards> |
synchronized php-1.17/extensions/ArticleFeedback/ArticleFeedback.php 'r87844' |
[production] |
21:29 |
<awjrichards> |
synchronized php-1.17/extensions/ArticleFeedback/ArticleFeedback.i18n.php 'r87844' |
[production] |
21:28 |
<awjrichards> |
synchronized php-1.17/extensions/ArticleFeedback/populateAFStatistics.php 'r87844' |
[production] |
21:28 |
<awjrichards> |
synchronized php-1.17/extensions/ArticleFeedback/SpecialArticleFeedback.php 'r87844' |
[production] |
21:02 |
<Ryan_Lane> |
powercycling srv281 since it's dead |
[production] |
21:01 |
<Ryan_Lane> |
powercycling srv247 since it's dead |
[production] |
20:56 |
<Ryan_Lane> |
mobile has stablized, moving weights back to normal |
[production] |
20:50 |
<Ryan_Lane> |
lowering weight on mobile1 and mobile2 |
[production] |
20:45 |
<Ryan_Lane> |
restarted apache on mobile2. |
[production] |
19:16 |
<mark> |
Moved all inter-subnet routing from csw5-pmtpa to csw1-pmtpa |
[production] |
19:16 |
<mark> |
Reconfigured pybal on lvs3 to BGP connect to csw1-sdtpa instead of csw5-pmtpa, and restarted it |
[production] |
17:53 |
<mark> |
Moved vlan 2 routing from csw5-pmtpa:ve2 to csw1-pmtpa:ve2 |
[production] |
16:27 |
<apergos> |
or at least srv254. |
[production] |
16:25 |
<apergos> |
nagios is whining about some services that are actually in operation (eg srv250) |
[production] |
16:07 |
<Ryan_Lane> |
or not, mgmt interface isn't letting me in |
[production] |
16:06 |
<Ryan_Lane> |
powercycling srv250 |
[production] |
13:56 |
<Tim> |
removed myself from SMS notification in nagios, since the configuration is so broken that it sends me a text every 10 minutes day or night |
[production] |
13:49 |
<RobH> |
nagios sms pages seem to be delayed from the actual service outage by a long, long delay |
[production] |
13:10 |
<Tim> |
did the above in both puppet and nagios directly since puppet seems to be still broken |
[production] |
13:07 |
<Tim> |
increased the number of retries of payments.wikimedia.org before sending an SMS from 1 to 20 |
[production] |
02:02 |
<notpeter> |
accidentally deleted sockpuppet:/var/lib/puppet/ssl am regnerating keys and will begin signing soon |
[production] |
2011-05-09
§
|
23:36 |
<notpeter> |
so annoying. doubling puppet freshness check timing to 4 hours. |
[production] |
22:01 |
<awjrichards> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'Disabling dashboard on enwiki pending fixes to data collection script' |
[production] |
21:20 |
<catrope> |
synchronizing Wikimedia installation... Revision: 87774: |
[production] |
21:20 |
<RoanKattouw> |
Running scap to deploy UploadWizard changes |
[production] |
20:54 |
<RoanKattouw> |
Running extensions/ArticleFeedback/populateAFStatistics.php on enwiki |
[production] |
20:49 |
<catrope> |
synchronized php-1.17/extensions/ArticleFeedback/populateAFStatistics.php 'r87770' |
[production] |
20:49 |
<catrope> |
synchronized php-1.17/extensions/ArticleFeedback/modules/ext.articleFeedback/ext.articleFeedback.js 'r87770' |
[production] |
20:30 |
<awjrichards> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enabling ArticleFeedback dashboard on testwiki' |
[production] |
20:28 |
<RoanKattouw> |
Running populateAFStatistics.php on testwiki |
[production] |
19:39 |
<catrope> |
synchronized php-1.17/resources/startup.js 'Dummy sync to push config change through in ResourceLoader' |
[production] |
19:39 |
<catrope> |
synchronized php-1.17/wmf-config/CommonSettings.php 'Re-enable AFT click tracking at 2%, and bump tracking version' |
[production] |
19:32 |
<catrope> |
synchronized php-1.17/resources/startup.js 'Force config changes to update in startup module' |
[production] |
19:21 |
<catrope> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'Raise AFT lottery odds to the target value of 2.7%' |
[production] |
19:03 |
<catrope> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'Raise AFT lottery odds to 2%' |
[production] |
18:41 |
<catrope> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'Raise AFT lottery odds to 1%' |
[production] |