|
2010-02-16
§
|
| 22:03 |
<logmsgbot_> |
mark synchronized php-1.5/wmf-config/checkers.php 'Exceptions' |
[production] |
| 20:30 |
<Rob> |
srv127 is online, but not in LVS. Its cert was accepted on sockpuppet, but puppetd --test results in a cert failure on retrieval from sockpuppet. |
[production] |
| 20:17 |
<Rob> |
rebooting srv127 |
[production] |
| 20:15 |
<mark> |
Increased membufs to 40 per COSS dir on the pmtpa upload backend squids |
[production] |
| 19:51 |
<mark> |
Increased membufs per COSS dir from 10 to 20 on the new pmtpa squids |
[production] |
| 18:10 |
<apergos> |
but documentation can save people precious time when things are on fire |
[production] |
| 18:02 |
<mark> |
Documentation is not a substitution for thinking |
[production] |
| 16:44 |
<mark> |
Fixed puppet on most servers |
[production] |
| 16:23 |
<domas> |
anyone knows why mysqldump on snapshot3 is locking tables? maybe --single-transaction could work better?!!!? |
[production] |
| 16:22 |
<RoanKattouw> |
Strike my last, I hear it'll fix itself in a day |
[production] |
| 16:21 |
<RoanKattouw> |
Oops, meant to type srv187-189 |
[production] |
| 16:21 |
<RoanKattouw> |
Ganglia not picking up data for srv1987-189, 193, 214-218, 250-253, 257 even though the boxes are up and gmond is running; been like this for 3 days |
[production] |
| 15:21 |
<mark> |
Removing all puppet certs/private keys on all machines |
[production] |
| 14:55 |
<mark> |
Puppetmaster is screwed up by a wrong command that deleted all files under /var/lib/puppet |
[production] |
| 14:25 |
<logmsgbot_> |
midom synchronized php-1.5/includes/diff/DifferenceInterface.php 'instrumenting costs by revision pairs' |
[production] |
| 13:46 |
<logmsgbot_> |
root synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 22444 and 22330 enabling collection on dawiki and ruwiki' |
[production] |
| 13:22 |
<domas> |
oh wait, it does work, something else.... |
[production] |
| 13:16 |
<domas> |
apparently my squid rule for Apple browsers didn't really work, hence the continuing spikes |
[production] |
| 03:16 |
<domas> |
blocked MacOSX Atom syndication on the edge, this will save terabytes of diskspace on unsuspecting Safari user computers :-) |
[production] |
| 02:13 |
<logmsgbot_> |
tstarling synchronized php-1.5/wmf-config/checkers.php |
[production] |
|
2010-02-15
§
|
| 23:42 |
<logmsgbot_> |
midom synchronized php-1.5/includes/StubObject.php 'uninstrumenting' |
[production] |
| 23:09 |
<logmsgbot_> |
midom synchronized php-1.5/wmf-config/checkers.php |
[production] |
| 23:09 |
<domas> |
banned all UA-less requests |
[production] |
| 22:53 |
<logmsgbot_> |
midom synchronized php-1.5/wmf-config/checkers.php 'un-disabling UA check on API' |
[production] |
| 22:14 |
<RoanKattouw> |
Resynced srv114, was returning empty responses |
[production] |
| 22:10 |
<logmsgbot_> |
mark synchronized php-1.5/wmf-config/CommonSettings.php 'Test with 220px default thumb size on enwiki (bug #21117)' |
[production] |
| 22:10 |
<logmsgbot_> |
mark synchronized php-1.5/wmf-config/InitialiseSettings.php 'Test with 220px default thumb size on enwiki (bug #21117)' |
[production] |
| 22:02 |
<logmsgbot_> |
catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix' |
[production] |
| 21:58 |
<logmsgbot_> |
catrope synchronized php-1.5/extensions/UsabilityInitiative/Vector/Vector.combined.min.js 'r62553' |
[production] |
| 21:39 |
<logmsgbot_> |
catrope synchronized php-1.5/includes/api/ApiMain.php 'r62550' |
[production] |
| 21:30 |
<logmsgbot_> |
mark synchronized php-1.5/wmf-config/InitialiseSettings.php 'raised wgSearchSuggestCacheExpiry from 1200 to 86400' |
[production] |
| 21:27 |
<logmsgbot_> |
catrope synchronized php-1.5/includes/api/ApiOpenSearch.php 'r62548' |
[production] |
| 21:27 |
<logmsgbot_> |
catrope synchronized php-1.5/includes/api/ApiMain.php 'r62548' |
[production] |
| 21:27 |
<logmsgbot_> |
catrope synchronized php-1.5/includes/DefaultSettings.php 'r62548' |
[production] |
| 20:54 |
<logmsgbot_> |
catrope synchronized php-1.5/includes/api/ApiBase.php 'r62546' |
[production] |
| 20:54 |
<logmsgbot_> |
catrope synchronized php-1.5/api.php 'r62546' |
[production] |
| 20:53 |
<domas> |
Roan and me are doing some perf engineering on opensearch & related stuff. \\o/ |
[production] |
| 20:40 |
<logmsgbot_> |
midom synchronized php-1.5/includes/StubObject.php 'readding stub trace' |
[production] |
| 20:34 |
<logmsgbot_> |
midom synchronized php-1.5/wmf-config/checkers.php 'perf comment action' |
[production] |
| 20:30 |
<logmsgbot_> |
midom synchronized php-1.5/includes/StubObject.php 'removing trace hooks' |
[production] |
| 20:28 |
<logmsgbot_> |
midom synchronized php-1.5/includes/StubObject.php |
[production] |
| 20:24 |
<logmsgbot_> |
catrope synchronized php-1.5/includes/api/ApiOpenSearch.php 'r62543' |
[production] |
| 19:32 |
<mark> |
Allowing API requests to be sent to the regular app servers temporarily |
[production] |
| 19:27 |
<mark> |
Restarted sq31 and sq31 API backend squids with slightly reduced memory settings |
[production] |
| 19:18 |
<logmsgbot_> |
mark synchronized php-1.5/wmf-config/InitialiseSettings.php |
[production] |
| 17:51 |
<Rob> |
set srv127 to false on lvs3, will re-enabled once in DC to reboot it manually |
[production] |
| 17:49 |
<logmsgbot_> |
root synchronized php-1.5/wmf-config/mc.php 'srv127 down taking out of mc rotation' |
[production] |
| 16:02 |
<RoanKattouw> |
Resynced srv134 to fix HTTP 500 |
[production] |
| 15:13 |
<RoanKattouw> |
Resynced srv232, was serving HTTP 500s |
[production] |
| 02:02 |
<mark> |
Fixed stale PHP file removal apache cron job in Puppet |
[production] |