|
2009-10-30
§
|
| 23:00 |
<Fred> |
depooled srv194 while testing / fixing |
[production] |
| 22:08 |
<Fred> |
actually stopped apache on srv194 since the restart did not help. Investigating.... |
[production] |
| 22:07 |
<Fred> |
restarted apache on srv194 since it was throwing a lot of ""is not a valid magic thingie for xxxx" |
[production] |
| 20:05 |
<Fred> |
scaped as root to resolve cache problems on some boxes. |
[production] |
| 20:02 |
<fvassard> |
ran sync-common-all |
[production] |
| 19:50 |
<RoanKattouw> |
Scapping in the hopes of resolving the MagicWordsArray exception; looks like it could be a problem with localisation cache |
[production] |
| 19:37 |
<catrope> |
synchronized php-1.5/includes/MagicWord.php 'Live hack for debugging: make MagicWordArray::parseMatch parameter not found error more verbose' |
[production] |
| 18:36 |
<catrope> |
synchronized php-1.5/extensions/LocalisationUpdate/LocalisationUpdate.class.php 'Deploy r58359 (LocalisationUpdate fix for languages with a '-' in their name)' |
[production] |
| 18:33 |
<catrope> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'Per bug 19865, enable the enhanced toolbar by default on enwikinews and disable the OptIn extension' |
[production] |
| 16:39 |
<RoanKattouw> |
sync-common-all threw lots of errors like srv175: rsync: failed to set times on "/usr/local/apache/common-local/wmf-deployment/includes/.svn": Operation not permitted (1) |
[production] |
| 16:38 |
<catrope> |
ran sync-common-all |
[production] |
| 16:36 |
<RoanKattouw> |
Scapping so srv194 will receive updates |
[production] |
| 15:34 |
<apergos> |
cleared out some files in /tmp and ran apt-get clean on srv194 to get back about 1gb space on / |
[production] |
| 14:08 |
<RoanKattouw> |
@wikimediatech feed on identi.ca has been broken for the past 3 days |
[production] |
| 14:01 |
<RoanKattouw> |
srv194 has full disk, causing syncs to fail; throws PHP exception "MagicWordArray::parseMatch: parameter not found" |
[production] |
| 11:47 |
<rainman-sr> |
searchidx1 ssh dead, shows 100% I/O, disk dead? |
[production] |
|
2009-10-29
§
|
| 22:49 |
<catrope> |
synchronized php-1.5/includes/upload/UploadFromUrl.php 'Re-sync for srv226' |
[production] |
| 22:48 |
<RoanKattouw> |
Scapping again for previous deployment because srv226 wouldn't let me log in at that time |
[production] |
| 20:27 |
<catrope> |
synchronized php-1.5/includes/upload/UploadFromUrl.php 'Deploy r58339 (upload API fix)' |
[production] |
| 18:51 |
<mark> |
Cleaned up lighttpd.conf on khaldun |
[production] |
| 17:28 |
<mark> |
OS-installed brewster.wikimedia.org |
[production] |
| 07:53 |
<tfinc> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'removing tidy from wmf wiki since its really in CommonSettings.php' |
[production] |
| 07:51 |
<tfinc> |
synchronized php-1.5/wmf-config/CommonSettings.php 'testing disabling tidy on wmfwiki .. now one level higher ' |
[production] |
| 07:44 |
<tfinc> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'testing disabling tidy on wmf' |
[production] |
| 06:15 |
<domas> |
deployed ipc semaphore cleanup cron via puppet |
[production] |
| 05:51 |
<tfinc> |
synchronized php-1.5/wmf-config/CommonSettings.php 'adding new donate skin for wmf wiki' |
[production] |
| 05:44 |
<tfinc> |
synchronized php-1.5/extensions/skins/Donate/Donate.class.php |
[production] |
|
2009-10-28
§
|
| 20:58 |
<Tim> |
deploying r58282, SecurePoll update |
[production] |
| 18:17 |
<tfinc> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'adding vector as defaul skin for outreachwiki' |
[production] |
| 18:00 |
<mark> |
Found an upload squid in the text squid configuration list, causing these 404s - removed |
[production] |
| 17:56 |
<mark> |
Removed stripping of caching headers from frontend squids - needed for debugging |
[production] |
| 17:50 |
<mark> |
synchronized live-1.5/404.php |
[production] |
| 17:46 |
<mark> |
synchronized docroot/wikipedia.org/w/404.php |
[production] |
| 17:28 |
<Fred> |
(re)started apache on srv219,234 |
[production] |
| 16:44 |
<rainman-sr> |
revert r58235 on search1/4 because it seems to confuse pybal and make is depool then randomly |
[production] |
| 16:33 |
<mark> |
Increased memory cache of text frontend squids from 10 to 50 MB |
[production] |
| 02:26 |
<Tim> |
deploying r58239, LiquidThreads revert |
[production] |
| 01:44 |
<Tim> |
trying to use debugging symbols on srv189, shutting down apache there temporarily |
[production] |
|
2009-10-27
§
|
| 21:14 |
<tstarling> |
synchronized wmf-deployment/cache/interwiki.cdb |
[production] |
| 21:13 |
<Tim> |
fixed outreach.wikimedia.org |
[production] |
| 21:12 |
<tstarling> |
ran sync-common-all |
[production] |
| 21:10 |
<hcatlin> |
pushed fix to native-iphone-app html... cleared memcached and.... a *momentary meltdown* occured |
[production] |
| 21:08 |
<RoanKattouw> |
Last scap deployed r58216 |
[production] |
| 21:06 |
<tstarling> |
ran sync-common-all |
[production] |
| 20:59 |
<RoanKattouw> |
Running LocalisationUpdate/update.php on test |
[production] |
| 20:58 |
<RoanKattouw> |
running svn up on test |
[production] |
| 20:54 |
<Tim> |
deployed r58215 |
[production] |
| 20:46 |
<catrope> |
synchronized php-1.5/wmf-config/CommonSettings.php 'Fix config for strategywiki and usabilitywiki so the enhanced toolbar is enabled by default again' |
[production] |
| 20:18 |
<midom> |
synchronized php-1.5/includes/memcached-client.php 'bumping up timeout to 0.1s up from 0.05s - as we do have some megabyte sized objects....' |
[production] |
| 19:32 |
<mark> |
Ran "deluser catrope" across the cluster to prompt puppet to recreate |
[production] |