|
2010-08-17
§
|
| 22:57 |
<mark> |
Shutdown ve1 on csw1 to force VRRP backup |
[production] |
| 22:53 |
<mark> |
Packet loss, authdns-scenario esams-down |
[production] |
| 22:48 |
<mark> |
authdns-scenario normal |
[production] |
| 22:43 |
<mark> |
Configured all VRRP instances on csw1-esams to have priority 1, to reliably stay in backup mode |
[production] |
| 22:11 |
<RobH> |
dns changed to route traffic to tampa |
[production] |
| 20:13 |
<RobH> |
set srv278 to false in lvs, taking it down for hardware testing per rt#24 |
[production] |
| 18:06 |
<rainman-sr> |
disabling interwiki search on all wikis, not only en.wp until we figure out what is going on |
[production] |
| 16:49 |
<rainman-sr> |
search11 is fully up with all features, and seems to work fine .. will keep an eye on it |
[production] |
| 16:43 |
<RobH> |
srv230 online |
[production] |
| 16:41 |
<rainman-sr> |
all of search up, still fiddling with search11 to see why it gave strange I/O spikes during the batch2 migration |
[production] |
| 16:37 |
<RobH> |
investigating srv230. |
[production] |
| 16:32 |
<RobH> |
srv230 back online with memory replacement, synced and back in cluster |
[production] |
| 16:31 |
<robh> |
synchronized php-1.5/wmf-config/lucene.php |
[production] |
| 16:29 |
<robh> |
synchronized php-1.5/wmf-config/lucene.php 'Returning all search values to normal, should restore full search functionality.' |
[production] |
| 16:22 |
<rainman-sr> |
bringing up search5,12, 13-20 |
[production] |
| 16:21 |
<RobH> |
shutting down srv230 to swap out bad memory |
[production] |
| 15:50 |
<RobH> |
search13-search20 relocated to b3-sdtpa. All servers are online, working to bring search back to full deployment. |
[production] |
| 14:36 |
<rainman-sr> |
search5,12 will also show offline because they run parts of the services that are temporarly disabled |
[production] |
| 14:30 |
<RobH> |
search13-search20 will show offline during their relocation, approx until 16:00 if all things go well |
[production] |
| 14:09 |
<robh> |
synchronized php-1.5/wmf-config/lucene.php 'Disabling wgEnableLucenePrefixSearch on projects while search servers are relocated' |
[production] |
| 14:04 |
<rainman-sr> |
starting relocation of second batch of search servers |
[production] |
| 13:05 |
<andrew> |
synchronized php-1.5/extensions/LiquidThreads_alpha/i18n/Lqt.namespaces.php |
[production] |
| 13:05 |
<andrew> |
synchronized php-1.5/extensions/LiquidThreads/i18n/Lqt.namespaces.php |
[production] |
| 12:19 |
<andrew> |
synchronizing Wikimedia installation... Revision: 71190 |
[production] |
| 12:19 |
<Andrew> |
running scap to update LiquidThreads localisation |
[production] |
| 11:01 |
<andrew> |
synchronized php-1.5/wmf-config/CommonSettings.php 'Add LiquidThreads to ptwikibooks' |
[production] |
| 10:34 |
<andrew> |
synchronized php-1.5/extensions/LiquidThreads_alpha/LiquidThreads.php |
[production] |
| 10:34 |
<andrew> |
synchronized php-1.5/extensions/LiquidThreads_alpha/i18n/Lqt.namespaces.php |
[production] |
| 10:33 |
<andrew> |
synchronized php-1.5/extensions/LiquidThreads_alpha/i18n/Lqt.i18n.php |
[production] |
| 10:33 |
<andrew> |
synchronized php-1.5/extensions/LiquidThreads/LiquidThreads.php |
[production] |
| 10:33 |
<andrew> |
synchronized php-1.5/extensions/LiquidThreads/i18n/Lqt.namespaces.php |
[production] |
| 10:33 |
<andrew> |
synchronized php-1.5/extensions/LiquidThreads/i18n/Lqt.i18n.php |
[production] |
| 05:56 |
<jeluf> |
synchronized php-1.5/wmf-config/CommonSettings.php '22109 - add fourth level subdomains of wikimedia to ' |
[production] |
| 05:42 |
<jeluf> |
synchronized php-1.5/wmf-config/InitialiseSettings.php '24808 - Localize Wikipedia name for newiki' |
[production] |
| 05:34 |
<jeluf> |
synchronized php-1.5/wmf-config/InitialiseSettings.php '24355 - Namespace changes - si.wikipedia' |
[production] |
| 05:31 |
<jeluf> |
synchronized php-1.5/wmf-config/InitialiseSettings.php '24255 - Change meta namespace names on siwikibooks' |
[production] |
| 05:24 |
<jeluf> |
synchronized php-1.5/wmf-config/InitialiseSettings.php '24579 - Reconfigure Turkish Wikibooks' |
[production] |
| 05:07 |
<jeluf> |
synchronized php-1.5/wmf-config/InitialiseSettings.php '24822 - XML-import for de-wiktionary' |
[production] |
| 05:05 |
<jeluf> |
synchronized php-1.5/wmf-config/InitialiseSettings.php '24823 - alias for de-wiktionary' |
[production] |
| 04:09 |
<Tim> |
wikimedia-task-appserver on srv224 was half-configured because parentless convert processes stuck in a deadlock were holding the *:80 listen filehandle open, preventing apache from starting. Killed the convert processes and reinstalled. |
[production] |
| 04:05 |
<Tim> |
deployment of the new wikimedia-task-appserver led to nagios alerts about half-configured status on 16 servers due to a warning in rsync sync-common about file permissions. Fixed by deleting Mark's vim swapfile and reinstalling. |
[production] |
| 03:36 |
<tstarling> |
synchronized php-1.5/wmf-config/CommonSettings.php 'disabling PagedTiffHandler, no identify present on non-scalers so upload fails' |
[production] |
| 03:26 |
<tstarling> |
synchronized php-1.5/wmf-config/CommonSettings.php 'enabling PagedTiffHandler' |
[production] |
| 03:23 |
<tstarling> |
synchronizing Wikimedia installation... Revision: 71190 |
[production] |
| 03:22 |
<Tim> |
doing another scap to update ExtensionMessages.php |
[production] |