|
2010-08-18
§
|
| 20:03 |
<RobH> |
sq57 drive replaced, but raid didnt work (seems like grub wasnt copied to both drives) leaving offline for now, will investigate later |
[production] |
| 19:42 |
<RobH> |
sq57 set to false in lvs, replacing bad disk. |
[production] |
| 18:33 |
<RobH> |
kicking around db16, trying to fix it |
[production] |
| 10:37 |
<mark> |
Restored VRRP priorities to original state |
[production] |
| 10:33 |
<mark> |
authdns-scenario normal |
[production] |
| 10:30 |
<mark> |
Enabled ve1 on csw1-esams |
[production] |
| 02:31 |
<Tim> |
also edited /etc/gai.conf on fenari to prefer IPv6, to fix ExtensionDistributor |
[production] |
| 02:28 |
<Tim> |
edited /etc/gai.conf on kaulen to avoid broken IPv6 connection to mayflower, so CR will start working again |
[production] |
|
2010-08-17
§
|
| 22:57 |
<mark> |
Shutdown ve1 on csw1 to force VRRP backup |
[production] |
| 22:53 |
<mark> |
Packet loss, authdns-scenario esams-down |
[production] |
| 22:48 |
<mark> |
authdns-scenario normal |
[production] |
| 22:43 |
<mark> |
Configured all VRRP instances on csw1-esams to have priority 1, to reliably stay in backup mode |
[production] |
| 22:11 |
<RobH> |
dns changed to route traffic to tampa |
[production] |
| 20:13 |
<RobH> |
set srv278 to false in lvs, taking it down for hardware testing per rt#24 |
[production] |
| 18:06 |
<rainman-sr> |
disabling interwiki search on all wikis, not only en.wp until we figure out what is going on |
[production] |
| 16:49 |
<rainman-sr> |
search11 is fully up with all features, and seems to work fine .. will keep an eye on it |
[production] |
| 16:43 |
<RobH> |
srv230 online |
[production] |
| 16:41 |
<rainman-sr> |
all of search up, still fiddling with search11 to see why it gave strange I/O spikes during the batch2 migration |
[production] |
| 16:37 |
<RobH> |
investigating srv230. |
[production] |
| 16:32 |
<RobH> |
srv230 back online with memory replacement, synced and back in cluster |
[production] |
| 16:31 |
<robh> |
synchronized php-1.5/wmf-config/lucene.php |
[production] |
| 16:29 |
<robh> |
synchronized php-1.5/wmf-config/lucene.php 'Returning all search values to normal, should restore full search functionality.' |
[production] |
| 16:22 |
<rainman-sr> |
bringing up search5,12, 13-20 |
[production] |
| 16:21 |
<RobH> |
shutting down srv230 to swap out bad memory |
[production] |
| 15:50 |
<RobH> |
search13-search20 relocated to b3-sdtpa. All servers are online, working to bring search back to full deployment. |
[production] |
| 14:36 |
<rainman-sr> |
search5,12 will also show offline because they run parts of the services that are temporarly disabled |
[production] |
| 14:30 |
<RobH> |
search13-search20 will show offline during their relocation, approx until 16:00 if all things go well |
[production] |
| 14:09 |
<robh> |
synchronized php-1.5/wmf-config/lucene.php 'Disabling wgEnableLucenePrefixSearch on projects while search servers are relocated' |
[production] |
| 14:04 |
<rainman-sr> |
starting relocation of second batch of search servers |
[production] |
| 13:05 |
<andrew> |
synchronized php-1.5/extensions/LiquidThreads_alpha/i18n/Lqt.namespaces.php |
[production] |
| 13:05 |
<andrew> |
synchronized php-1.5/extensions/LiquidThreads/i18n/Lqt.namespaces.php |
[production] |
| 12:19 |
<andrew> |
synchronizing Wikimedia installation... Revision: 71190 |
[production] |
| 12:19 |
<Andrew> |
running scap to update LiquidThreads localisation |
[production] |
| 11:01 |
<andrew> |
synchronized php-1.5/wmf-config/CommonSettings.php 'Add LiquidThreads to ptwikibooks' |
[production] |
| 10:34 |
<andrew> |
synchronized php-1.5/extensions/LiquidThreads_alpha/LiquidThreads.php |
[production] |
| 10:34 |
<andrew> |
synchronized php-1.5/extensions/LiquidThreads_alpha/i18n/Lqt.namespaces.php |
[production] |
| 10:33 |
<andrew> |
synchronized php-1.5/extensions/LiquidThreads_alpha/i18n/Lqt.i18n.php |
[production] |
| 10:33 |
<andrew> |
synchronized php-1.5/extensions/LiquidThreads/LiquidThreads.php |
[production] |
| 10:33 |
<andrew> |
synchronized php-1.5/extensions/LiquidThreads/i18n/Lqt.namespaces.php |
[production] |
| 10:33 |
<andrew> |
synchronized php-1.5/extensions/LiquidThreads/i18n/Lqt.i18n.php |
[production] |
| 05:56 |
<jeluf> |
synchronized php-1.5/wmf-config/CommonSettings.php '22109 - add fourth level subdomains of wikimedia to ' |
[production] |
| 05:42 |
<jeluf> |
synchronized php-1.5/wmf-config/InitialiseSettings.php '24808 - Localize Wikipedia name for newiki' |
[production] |