1-50 of 8120 results (6ms)
2010-08-18 §
20:03 <RobH> sq57 drive replaced, but raid didnt work (seems like grub wasnt copied to both drives) leaving offline for now, will investigate later [production]
19:42 <RobH> sq57 set to false in lvs, replacing bad disk. [production]
18:33 <RobH> kicking around db16, trying to fix it [production]
10:37 <mark> Restored VRRP priorities to original state [production]
10:33 <mark> authdns-scenario normal [production]
10:30 <mark> Enabled ve1 on csw1-esams [production]
02:31 <Tim> also edited /etc/gai.conf on fenari to prefer IPv6, to fix ExtensionDistributor [production]
02:28 <Tim> edited /etc/gai.conf on kaulen to avoid broken IPv6 connection to mayflower, so CR will start working again [production]
2010-08-17 §
22:57 <mark> Shutdown ve1 on csw1 to force VRRP backup [production]
22:53 <mark> Packet loss, authdns-scenario esams-down [production]
22:48 <mark> authdns-scenario normal [production]
22:43 <mark> Configured all VRRP instances on csw1-esams to have priority 1, to reliably stay in backup mode [production]
22:11 <RobH> dns changed to route traffic to tampa [production]
20:13 <RobH> set srv278 to false in lvs, taking it down for hardware testing per rt#24 [production]
18:06 <rainman-sr> disabling interwiki search on all wikis, not only en.wp until we figure out what is going on [production]
16:49 <rainman-sr> search11 is fully up with all features, and seems to work fine .. will keep an eye on it [production]
16:43 <RobH> srv230 online [production]
16:41 <rainman-sr> all of search up, still fiddling with search11 to see why it gave strange I/O spikes during the batch2 migration [production]
16:37 <RobH> investigating srv230. [production]
16:32 <RobH> srv230 back online with memory replacement, synced and back in cluster [production]
16:31 <robh> synchronized php-1.5/wmf-config/lucene.php [production]
16:29 <robh> synchronized php-1.5/wmf-config/lucene.php 'Returning all search values to normal, should restore full search functionality.' [production]
16:22 <rainman-sr> bringing up search5,12, 13-20 [production]
16:21 <RobH> shutting down srv230 to swap out bad memory [production]
15:50 <RobH> search13-search20 relocated to b3-sdtpa. All servers are online, working to bring search back to full deployment. [production]
14:36 <rainman-sr> search5,12 will also show offline because they run parts of the services that are temporarly disabled [production]
14:30 <RobH> search13-search20 will show offline during their relocation, approx until 16:00 if all things go well [production]
14:09 <robh> synchronized php-1.5/wmf-config/lucene.php 'Disabling wgEnableLucenePrefixSearch on projects while search servers are relocated' [production]
14:04 <rainman-sr> starting relocation of second batch of search servers [production]
13:05 <andrew> synchronized php-1.5/extensions/LiquidThreads_alpha/i18n/Lqt.namespaces.php [production]
13:05 <andrew> synchronized php-1.5/extensions/LiquidThreads/i18n/Lqt.namespaces.php [production]
12:19 <andrew> synchronizing Wikimedia installation... Revision: 71190 [production]
12:19 <Andrew> running scap to update LiquidThreads localisation [production]
11:01 <andrew> synchronized php-1.5/wmf-config/CommonSettings.php 'Add LiquidThreads to ptwikibooks' [production]
10:34 <andrew> synchronized php-1.5/extensions/LiquidThreads_alpha/LiquidThreads.php [production]
10:34 <andrew> synchronized php-1.5/extensions/LiquidThreads_alpha/i18n/Lqt.namespaces.php [production]
10:33 <andrew> synchronized php-1.5/extensions/LiquidThreads_alpha/i18n/Lqt.i18n.php [production]
10:33 <andrew> synchronized php-1.5/extensions/LiquidThreads/LiquidThreads.php [production]
10:33 <andrew> synchronized php-1.5/extensions/LiquidThreads/i18n/Lqt.namespaces.php [production]
10:33 <andrew> synchronized php-1.5/extensions/LiquidThreads/i18n/Lqt.i18n.php [production]
05:56 <jeluf> synchronized php-1.5/wmf-config/CommonSettings.php '22109 - add fourth level subdomains of wikimedia to ' [production]
05:42 <jeluf> synchronized php-1.5/wmf-config/InitialiseSettings.php '24808 - Localize Wikipedia name for newiki' [production]
05:34 <jeluf> synchronized php-1.5/wmf-config/InitialiseSettings.php '24355 - Namespace changes - si.wikipedia' [production]
05:31 <jeluf> synchronized php-1.5/wmf-config/InitialiseSettings.php '24255 - Change meta namespace names on siwikibooks' [production]
05:24 <jeluf> synchronized php-1.5/wmf-config/InitialiseSettings.php '24579 - Reconfigure Turkish Wikibooks' [production]
05:07 <jeluf> synchronized php-1.5/wmf-config/InitialiseSettings.php '24822 - XML-import for de-wiktionary' [production]
05:05 <jeluf> synchronized php-1.5/wmf-config/InitialiseSettings.php '24823 - alias for de-wiktionary' [production]
04:09 <Tim> wikimedia-task-appserver on srv224 was half-configured because parentless convert processes stuck in a deadlock were holding the *:80 listen filehandle open, preventing apache from starting. Killed the convert processes and reinstalled. [production]
04:05 <Tim> deployment of the new wikimedia-task-appserver led to nagios alerts about half-configured status on 16 servers due to a warning in rsync sync-common about file permissions. Fixed by deleting Mark's vim swapfile and reinstalling. [production]
03:36 <tstarling> synchronized php-1.5/wmf-config/CommonSettings.php 'disabling PagedTiffHandler, no identify present on non-scalers so upload fails' [production]