9451-9500 of 10000 results (27ms)
2010-10-03 §
13:36 <mark> lvs3 service ips were unreachable from outside the subnet for some reason. Restarting PyBal (and therefore the BGP session to the router) fixed it [production]
13:22 <mark> Started pdns recursor on dobson again [production]
13:19 <mark> Once again stopped pnds-recursor on dobson [production]
13:18 <mark> Restarted Apache on the mobile servers one by one, to put the new resolv.conf in effect [production]
13:10 <mark> Swapped nameservers for the mobile servers, so they don't crash the mobile site while I reinstall the primary recursor [production]
13:02 <catrope> synchronized php-1.5/wmf-config/checkers.php 'Log no-UA requests for exempt IPs to /h/w/l/nouaexempt.log' [production]
12:45 <mark> Mobile site broke with missing primary DNS recursor, restarted it [production]
12:44 <mark> Stopped DNS recursor on dobson again [production]
12:20 <mark> Installed DNS recursors on all LVS servers to reduce SPOFfiness [production]
11:46 <domas> doing some DB maintenance [production]
11:44 <mark> Installed nscd on lvs4 and restarted PyBal [production]
11:30 <mark> Restarted pybal on lvs4 [production]
11:25 <catrope> synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix' [production]
11:21 <mark> Shutdown primary pmtpa DNS recursor (dobson) [production]
11:20 <catrope> synchronized php-1.5/skins/common/wikibits.js 'r74198' [production]
2010-10-02 §
17:01 <apergos> we were hoping that a reboot of dataset1 would clear up the ipv6 + lighty issue that it had after os upgrade; no dice. this needs further investiagtion. ipv6 left disabled in lighty conf for now. [production]
16:43 <RobH> dataset1 relocation and drive work done, system is back online [production]
16:20 <RobH> dataset1 moved, boots back online, still working on its degraded disk replacement [production]
15:55 <RobH> rebooted scs-a1-sdtpa since its web interface isnt accessible, not sure if its the device or the network [production]
15:06 <RobH> dataset1 downtime window starts now [production]
2010-10-01 §
20:03 <RoanKattouw> Batch-deleted ~1200 categories on cswiktionary, requested by Danny B [production]
15:54 <mark> Restarted apache on srv219-223, and killed stuck convert processes [production]
13:20 <mark> Removed unix accounts brion and fvassard (with puppet) [production]
2010-09-30 §
23:14 <Ryan_Lane> deleting /srv-old on prototype [production]
23:11 <Ryan_Lane> added a new 50G disk to prototype. rsync'd data from /srv to new mount. moved /srv to /srv-old. mounted new drive on /srv. rsync'd data from /srv-old to /srv. [production]
21:27 <tfinc> synchronized php-1.5/wmf-config/CommonSettings.php 'White listing new banners' [production]
17:13 <awjr> upgraded kernel on grosley.wikimedia.org to 2.6.24-28 [production]
00:57 <Ryan_Lane> added mail alias ryan -> rlane [production]
00:55 <Ryan_Lane> added mail alias carrie -> csmith on mchenry [production]
2010-09-29 §
22:59 <tfinc> synchronized php-1.5/wmf-config/InitialiseSettings.php 'Turning CN back on as default' [production]
22:55 <tfinc> synchronized php-1.5/extensions/CentralNotice/CentralNotice.php 'Picking up fix for db lookup' [production]
22:55 <Ryan_Lane> rebooting ersch and alsted [production]
22:54 <tfinc> synchronized php-1.5/wmf-config/CommonSettings.php 'Turning test wikipedia to non infrastructure mode' [production]
22:41 <tfinc> synchronized php-1.5/wmf-config/InitialiseSettings.php 'disabling cn' [production]
22:39 <tfinc> synchronized php-1.5/extensions/CentralNotice/CentralNotice.php 'Picking up fixes from 73993' [production]
22:38 <tfinc> synchronized php-1.5/extensions/CentralNotice/SpecialBannerController.php 'Picking up fixes from 73993' [production]
22:36 <tfinc> synchronized php-1.5/includes/Skin.php 'Picking up fixes from 73993' [production]
21:31 <tfinc> synchronized php-1.5/extensions/CentralNotice/SpecialBannerController.php 'Picking up fix for message translation' [production]
21:29 <tfinc> synchronized php-1.5/includes/MessageCache.php 'Picking up fix for message translation' [production]
21:28 <tfinc> synchronized php-1.5/includes/parser/CoreParserFunctions.php 'Picking up fix for message translation' [production]
21:27 <tfinc> synchronized php-1.5/includes/parser/ParserOptions.php 'Picking up fix for message translation' [production]
19:21 <mark> Permanently doubled sys.net.ipv4.tcp_max_tw_buckets to 360000, as the Varnish systems are constantly close to the limit (visible in /proc/net/sockstat) [production]
19:04 <mark> ...on all squid and varnish servers, in puppet sysctl.conf [production]
19:03 <mark> Increasing net.ipv4.tcp_max_syn_backlog = 262144 [production]
18:57 <mark> Experimentally doubled sys.net.ipv4.tcp_max_tw_buckets to 360000 on sq70 (manually, so not in sysctl.conf) [production]
16:45 <Ryan_Lane> make that the *new* tesla cluster subnet [production]
16:44 <Ryan_Lane> updating dhcpd.conf on brewster to include the tesla subnet (153.192) [production]
16:29 <mark> Setup IP helper-address to brewster for vlan 103 / ve 7 on csw5-pmtpa [production]
02:31 <Ryan_Lane> unmounted old data partition and replaced it with the newer, larger data partition and updated the fstab on tridge. [production]
00:51 <Ryan_Lane> destroyed resized data1 logical volume, and recreated the jfs filesystem on tridge. copying the data from /data to new partition; will remount data as new partition and update fstab [production]