2101-2150 of 10000 results (17ms)
2010-10-03 §
17:32 <mark> Stopped PyBal on amslvs1 [production]
17:30 <mark> Stopped IPVS backup sync daemon on amslvs4 [production]
17:27 <mark> Rebooting amslvs2 [production]
17:19 <mark> Stopped PyBal on amslvs2 [production]
17:14 <mark> Drive failed in thistle [production]
17:09 <mark> Started temporary LVS state syncing between amslvs1->amslvs3 and amslvs2->amslvs4, preparing for reboot of amslvs1-2 [production]
16:51 <mark> Rebooted (backup) LVS servers amslvs3 and amslvs4 [production]
16:36 <midom> synchronized php-1.5/wmf-config/db.php 'adding db28 to s5' [production]
16:20 <mark> Selectively updated pybal/bgp.py on lvs2-4 and restarted PyBal [production]
16:13 <mark> Upgraded PyBal to r0.1+74215 on amslvs1-4 [production]
14:27 <midom> synchronized php-1.5/wmf-config/db.php [production]
14:12 <domas> db22 was first wikimedia slave initialized using xtrabackup, hehe [production]
14:08 <midom> synchronized php-1.5/wmf-config/db.php 'adding db22 as commons 5.1 slave' [production]
14:06 <mark> Restored nameservers order for mobile serverswq [production]
13:36 <mark> lvs3 service ips were unreachable from outside the subnet for some reason. Restarting PyBal (and therefore the BGP session to the router) fixed it [production]
13:22 <mark> Started pdns recursor on dobson again [production]
13:19 <mark> Once again stopped pnds-recursor on dobson [production]
13:18 <mark> Restarted Apache on the mobile servers one by one, to put the new resolv.conf in effect [production]
13:10 <mark> Swapped nameservers for the mobile servers, so they don't crash the mobile site while I reinstall the primary recursor [production]
13:02 <catrope> synchronized php-1.5/wmf-config/checkers.php 'Log no-UA requests for exempt IPs to /h/w/l/nouaexempt.log' [production]
12:45 <mark> Mobile site broke with missing primary DNS recursor, restarted it [production]
12:44 <mark> Stopped DNS recursor on dobson again [production]
12:20 <mark> Installed DNS recursors on all LVS servers to reduce SPOFfiness [production]
11:46 <domas> doing some DB maintenance [production]
11:44 <mark> Installed nscd on lvs4 and restarted PyBal [production]
11:30 <mark> Restarted pybal on lvs4 [production]
11:25 <catrope> synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix' [production]
11:21 <mark> Shutdown primary pmtpa DNS recursor (dobson) [production]
11:20 <catrope> synchronized php-1.5/skins/common/wikibits.js 'r74198' [production]
2010-10-02 §
17:01 <apergos> we were hoping that a reboot of dataset1 would clear up the ipv6 + lighty issue that it had after os upgrade; no dice. this needs further investiagtion. ipv6 left disabled in lighty conf for now. [production]
16:43 <RobH> dataset1 relocation and drive work done, system is back online [production]
16:20 <RobH> dataset1 moved, boots back online, still working on its degraded disk replacement [production]
15:55 <RobH> rebooted scs-a1-sdtpa since its web interface isnt accessible, not sure if its the device or the network [production]
15:06 <RobH> dataset1 downtime window starts now [production]
2010-10-01 §
20:03 <RoanKattouw> Batch-deleted ~1200 categories on cswiktionary, requested by Danny B [production]
15:54 <mark> Restarted apache on srv219-223, and killed stuck convert processes [production]
13:20 <mark> Removed unix accounts brion and fvassard (with puppet) [production]
2010-09-30 §
23:14 <Ryan_Lane> deleting /srv-old on prototype [production]
23:11 <Ryan_Lane> added a new 50G disk to prototype. rsync'd data from /srv to new mount. moved /srv to /srv-old. mounted new drive on /srv. rsync'd data from /srv-old to /srv. [production]
21:27 <tfinc> synchronized php-1.5/wmf-config/CommonSettings.php 'White listing new banners' [production]
17:13 <awjr> upgraded kernel on grosley.wikimedia.org to 2.6.24-28 [production]
00:57 <Ryan_Lane> added mail alias ryan -> rlane [production]
00:55 <Ryan_Lane> added mail alias carrie -> csmith on mchenry [production]
2010-09-29 §
22:59 <tfinc> synchronized php-1.5/wmf-config/InitialiseSettings.php 'Turning CN back on as default' [production]
22:55 <tfinc> synchronized php-1.5/extensions/CentralNotice/CentralNotice.php 'Picking up fix for db lookup' [production]
22:55 <Ryan_Lane> rebooting ersch and alsted [production]
22:54 <tfinc> synchronized php-1.5/wmf-config/CommonSettings.php 'Turning test wikipedia to non infrastructure mode' [production]
22:41 <tfinc> synchronized php-1.5/wmf-config/InitialiseSettings.php 'disabling cn' [production]
22:39 <tfinc> synchronized php-1.5/extensions/CentralNotice/CentralNotice.php 'Picking up fixes from 73993' [production]
22:38 <tfinc> synchronized php-1.5/extensions/CentralNotice/SpecialBannerController.php 'Picking up fixes from 73993' [production]