651-700 of 983 results (5ms)
2008-12-05 §
22:26 <RobH> finished setup on srv146, back in apache pool [production]
21:32 <RobH> setting up packages on srv146 [production]
21:32 <RobH> reinstalling sq50 [production]
21:27 <brion> pointing SiteMatrix at local copy, not NFS master, of langlist file [production]
19:19 <RobH> added sq48, and sq49 back into pool. sq50 pending reinstallation. [production]
18:58 <mark> depooled broken squids sq1 and sq3 [production]
18:26 <RobH> depooled sq48-sq50 for relocation [production]
18:17 <RobH> added sq44-sq47 back into pybal, relocation complete. [production]
17:45 <brion> sync-common-all to add w/test-headers.php [production]
17:28 <RobH> shutting down sq44-sq47 for relocation. [production]
17:27 <RobH> sq41 - sq43 back online. [production]
17:17 <RobH> sq40 oddness, but its back up now [production]
16:44 <RobH> accidentally pulled power for sq38, opps! [production]
15:36 <RobH> removed sq41 - sq43 from pybal to relocate from pmtpa to sdtpa [production]
15:34 <domas> srv178 running usr.sbin.apache2 aa profile in complain mode [production]
15:34 <RobH> removed sq40 from pybal to relocate from pmtpa to sdtpa [production]
2008-12-04 §
22:50 <domas> job runners are no longer blue on ganglia CPU graphs :((((((( [production]
22:45 <domas> fc4 maintenance, reniced job runners to 20 (10 behind apaches), installed apc3.0.19 (APC3.0.13 seams to have hit severe lock contention/busylooping at overloads) [production]
22:04 <RobH> re-enabled sq38 in pybal. all is well [production]
22:02 <RobH> fired sq37-sq39 back up [production]
21:58 <RobH> shutdown sq37-sq39, cuz I need to balance the power distribution a bit better. [production]
21:40 <RobH> sq38 is trying to break my spirit, so i reinstalled it to show it who is boss (me!) [production]
21:02 <RobH> setup asw-a4-sdtpa and asw-a5-sdtpa on scs-a1-sdtpa [production]
20:52 <mark> Increased TCP buffers on srv88 (a Fedora), matching the Ubuntus - Fedora Apaches appear to get stuck/deadlocked on writes to Squids [production]
19:39 <RobH> pulled sq38 back out, as it is giving me issues. need to fix the msw-a3-sdtpa before i can fix sq38. [production]
19:35 <RobH> added sq38, sq39 back into pybal [production]
19:25 <RobH> added sq36, sq37 back into pybal [production]
18:14 <RobH> I need to stop forgetting about lunch and stop working through it, oh well. [production]
18:13 <RobH> depooled sq36-sq39 for move from pmtpa to sdtpa. [production]
18:12 <RobH> some tinkering with lvs4 and idleconnection timer was fixed by mark. [production]
17:46 <RobH> racked sq21-sq35 in sdtpa-a3. added back to pybal. [production]
16:31 <RobH> depooled sq31-sq35 from lvs4 to move from pmtpa to sdtpa [production]
15:15 <RobH> reinstalled storage1 to ubuntu 8.04, left data partition intact and untouched. [production]
2008-12-03 §
23:46 <JeLuF> performing importImage.php imports to commons for Duesentrieb [production]
19:13 <RobH> tested i/o on db17, issue where it pauses disk access is gone. [production]
19:02 <mark> Shutdown TeliaSonera (AS1299) BGP session, the link is flaky resuling in unidirectional traffic only for most of the day [production]
19:02 <RobH> replaced hardware in db17, reinstalled. [production]
18:58 <mark> Prepared search10, search11 and search12 as search servers [production]
17:26 <brion> investigating ploticus config breakage [[bugzilla:16085]] [production]
17:18 <brion> ploticus seems to be missing from most new apaches [production]
17:12 <RobH_DC> search10, search11, search12 racked and installed. [production]
14:29 <RobH_DC> srv136 was unresponsive, rebooted, synced, back in rotation. [production]
2008-12-02 §
23:33 <brion> scapping to update ContributionReporting ext [production]
23:11 <Tim> db7 wasn't deleting its relay logs for some reason, since August 21. Disk critical. Did a reset slave. [production]
20:03 <brion> rebuilt public_reporting with fixed encoding [production]
19:54 <brion> fudged charsets in triggers for donation db update, let's see if that helps [production]
12:11 <Tim> started squid (backend instance) on sq40, stopped for 13 days for no apparent reason [production]
12:08 <Tim> restarted apache on srv161, srv122, srv137, attempted on srv123 but it is waiting for dead NFS mount [production]
11:44 <Tim> took srv183 out of memcached rotation [production]
10:50 <Tim> purged binlogs on ixia and db1 (both critical) [production]