101-150 of 1214 results (2ms)
2009-02-21 §
12:55 <river> fixed reverse dns entries for ms3/ms4, which had got swapped somehow [production]
11:55 <Tim> re-enabled ExtensionDistributor [production]
11:16 <Tim> removed syslog.0 and messages.0 on srv170 and srv176, they had critical disk free on / [production]
03:25 <Tim> started apache on the image scaling servers [production]
02:51 <brion> ran sync-common on srv199 while i'm at it [production]
02:48 <brion> zeroing out stupid giant syslog files on srv199 [production]
02:46 <brion> srv199 is out of disk space [production]
02:46 <brion> copying hacked-up copies of InitialiseSettings/CommonSettings back to /home so the changes aren't lost this time [production]
02:23 <mark> db20 back up, for reals [production]
02:19 <mark> Rebooting db20 with upgraded RAID controller firmware [production]
02:13 <domas> flashing BIOS helped [production]
02:13 <mark> db20 up! [production]
02:04 <brion> services on bart (secure, planet) are temporarily offline while server is poked at [production]
01:50 <brion> seeing pages, yay [production]
01:49 <brion> running apache2ctl start or apachectl start for various apaches [production]
01:47 <domas> I FOUND HOW TO REVIVE APACHES [production]
01:46 <brion> think i killed em, now trying to restart apache procs [production]
01:43 <brion> poking to see if we can restart apaches... [production]
01:42 <brion> syncing fixed InitialiseSettings/COmmonSettings to apaches [production]
01:14 <brion> and flyingparchment [production]
01:14 <brion> domas and mark are attempting to restart the NFS server, but aren't mentioning any details in the public channel or log [production]
00:52 <domas> http://p.defau.lt/?_M1iGbA0PCz2OOt2_KKPug [production]
00:52 <mark> db20 in trouble [production]
00:39 <mark> @brion you don't need to wake up [production]
00:36 <domas> disabled 2006 fundraising cronjob on amane :-) [production]
2009-02-20 §
23:31 <Rob> upgraded squid and kernel on sq34-sq36 [production]
23:12 <Rob> upgraded kernel and squid on sq31-sq33, redeployed and online [production]
23:08 <brion> updating CentralNotice for improved test script (plus i8n update) [production]
22:54 <Rob> upgraded kernel + squid on sq28-sq30 [production]
22:29 <Rob> completed upgrades to sq25-sq27 [production]
22:12 <Rob> upgrading kernel and squid versions on sq25-sq27 (if i crash the site, i apologize in advance) [production]
22:08 <Rob> upgraded kernel and squid on sq24 [production]
21:59 <river> added current patches to ms4, set zil_disable=1 and rebooted [production]
21:30 <brion> srv31 seems to be down, so no dump activity [production]
21:08 <brion> scapping to update FlaggedRevs to r47588 (fixing fatal err) [production]
21:01 <Rob> updated kernel and squid on sq23 [production]
20:58 <Rob> updated kernel and squid on sq22 [production]
20:36 <Rob> updated kernel and squid on sq20 and sq21 [production]
20:25 <domas> some apaches in crashloop like this: http://p.defau.lt/?s9YhHD_0qHroVhauBdQb_g [production]
20:09 <Rob> restarted apache on srv73 [production]
20:03 <Rob> upgraded kernel and squid on sq19 [production]
19:50 <Rob> upgraded kernel + squid on sq18 [production]
19:34 <Rob> upgraded kernel + squid on sq17 [production]
19:19 <brion> updating FlaggedRevs to r47574 [production]
18:17 <river> test [production]
18:17 <river> test [production]
18:16 <river> set zil_disable on ms1 to improve nfs write performance [production]
18:15 <mark> Raised max-conns to 50 [production]
18:03 <mark> Cut down max conns even more (25) for pmtpa upload backend squids [production]
17:40 <mark> Limited maximum connections to backend (ms1) to 50 per squid on upload squids, 1000 per squid on text [production]