|
2009-02-21
§
|
| 11:16 |
<Tim> |
removed syslog.0 and messages.0 on srv170 and srv176, they had critical disk free on / |
[production] |
| 03:25 |
<Tim> |
started apache on the image scaling servers |
[production] |
| 02:51 |
<brion> |
ran sync-common on srv199 while i'm at it |
[production] |
| 02:48 |
<brion> |
zeroing out stupid giant syslog files on srv199 |
[production] |
| 02:46 |
<brion> |
srv199 is out of disk space |
[production] |
| 02:46 |
<brion> |
copying hacked-up copies of InitialiseSettings/CommonSettings back to /home so the changes aren't lost this time |
[production] |
| 02:23 |
<mark> |
db20 back up, for reals |
[production] |
| 02:19 |
<mark> |
Rebooting db20 with upgraded RAID controller firmware |
[production] |
| 02:13 |
<domas> |
flashing BIOS helped |
[production] |
| 02:13 |
<mark> |
db20 up! |
[production] |
| 02:04 |
<brion> |
services on bart (secure, planet) are temporarily offline while server is poked at |
[production] |
| 01:50 |
<brion> |
seeing pages, yay |
[production] |
| 01:49 |
<brion> |
running apache2ctl start or apachectl start for various apaches |
[production] |
| 01:47 |
<domas> |
I FOUND HOW TO REVIVE APACHES |
[production] |
| 01:46 |
<brion> |
think i killed em, now trying to restart apache procs |
[production] |
| 01:43 |
<brion> |
poking to see if we can restart apaches... |
[production] |
| 01:42 |
<brion> |
syncing fixed InitialiseSettings/COmmonSettings to apaches |
[production] |
| 01:14 |
<brion> |
and flyingparchment |
[production] |
| 01:14 |
<brion> |
domas and mark are attempting to restart the NFS server, but aren't mentioning any details in the public channel or log |
[production] |
| 00:52 |
<domas> |
http://p.defau.lt/?_M1iGbA0PCz2OOt2_KKPug |
[production] |
| 00:52 |
<mark> |
db20 in trouble |
[production] |
| 00:39 |
<mark> |
@brion you don't need to wake up |
[production] |
| 00:36 |
<domas> |
disabled 2006 fundraising cronjob on amane :-) |
[production] |
|
2009-02-20
§
|
| 23:31 |
<Rob> |
upgraded squid and kernel on sq34-sq36 |
[production] |
| 23:12 |
<Rob> |
upgraded kernel and squid on sq31-sq33, redeployed and online |
[production] |
| 23:08 |
<brion> |
updating CentralNotice for improved test script (plus i8n update) |
[production] |
| 22:54 |
<Rob> |
upgraded kernel + squid on sq28-sq30 |
[production] |
| 22:29 |
<Rob> |
completed upgrades to sq25-sq27 |
[production] |
| 22:12 |
<Rob> |
upgrading kernel and squid versions on sq25-sq27 (if i crash the site, i apologize in advance) |
[production] |
| 22:08 |
<Rob> |
upgraded kernel and squid on sq24 |
[production] |
| 21:59 |
<river> |
added current patches to ms4, set zil_disable=1 and rebooted |
[production] |
| 21:30 |
<brion> |
srv31 seems to be down, so no dump activity |
[production] |
| 21:08 |
<brion> |
scapping to update FlaggedRevs to r47588 (fixing fatal err) |
[production] |
| 21:01 |
<Rob> |
updated kernel and squid on sq23 |
[production] |
| 20:58 |
<Rob> |
updated kernel and squid on sq22 |
[production] |
| 20:36 |
<Rob> |
updated kernel and squid on sq20 and sq21 |
[production] |
| 20:25 |
<domas> |
some apaches in crashloop like this: http://p.defau.lt/?s9YhHD_0qHroVhauBdQb_g |
[production] |
| 20:09 |
<Rob> |
restarted apache on srv73 |
[production] |
| 20:03 |
<Rob> |
upgraded kernel and squid on sq19 |
[production] |
| 19:50 |
<Rob> |
upgraded kernel + squid on sq18 |
[production] |
| 19:34 |
<Rob> |
upgraded kernel + squid on sq17 |
[production] |
| 19:19 |
<brion> |
updating FlaggedRevs to r47574 |
[production] |
| 18:17 |
<river> |
test |
[production] |
| 18:17 |
<river> |
test |
[production] |
| 18:16 |
<river> |
set zil_disable on ms1 to improve nfs write performance |
[production] |
| 18:15 |
<mark> |
Raised max-conns to 50 |
[production] |
| 18:03 |
<mark> |
Cut down max conns even more (25) for pmtpa upload backend squids |
[production] |
| 17:40 |
<mark> |
Limited maximum connections to backend (ms1) to 50 per squid on upload squids, 1000 per squid on text |
[production] |
| 16:19 |
<NicDumZ> |
nagios definitely needs a nagia |
[production] |
| 16:17 |
<domas> |
plenty of fedoras had futex deadlocks |
[production] |