View previous topic :: View next topic |
Author |
Message |
chris Forum Fanatic
Joined: 18 Apr 2002 Posts: 16777201 Location: ~/security-forums
|
Posted: Fri Nov 22, 2002 11:28 am Post subject: linux problems [too many open files] |
|
|
Ive had this problem a few times now, the box has locked out the pop3 daemon totally so noone can check mail.
Here is the tail of the exim_paniclog
Quote: |
Too many open files in system
|
Then kern.log
Quote: |
Nov 14 21:46:23 bleh kernel: VFS: file-max limit 4096 reached
Nov 15 10:11:07 bleh kernel: Unable to load interpreter /lib/ld-linux.so.2
Nov 15 10:11:09 bleh last message repeated 2 times
Nov 15 10:12:28 bleh last message repeated 8 times
|
When this happens even doing commands such as ls bring up the too many open files in system. Its a highish spec box which has previously been up for months with no problems, only recent additions are the latest apache and openSSL.
When this happened the wierdest thing is nothing is shown in top as looking dodgy so still havent managed to narrow this down.
Ive found similar probs through google with the max files
/proc/sys/fs/file-max
so this has been increased from 4k to 16k.
Problem is I cant see how that many files can be open on the system !!
Any help appreciated
Last edited by chris on Mon Nov 25, 2002 11:18 pm; edited 1 time in total |
|
Back to top |
|
|
ShaolinTiger Forum Fanatic
Joined: 18 Apr 2002 Posts: 16777215 Location: Kuala Lumpur, Malaysia
|
Posted: Fri Nov 22, 2002 11:35 am Post subject: |
|
|
Check it, it's probably gone back down again..
echo 32768 > /proc/sys/fs/file-max
Found this too
"for dynamic updates, use sysctl and put it in rc.local. If you want it to be
permanent, you can recompile the kernel (this way, you can include the
#define NR_TASKS change).
on my case, i just used the file-max and inode-max kernel settings. It would have been better if NR_TASKS was included in sysctl params "
I think it's probably a limitation/bug in VFS.
Maybe someone else can enlighten you a little more.
|
|
Back to top |
|
|
chris Forum Fanatic
Joined: 18 Apr 2002 Posts: 16777201 Location: ~/security-forums
|
Posted: Fri Nov 22, 2002 12:12 pm Post subject: |
|
|
chris@anfield:/var/log$ cat /proc/sys/fs/file-max
16384
Sorry forgot to mention these are already added to rc.d
# Increase open file limit (from 4096 files
echo 16384 > /proc/sys/fs/file-max
# Plus more inodes
echo 65536 > /proc/sys/fs/inode-max
Since having these changed and the whole machine being restarted its still done the too many open files.
Although I do want a quick fix would also like to see how there could be this many open files at any one time, doesnt seem possible to what the box is actually doing
|
|
Back to top |
|
|
ShaolinTiger Forum Fanatic
Joined: 18 Apr 2002 Posts: 16777215 Location: Kuala Lumpur, Malaysia
|
Posted: Fri Nov 22, 2002 12:25 pm Post subject: |
|
|
lsof | more
Have a look at ps, strace, tree etc
But lsof is the main one you want.
And there will be a lot of files open.
|
|
Back to top |
|
|
chris Forum Fanatic
Joined: 18 Apr 2002 Posts: 16777201 Location: ~/security-forums
|
Posted: Fri Nov 22, 2002 12:34 pm Post subject: |
|
|
Was looking for that type of command
lsof doesnt exist though (slackware)
Looked through processes
Nothing dodgy on there
|
|
Back to top |
|
|
ShaolinTiger Forum Fanatic
Joined: 18 Apr 2002 Posts: 16777215 Location: Kuala Lumpur, Malaysia
|
|
Back to top |
|
|
chris Forum Fanatic
Joined: 18 Apr 2002 Posts: 16777201 Location: ~/security-forums
|
Posted: Mon Nov 25, 2002 11:26 pm Post subject: |
|
|
Cheers
I scheduled a cron job to run this a few times and date the results:
Quote: |
0,20,40, * * * * /home/chris/lsof/lsof > "openfiles`date '+%X'`"
|
Seems to be quite ..or is it ?
Quote: |
root@bleh:/home/chris/lsof/ wc -l results
2809 results
|
Its not gone over the original 4000 odd limit yet since.
Most connections by far are taken up by gnu-pop3 followed by apache then mysqld.
1300 odd of these are pop3, anything else I can check, perhaps lowering timeouts or similar server side.
|
|
Back to top |
|
|
|