One of my processes running on Debian server started throwing a socket: too many open files error. I then started debugging the error. I came through the following scenario.

When I ran lsof | grep 4300, the output is as follows

When I ran lsof -i TCP:4300, the output is

Now the questions are:

1. What's the difference between both of them? (My observation is that the first one has thread Id and latter one doesn't have it)




2. Why there are multiple lines for the same connection in the first one? (SS is giving 4 lines only.)




3. How many file descriptors are open?




4. How to debug this scenario?

I am referencing the manpage of lsof for my answer.

lsof | grep 4300

In the absence of any options, lsof lists all open files belonging to all active processes.

So this command lists all open files belonging to all process, you have piped its output to grep searching for only the ones that match the search operand "4300".

lsof -i TCP:4300 

This option selects the listing of files any of whose Internet address matches the address specified in i. If no address is specified, this option selects the listing of all Internet and x.25 (HP-UX) network files.

With this command you have specified that you wish to only list files with an internet address or network location, specifying further that you only want those that match the tcp protocol on port 4300.

Answering Your Questions

1. The results of the two commands are different in that the first command is list all open files but the output is filtered by grep to all lines that have "4300". The second command specifies only open internet or network files that are using the tcp protocol on port 4300.




2. According to this post your process could simply have the file open multiple times. This is not unexpected behavior of lsof. It definitely is your issue and I would look into what each of those processes are doing and why they need to have several copies of those files opened.




3. If you have included all the output of your commands than with your first one there are 3 processes which have their respective files open 5 times. In your second command there are 3 processes with open network files using the tcp protocol on port 4300 and a process that is listening to that protocol and port.




4. I would look into what each process is based off the pid. Referencing this post you can look up a process by pid using this command: ps -p <PID> -o comm=. Alternatively, I have had a lot of success with ps aux | grep <PID> to identify a process.

Conclusion

From here you will need to look up if that is expected behavior of the process and how to remedy that if they are the cause of the initial error. Do not forget to reference any logs that the error and processes could be associated with.

If there are any misconceptions or question about this answer please comment. I will edit this post as needed to improve upon the answer.

Best of Luck!