Crash, Boom: Too Many Open Files…

We’ve been plagued by an error similar to this in some long-running server code:

Exception in thread “main” java.io.IOException: Cannot run program “…whatever”: error=24, Too many open files

I am pretty sure I tracked down the problem: failure to correctly close Process resources. Consider the case where you create a Process and read data from its InputStream:

(note: this is not a complete example, plus it is wrong!)

// same thing occurs with Runtime.exec(...)
ProcessBuilder pb = new ProcessBuilder(args);
Process proc = pb.start();
InputStream in = proc.getInputStream();
// ...read from the stream in a thread
proc.waitFor();

// ...later, in a finally block, close the InputStream:
in.close();

That’s the general idea. Create a Process, capture output from one or more of the streams, and close the streams when complete.

Unfortunately, this is wrong.

Poor Documentation

The documentation does not really help much. For instance, here is what the comment says for the Process.destroy() method:

Kills the subprocess. The subprocess represented by this Process object is forcibly terminated.

Hmm…that doesn’t really say how the streams are handled. I’ve always assumed if I used one of the streams, I had to close the one I used.

Here are three bug reports asking for documentation clarification:

• Bug 4523660
• Bug 4801027
• Bug 6462165

After studying these reports, I now feel confident that you must first close all three streams, then you must call destroy() in order to fully reclaim resources.

Example Code

Here is a simple example that shows what I’m talking about. Despite the fact that I do not actually use any of the streams, I still need to close them:

public class ProcDemo {
  public static void main(String[] args) throws
        InterruptedException, IOException {
    Process proc = null;
    try {
      ProcessBuilder pb = new ProcessBuilder(args);
      proc = pb.start();
      proc.waitFor();
    } finally {
      if (proc != null) {
        close(proc.getOutputStream());
        close(proc.getInputStream());
        close(proc.getErrorStream());
        proc.destroy();
      }
    }
  }

  private static void close(Closeable c) {
    if (c != null) {
      try {
        c.close();
      } catch (IOException e) {
        // ignored
      }
    }
  }
}

If you study the code for java.lang.ProcessImpl, you’ll see that when you construct the Process object, all three streams are created right away. Thus, regardless of whether you use a stream, you must close all of them.

And even though the Javadoc for destroy() is not all that clear, from reading the bug reports, it is pretty clear you should call it as well.

Problem Solved?

It will probably be several days before I know if this truly solves our problem. While I was able to clean up all of the code I wrote, I don’t have much control over the various libraries we use. Does JGroups always close Process resources correctly? How about Svnkit? We often see these “Too many open files” errors immediately after a large Subversion checkout, so I am somewhat skeptical.

Posted on Wednesday, November 28th, 2007 at 10:05 pm in Java.