Bugzilla has received a substantial upgrade. As such, a number of functions and behaviors have changed. Please review the Bugzilla Upgrade Slidedeck and the Help Page for more information.
Bug 557921 - System.Diagnostics.Process: error creating process handle
System.Diagnostics.Process: error creating process handle
Status: NEW
Classification: Mono
Product: Mono: Runtime
Classification: Mono
Component: io-layer
2.4.x
x86 Linux
: P5 - None : Normal
: ---
Assigned To: Mono Bugs
Mono Bugs
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2009-11-24 00:23 UTC by Brian Luczkiewicz
Modified: 2011-06-28 18:01 UTC (History)
2 users (show)

See Also:
Found By: ---
Services Priority:
Business Priority:
Blocker: ---


Attachments
Work-around diff against /tags/mono-2-4-2-3/mono (1.68 KB, patch)
2010-02-19 21:32 UTC, Chris Eldredge
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Brian Luczkiewicz 2009-11-24 00:23:54 UTC
User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/532.0 (KHTML, like Gecko) Chrome/3.0.195.27 Safari/532.0

It appears that System.Diagnostics.Process is leaking "process handles". After kicking off approximately 500 processes (and disposing each Process object immediately after calling Start()), I start getting warning trace each time I attempt to start the process. No exception is thrown yet the process is not started.

Since this bug was encountered in context of a production system, I've included a tiny test program below that reproduces the bug reliably and quickly on mono 2.4.2.3.

The test program creates a lot of processes very quickly. The production system creates processes at a significantly slower rate: less than one per minute. As a result, the production system doesn't fail until it's been running for many hours. Both exhibit the same failure, however. It doesn't seem that the rate of process creation is at fault here.

If I kill the program and immediately  run it again, then mono segv's (stack dump is included below). rm'ing ~/.wapi seems to repair the leak temporarily, as does waiting a few minutes before starting the program again runs.

If I add a call to p.WaitForExit() directly after the call to p.Start() then all failures go away and it runs for hundreds of thousands of iterations without issue. 

The .net api as published by microsoft does not require that WaitForExit is called, nor does it fail to execute this code. The call to Dispose() on the process object should be sufficient to release all mono-spefic resources associated with the process.

I see that there was a bug for this years ago that was closed a long time ago (https://bugzilla.novell.com/show_bug.cgi?id=79040). Other assorted googling seems to suggest that there is uncertainty of whether the "fix" for this past bug actually addressed the root cause.

Also of note: 

- this program seems to leak memory as it continues to iteratively print the WARNING message. After letting it run for five minutes it has ~100mb resident, compared to ~15mb at startup.
- in the midst of printing the WARNING message, after a few minutes, it will start to successfully create 'ls' processes once again, later reverting to the failure behavior.




Reproducible: Always

Steps to Reproduce:
Compile and run this code:

using System;
using System.Diagnostics;

public class Test
{
    public static void Main(string[] args)
    {
        int count = 0;
        while (true) {
            Console.WriteLine("Starting Process #{0}", (++count));
            using (Process p = new Process()) {
                p.StartInfo.FileName = "/bin/ls";
                p.StartInfo.UseShellExecute = false;
                p.StartInfo.CreateNoWindow = true;
                p.StartInfo.WorkingDirectory = "/";
                p.Start();
            }
            count++;
        }
    }
}

Actual Results:  
Program runs for about 500 iterations of the main loop, each time printing the directory listing of "/". After approximately 500 iterations, it start printing the following once per call to p.Start():

** (testprocess.exe:10013): WARNING **: CreateProcess: error creating process handle

If I kill the program and run it again immediately, mono segfaults with the following output:

brian@zuu test $ mono testprocess.exe

** (testprocess.exe:10533): WARNING **: process_set_current: error creating process handle
Starting Process #1

** (testprocess.exe:10533): WARNING **: CreateProcess: error creating process handle
Stacktrace:

  at (wrapper managed-to-native) System.Diagnostics.Process.CreateProcess_internal (System.Diagnostics.ProcessStartInfo,intptr,intptr,intptr,System.Diagnostics.Process/ProcInfo&) <0x00004>
  at (wrapper managed-to-native) System.Diagnostics.Process.CreateProcess_internal (System.Diagnostics.ProcessStartInfo,intptr,intptr,intptr,System.Diagnostics.Process/ProcInfo&) <0xffffffff>
  at System.Diagnostics.Process.Start_noshell (System.Diagnostics.ProcessStartInfo,System.Diagnostics.Process) <0x00847>
  at System.Diagnostics.Process.Start_common (System.Diagnostics.ProcessStartInfo,System.Diagnostics.Process) <0x000d9>
  at System.Diagnostics.Process.Start () <0x00042>
  at (wrapper remoting-invoke-with-check) System.Diagnostics.Process.Start () <0xffffffff>
  at Test.Main (string[]) <0x00118>
  at (wrapper runtime-invoke) Test.runtime_invoke_void_object (object,intptr,intptr,intptr) <0xffffffff>

Native stacktrace:

        mono [0x80cc044]
        mono [0x80f671b]
        [0xb7f5540c]
        mono [0x81b16b6]
        mono [0x81e45d8]
        [0xb78ae229]
        [0xb78ad238]
        [0xb78ac87a]
        [0xb78ac773]
        [0xb78ac6fc]
        [0xb789f3a1]
        [0xb789f203]
        mono(mono_runtime_exec_main+0xe5) [0x8185625]
        mono(mono_runtime_run_main+0x16b) [0x8185dcb]
        mono(mono_main+0x190a) [0x80b38fa]
        mono [0x805afb1]
        /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe5) [0xb7cf6775]
        mono [0x805aee1]

Debug info from gdb:

[Thread debugging using libthread_db enabled]
[New Thread 0xb7cac6f0 (LWP 10533)]
[New Thread 0xb74e1b90 (LWP 10535)]
[New Thread 0xb7505b90 (LWP 10534)]
0xb7f55424 in __kernel_vsyscall ()
  3 Thread 0xb7505b90 (LWP 10534)  0xb7f55424 in __kernel_vsyscall ()
  2 Thread 0xb74e1b90 (LWP 10535)  0xb7f55424 in __kernel_vsyscall ()
  1 Thread 0xb7cac6f0 (LWP 10533)  0xb7f55424 in __kernel_vsyscall ()

Thread 3 (Thread 0xb7505b90 (LWP 10534)):
#0  0xb7f55424 in __kernel_vsyscall ()
#1  0xb7e768f6 in nanosleep () from /lib/tls/i686/cmov/libpthread.so.0
#2  0x081bbb48 in collection_thread (unused=0x0) at collection.c:34
#3  0xb7e6f4ff in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
#4  0xb7dc449e in clone () from /lib/tls/i686/cmov/libc.so.6

Thread 2 (Thread 0xb74e1b90 (LWP 10535)):
#0  0xb7f55424 in __kernel_vsyscall ()
#1  0xb7e753f5 in sem_wait@@GLIBC_2.1 () from /lib/tls/i686/cmov/libpthread.so.0
#2  0x08162309 in finalizer_thread (unused=0x0) at gc.c:1058
#3  0x08190e38 in start_wrapper (data=0x894f040) at threads.c:623
#4  0x081b5a26 in thread_start_routine (args=0x895161c) at threads.c:286
#5  0x081d0218 in GC_start_routine (arg=0x35f20) at pthread_support.c:1382
#6  0xb7e6f4ff in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
#7  0xb7dc449e in clone () from /lib/tls/i686/cmov/libc.so.6

Thread 1 (Thread 0xb7cac6f0 (LWP 10533)):
#0  0xb7f55424 in __kernel_vsyscall ()
#1  0xb7dc0a87 in syscall () from /lib/tls/i686/cmov/libc.so.6
#2  0x080cc16d in mono_handle_native_sigsegv (signal=11, ctx=0xb7b65d0c) at mini-exceptions.c:1526
#3  0x080f671b in mono_arch_handle_altstack_exception (sigctx=0xb7b65d0c, fault_addr=0x3b80, stack_ovf=0)
    at exceptions-x86.c:881
#4  <signal handler called>
#5  _wapi_handle_unref (handle=0x3dd70) at handles.c:1048
#6  0x081b16b6 in CloseHandle (handle=0x70) at handles.c:1327
#7  0x081e45d8 in ves_icall_System_Diagnostics_Process_CreateProcess_internal (proc_start_info=0x3dd70,
    stdin_handle=0x0, stdout_handle=0x1, stderr_handle=0x2, process_info=0xbfb733fc) at process.c:782
#8  0xb78ae229 in ?? ()
#9  0xb78ad238 in ?? ()
#10 0xb78ac87a in ?? ()
#11 0xb78ac773 in ?? ()
#12 0xb78ac6fc in ?? ()
#13 0xb789f3a1 in ?? ()
#14 0xb789f203 in ?? ()
#15 0x08185625 in mono_runtime_exec_main (method=0x76f50, args=0x3be70, exc=0x0) at object.c:3309
#16 0x08185dcb in mono_runtime_run_main (method=0x8937264, argc=0, argv=0xbfb73808, exc=0x0) at object.c:3089
#17 0x080b38fa in mono_main (argc=2, argv=0xbfb73804) at driver.c:972
#18 0x0805afb1 in main (argc=144297728, argv=0x0) at main.c:34
#0  0xb7f55424 in __kernel_vsyscall ()

=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries
used by your application.
=================================================================

Aborted

If I rm -Rf ~/.wapi or wait a few minutes between runs, then I can run it again and get 500 more successful p.Start() calls before the failures begin.

Expected Results:  
Program runs indefinitely, printing the directory listing of "/" over and over again.
Comment 1 Andrew Buchanan 2010-01-06 23:16:48 UTC
I also suffer from this problem.  I found that be keeping references to the process handles and then checking whether they had exited at a later time and then closing the process handle manually mostly worked.  But regardless this is actually my biggest problem with mono right now.  


Mono JIT compiler version 2.4.2.3 (tarball Tue Aug 25 03:12:28 EDT 2009)
Copyright (C) 2002-2008 Novell, Inc and Contributors. www.mono-project.com
        TLS:           __thread
        GC:            Included Boehm (with typed GC)
        SIGSEGV:       altstack
        Notifications: epoll
        Architecture:  amd64
        Disabled:      none
[root@abnvr64 ~]# uname -a
Linux abnvr64 2.6.31.9-174.fc12.x86_64 #1 SMP Mon Dec 21 05:33:33 UTC 2009 x86_64 x86_64 x86_64 GNU/Linux
[root@abnvr64 ~]#



** (/usr/bin/abvisor/abvisor.exe:20670): WARNING **: CreateProcess: error creating process handle
Comment 2 Chris Eldredge 2010-02-19 21:30:08 UTC
We ran into this as well.  It looks like mono/io-layer/handles.c:481 is the culprit.  There are initially 256 shared handles available for allocation and if this number is exceeded no more can be allocated.  There's a command that says, "FIXME: grow the arrays."

I suspect when this happens the private handle previously allocated in the same function is leaked.
Increasing _WAPI_HANDLE_INITIAL_COUNT (wapi-private.h:125) to some arbitrarily high number works around the issue but doesn't really fix it.

Additionally, processes.c should set ret = FALSE near line 1037 to tell the calling code that it failed.  It doesn't and as a result no Win32Exception gets thrown.  In any case, once it fails it seems to get stuck even after some shared handles are deallocated by processes that exit.

Backporting the fix from bug 536776 (r141381) does not resolve the issue.
-----------------------------------------------------------------------------

Debian 5.0 (Linux localhost 2.6.26-2-amd64 #1 SMP Thu Nov 5 02:23:12 UTC 2009 x86_64 GNU/Linux)

Mono JIT compiler version 2.4.2.3 (/tags/mono-2-4-2-3/mono r146859 Fri Feb 19 15:54:27 EST 2010)
Copyright (C) 2002-2008 Novell, Inc and Contributors. www.mono-project.com
        TLS:           __thread
        GC:            Included Boehm (with typed GC)
        SIGSEGV:       altstack
        Notifications: epoll
        Architecture:  amd64
        Disabled:      none
Comment 3 Chris Eldredge 2010-02-19 21:32:20 UTC
Created attachment 343600 [details]
Work-around diff against /tags/mono-2-4-2-3/mono
Comment 4 Gonzalo Paniagua Javier 2010-02-20 08:03:51 UTC
This problem does not happen in mono-2-6 or trunk. I believe it has been fixed in mono-2-4 too, after 2.4.2.4 was released.