This project has moved. For the latest updates, please go here.

VLD 2.0, VC++2005, Crash after stack overflow on calling RtlAllocateHeap() from vldnew()

Aug 19, 2010 at 7:48 AM
Edited Aug 19, 2010 at 8:18 AM

I wrote CSP (Cryptographic Service Provider) DLL Library (little MFC based, MFC statically linked, CRT statically linked).

I compile all my code with Visual Studio 2005 Standard Edition.

When I use my library from my (rather simple) application, works ok, VLD reports leaks if there are any and application works fine.

When I use it from Internet Explorer for SSL connection, without VLD enabled ("vld.h" not included and vld.lib not linked in) it works fine.

When I enable VLD, iexplore crashes and on debugger output is:

'iexplore.exe': Loaded 'D:\users\mskalski\projekty\ENCARD\exe\enigmacsp_debug.dll', Symbols loaded.
'iexplore.exe': Loaded 'C:\opt\VisualLeakDetector\bin\Win32\vld_x86.dll', Symbols loaded.
'iexplore.exe': Loaded 'C:\Windows\system32\dbghelp.dll', Exports loaded.
Visual Leak Detector Version 2.0 installed.
    Outputting the report to the debugger and to c:\log\vld\memory_leak_report.txt
    Perfoming a memory leak self-test.
First-chance exception at 0x7c90e8e5 (ntdll.dll) in iexplore.exe: 0xC00000FD: Stack overflow.
First-chance exception at 0x7c90e8e5 (ntdll.dll) in iexplore.exe: 0xC0000005: Access violation writing location 0x00040eb4.
Unhandled exception at 0x7c90e8e5 (ntdll.dll) in iexplore.exe: 0xC0000005: Access violation writing location 0x00040eb4.
The program '[4440] iexplore.exe: Native' has exited with code 0 (0x0).

enigmacsp_debug.dll is my CSP library.

dbghelp.dll's version is 6.11.1.404, it is the same which is distributed with vld.

vld.dll's version is 2.0a, compiled by me with Visual Studio 2005, with dbghelp.h and dbghelp.lib from MS SDK for Windows Server 2008 (6.0.6001.18000)

Debugger breaks on vldnew() function, on call to RtlAllocateHeap() and call stack shows deep recursion:

 	ntdll.dll!strchr()  + 0xd8 bytes	
 	[Frames below may be incorrect and/or missing, no symbols loaded for ntdll.dll]	
 	ntdll.dll!RtlpNtMakeTemporaryKey()  + 0x749c bytes	
 	ntdll.dll!LdrAlternateResourcesEnabled()  + 0x2b05 bytes	
 	ntdll.dll!RtlDosSearchPath_Ustr()  + 0x310 bytes	
>	vld_x86.dll!vldnew(unsigned int size=0x00000014, const char * file=0x03aaedf0, int line=0x00000472)
            Line 188 + 0x18 bytes	C++
 	vld_x86.dll!operator new(unsigned int size=0x00000014, const char * file=0x03aaedf0, int line=0x00000472)
            Line 111 + 0x11 bytes	C++
 	vld_x86.dll!VisualLeakDetector::gettls()  Line 1138 + 0x15 bytes	C++
 	vld_x86.dll!VisualLeakDetector::enabled()  Line 1051 + 0xa bytes	C++
 	vld_x86.dll!VisualLeakDetector::_RtlAllocateHeap(void * heap=0x00150000, unsigned long flags=0x00000008,
            unsigned long size=0x00001000)  Line 2513 + 0x10 bytes	C++
 	kernel32.dll!SetUnhandledExceptionFilter()  + 0x1e6 bytes	
 	vld_x86.dll!VisualLeakDetector::gettls()  Line 1139 + 0x16 bytes	C++
 	vld_x86.dll!VisualLeakDetector::enabled()  Line 1051 + 0xa bytes	C++
 	vld_x86.dll!VisualLeakDetector::_RtlAllocateHeap(void * heap=0x00150000, unsigned long flags=0x00000008,
            unsigned long size=0x00001000)  Line 2513 + 0x10 bytes	C++
 	kernel32.dll!SetUnhandledExceptionFilter()  + 0x1e6 bytes	
 	vld_x86.dll!VisualLeakDetector::gettls()  Line 1139 + 0x16 bytes	C++
 	vld_x86.dll!VisualLeakDetector::enabled()  Line 1051 + 0xa bytes	C++
 	vld_x86.dll!VisualLeakDetector::_RtlAllocateHeap(void * heap=0x00150000, unsigned long flags=0x00000008,
            unsigned long size=0x00001000)  Line 2513 + 0x10 bytes	C++

Crash occurs before any of my regular DLL functions are executed, and I think before initialization of C/C++ objects (DLL has many global C++ objects treated as const, but normally initialized), they are not initialized yet at time of crash.

After small investigation in debugger I checked that error occurs in that code of CRT function _heap_init(), around line 200 of heapinit.c in CRT.

        //  Initialize the "big-block" heap first.
        if ( (_crtheap = HeapCreate( mtflag ? 0 : HEAP_NO_SERIALIZE,
                                     BYTES_PER_PAGE, 0 )) == NULL )
            return 0;

call to HeapCreate causes crash, which is called from DLL initialization function:

BOOL WINAPI _CRT_INIT( HANDLE  hDllHandle, DWORD   dwReason, LPVOID  lpreserved )
here call to _heap_init() causes stack overflow and then crash.

I don't know if it is real cause of crash, stack overflow (caused by infinite recurse calling of RtlAllocateHeap()) occurs in other thread.

After some further investigation recursive invocation comes from VisualLeakDetector::gettls() function, from TlsSetValue(m_tlsindex, tls) invocation. every call to TlsSetValue() triggers new allocation.

Here is call stack after few steps:

>	vld_x86.dll!VisualLeakDetector::gettls()  Line 1139	C++
 	vld_x86.dll!VisualLeakDetector::enabled()  Line 1051 + 0xa bytes	C++
 	vld_x86.dll!VisualLeakDetector::_RtlAllocateHeap(void * heap=0x00150000, unsigned long flags=0x00000008,
            unsigned long size=0x00001000)  Line 2513 + 0x10 bytes	C++
 	kernel32.dll!SetUnhandledExceptionFilter()  + 0x1e6 bytes	
 	[Frames below may be incorrect and/or missing, no symbols loaded for kernel32.dll]	
 	vld_x86.dll!VisualLeakDetector::gettls()  Line 1139 + 0x16 bytes	C++
 	vld_x86.dll!VisualLeakDetector::enabled()  Line 1051 + 0xa bytes	C++
 	vld_x86.dll!VisualLeakDetector::_RtlAllocateHeap(void * heap=0x00150000, unsigned long flags=0x00000008,
            unsigned long size=0x00001000)  Line 2513 + 0x10 bytes	C++
 	kernel32.dll!SetUnhandledExceptionFilter()  + 0x1e6 bytes	
 	vld_x86.dll!VisualLeakDetector::gettls()  Line 1139 + 0x16 bytes	C++
 	vld_x86.dll!VisualLeakDetector::enabled()  Line 1051 + 0xa bytes	C++
 	vld_x86.dll!VisualLeakDetector::_RtlAllocateHeap(void * heap=0x00150000, unsigned long flags=0x00000008,
            unsigned long size=0x00001000)  Line 2513 + 0x10 bytes	C++
 	kernel32.dll!SetUnhandledExceptionFilter()  + 0x1e6 bytes	
 	vld_x86.dll!VisualLeakDetector::gettls()  Line 1139 + 0x16 bytes	C++
 	vld_x86.dll!VisualLeakDetector::enabled()  Line 1051 + 0xa bytes	C++
 	vld_x86.dll!VisualLeakDetector::_RtlAllocateHeap(void * heap=0x00150000, unsigned long flags=0x00000008,
            unsigned long size=0x00001000)  Line 2513 + 0x10 bytes	C++
 	kernel32.dll!SetUnhandledExceptionFilter()  + 0x1e6 bytes	
 	vld_x86.dll!VisualLeakDetector::gettls()  Line 1139 + 0x16 bytes	C++
 	vld_x86.dll!VisualLeakDetector::enabled()  Line 1051 + 0xa bytes	C++
 	vld_x86.dll!VisualLeakDetector::_RtlAllocateHeap(void * heap=0x00150000, unsigned long flags=0x00000008,
            unsigned long size=0x00001000)  Line 2513 + 0x10 bytes	C++
 	kernel32.dll!SetUnhandledExceptionFilter()  + 0x1e6 bytes	
 	vld_x86.dll!VisualLeakDetector::gettls()  Line 1139 + 0x16 bytes	C++
 	vld_x86.dll!VisualLeakDetector::enabled()  Line 1051 + 0xa bytes	C++
 	vld_x86.dll!VisualLeakDetector::_RtlAllocateHeap(void * heap=0x00150000, unsigned long flags=0x00000008,
            unsigned long size=0x00001000)  Line 2513 + 0x10 bytes	C++
 	kernel32.dll!SetUnhandledExceptionFilter()  + 0x1e6 bytes	
 	vld_x86.dll!VisualLeakDetector::gettls()  Line 1139 + 0x16 bytes	C++
 	vld_x86.dll!VisualLeakDetector::enabled()  Line 1051 + 0xa bytes	C++
 	vld_x86.dll!VisualLeakDetector::_RtlAllocateHeap(void * heap=0x00150000, unsigned long flags=0x00000000,
            unsigned long size=0x00000130)  Line 2513 + 0x10 bytes	C++
 	rpcrt4.dll!I_RpcBCacheFree()  + 0x1b66 bytes	
 	rpcrt4.dll!I_RpcBCacheFree()  + 0x1b3d bytes	
 	rpcrt4.dll!RpcSsDestroyClientContext()  + 0x918 bytes	
 	rpcrt4.dll!NdrSimpleStructFree()  + 0x239 bytes	
 	rpcrt4.dll!I_RpcBCacheFree()  + 0x61c bytes	
 	rpcrt4.dll!I_RpcBCacheFree()  + 0x43e bytes	
 	rpcrt4.dll!I_RpcBCacheFree()  + 0x604 bytes	
 	kernel32.dll!GetModuleFileNameA()  + 0x1ba bytes	

It looks like a some kind of race condition. There are about 17 threads at the moment.

I don't know it is important, but I have problem with RPC on my computer - maybe too restrictive intranet firewall?

First-chance exception at 0x7c812afb (kernel32.dll) in iexplore.exe: 0x000006BA: Serwer RPC jest niedostępny.
First-chance exception at 0x7c812afb (kernel32.dll) in iexplore.exe: 0x000006D9: Nie ma więcej dostępnych
            punktów końcowych z programu mapowania punktów końcowych.
First-chance exception at 0x7c812afb (kernel32.dll) in iexplore.exe: 0x000006BA: Serwer RPC jest niedostępny.
First-chance exception at 0x7c812afb (kernel32.dll) in iexplore.exe: 0x000006D9: Nie ma więcej dostępnych
            punktów końcowych z programu mapowania punktów końcowych.

In English: 0x000006BA: RPC Server is unavailable and 0x000006D9: There are no more endpoints available

PS Sorry, first time, I didn't format correctly

Aug 19, 2010 at 8:13 AM
Edited Aug 19, 2010 at 7:42 PM
It doesn't help if I start vld disabled and enable it in first interesting for me function in my dll. It ends in stack overflow, thread showed in debugger is
4272	SoftwareUpdateMessageBox	VisualLeakDetector::gettls	Normal	0
The thread 'Win32 Thread' (0x1468) has exited with code 0 (0x0).
'iexplore.exe': Loaded 'C:\Windows\system32\encardcsp_debug-3.0.dll', Symbols loaded.
'iexplore.exe': Loaded 'D:\users\mskalski\projekty\ENCARD\exe\enigmacsp_debug.dll', Symbols loaded.
'iexplore.exe': Loaded 'C:\opt\VisualLeakDetector\bin\Win32\vld_x86.dll', Symbols loaded.
'iexplore.exe': Loaded 'C:\Windows\system32\dbghelp.dll', Exports loaded.
Visual Leak Detector Version 2.0 installed.
    Outputting the report to the debugger and to c:\log\vld\memory_leak_report.txt
    Starting with memory leak detection disabled.
First-chance exception at 0x7c90e8e5 (ntdll.dll) in iexplore.exe: 0xC00000FD: Stack overflow.
First-chance exception at 0x7c90e8e5 (ntdll.dll) in iexplore.exe: 0xC0000005: Access violation writing location 0x01f00f7c.
Unhandled exception at 0x7c90e8e5 (ntdll.dll) in iexplore.exe: 0xC0000005: Access violation writing location 0x01f00f7c.
Aug 19, 2010 at 8:24 AM
Just an idea: There are 17 threads, maybe setting TLS value for that 17th thread triggers additional reallocation (earlier 16 had tls pointers preallocated)?
Dec 3, 2013 at 2:45 AM
It looks like a some kind of race condition, still occurs in the latest version of VLD.
Dec 7, 2013 at 5:22 AM
Edited Dec 7, 2013 at 5:25 AM
not race condition. The hook routine of RtlAllocateHeap just indirectly calls RtlAllocateHeap itself, so the endless recursive call occurs and leads to stack overflow.

call chain((actually a loop):

RtlAllocateHeap -> RtlAllocateHeap hook routine ->enabled ->gettls -> TlsSetValue -> VirtualUnlock -> RtlAllocateHeap
Apr 6, 2014 at 10:17 AM
So, is there any solution or workaround which breaks this loop?