Wednesday, August 03, 2011

g++ dyamic linking fail

I have a DLL which works when built, but stops working later. Simply rebuilding it creates another copy which again works and stops working later. LoadLibrary returns error 998, which mean "invalid access to memory location". In other words, a crash occurs in the DLL being loaded. The crash occurs when calling _initterm from code at the DLL entry point. At __imp__initterm there should be a pointer to _initterm in msvcrt.dll, but the location instead still contains the value that's in the file on disk. As a result, if msvcrt.dll is located at a different location, the program jumps to the wrong location. I guess this happens because msvcrt.dll is imported two times. The first import just uses _write, which is properly dynamically linked. The second import lists all the other functions that are used, and dynamic linking wasn't done for any of those. The _write from the first import is just used to print "pure virtual method called\n".

I suspect that g++ is to blame, but I'm still not sure. The problem is now happening with g++ 4.5.3 from the 32-bit MinGW-w64 Cygwin package, but the same symptoms occurred when using the old Cygwin compiler that is based on GCC 3.

Update: The problem seems to be related to statically linking libstdc++. The duplicate msvcrt.dll exists when using -static or -static-libstdc++ (new in GCC 4.5).

Update 2: Packing the DLL with UPX 3.07 seems to fix the problem. When a packer compresses the IAT, it needs to handle dynamic linking. Apparently UPX is able to properly handle the defective IAT.

No comments: