Using Detours Express from GCC

Detours Express must be compiled using a Microsoft C++ compiler. It can be used from GCC, but some significant issues do come up.

When attempting to use Detours from GCC, the first problem is that GNU ld cannot resolve some symbols in static libraries that were created by Microsoft compilers. This can be solved by creating a DLL. First, a .def Module Definition File needs to be created to export the needed symbols. The list of functions can easily be obtained from the header file, using sed -n "s/^.* WINAPI \([^(]*\)(.*$/\1/p" detours.h. Once you have that, you just have to add a single data export and the statements that go at the beginning of the .def file:

..\detoured.lib msvcrt.lib kernel32.lib /out:detours.dll
LIBRARY detours
EXPORTS
DETOUR_EXE_RESTORE_GUID DATA

If you want, you can create a resource file with VERSIONINFO. Simply copy detours.rc in the Detours source directory and then edit it. The only fields that need to be edited are the FileDescription and DLL names. It's probably also a good idea to set the VS_FF_PRIVATEBUILD flag in FILEFLAGS by setting FILEFLAGS to 0x8L, and below provide a "PrivateBuild" VALUE with information about your build.

If you want to name your DLL detours.dll, it must not be built in the lib directory, because that would overwrite detours.lib, which is your input file. So, go to another directory and create the DLL:

set LIB=C:\WinDDK\7600.16385.1\lib\wxp\i386;C:\WinDDK\7600.16385.1\lib\Crt\i386
link /release /machine:x86 /dll /def:detours.def /incremental:no /subsystem:console detours.res ..\detours.lib ..\detoured.lib msvcrt.lib kernel32.lib /out:detours.dll

At this point, you have a usable DLL, but it's a good idea to create a corresponding .dll.a interface library file for GCC. This is a simple process, but a stdcall function decoration issue needs to be dealt with. The Detours Express API uses the stdcall calling convention, but the functions in the DLL have no decoration. (Windows system DLLs like kernel32.dll do the same thing.) It can be a problem, because GCC will attempt to link stdcall functions to decorated names such as DetourAttach@8. To solve this, create an entirely new .def file using gendef. Use the the -a switch, because gendef can't detect that zero-argument functions use stdcall, and do it in a new directory if you want to keep the old detours.def. DetourGetDetouredMarker forwards to Detoured in detoured.dll, so either run gendef on detoured.dll first or add the @0 to that name manually. Use dlltool to create the interface library, using the -k switch so the decorated names in the interface library can dynamically link with the undecorated exports in detours.dll:

dlltool -k -d detours.def -D detours.dll -l libdetours.a

Finally, everything is ready for compilation. The sample programs provide a simple way to test Detours Express, and with some minor changes, they can be compiled with g++. The main issue is that g++ follows the standard and refuses to automatically convert function pointers to void pointers, but that's easy to fix by adding (PVOID) casts.

Unfortunately, when using GNU ld from binutils 2.20.51.20100410, functions which are imported from DLLs are intercepted in the current module, bit not in any other modules. For example, the simple sample outputs that it "slept 0 ticks". This is because the thunk (which is an indirect jump to the actual function) is intercepted instead of the actual function. Detours attempts to correct this via DetourCodeFromPointer, but it fails because GNU ld doesn't fill out the data directory structure for the import address table and detour_is_imported returns false. (You can use objdump -p to view the data directory.) This bug was fixed in September 2010, and the binutils-2.21.51.tar.bz2 snapshot contains the fix. (Here is a thread discussing the fix and one CVS log entry from the fix.)

If you're using a buggy ld, it's also possible to work around this problem by declaring imported functions with __declspec(dllimport). When using the MinGW headers, Windows API functions may be declared this way by defining __W32API_USE_DLLIMPORT__ before including header files. This causes the corresponding function addresses to be correct, but no longer constant (due to dynamic linking). When compiling via g++, this is not a problem because the compiler automatically generates code to initialize global variables. However, gcc cannot do this and so code must be added to initialize such global variables at runtime.