수안이의 컴퓨터 연구실

Abstract

As a developer we often find ourselves surrounded by problems that actually have nothing to do with our productivity or even the project we're working on. One of them is the issue of compatibility between the Dynamic Link Libraries developed using different tools. Recently, while working on a project, I faced this problem, and in this article I'd  like to present my analysis. We'll also talk about writing DLLs that are easy for us develop and don't cause others to face similar problems when they try to use them.

I assume that you're familiar with using Microsoft Visual C++ and Borland C++ Builder, because these are the tools I'll use to expose the nature of problem and the solution.

Some people may argue that they can't imagine why someone would ever need to use DLLs written using one tool in some different tool. An elegant solution would be to have a special build of the DLL for every other tool that can consume the DLL; but, like many other people, I don't find it appealing. Moreover, there are circumstances where you're bound to work on these issues, such as:

• You do not have the source to the DLL, or even the import library. Don't know what an import library is? Don't worry; we'll talk about that shortly.
  
  
• You have existing applications to maintain and, to add a specific feature, you company has purchased a library. The source is, of course, not available, and you still need to figure out how to call the function when linker gives up saying there are unresolved externs.

I can add more to this list but I believe the above two points give you the idea. So let's see what the problem is, apart from understanding the terminology used in the context and finding a working solution for all of these problems. What I explain here will be specific to the tools I've been using, but I'm sure problems related to other tools won't be much different.

DLL Conventions: Issues and Solutions, Part I - A simple problem
(Page 2 of 4 )

Before we move on I'll explain a generic problem that you might have experienced first hand in your career as a developer, but dismissed because someone did that for you, or you took an entirely different approach to solve that problem.

Say you just downloaded the PDFLib PDF library and purchased the license to it. Prior to your purchase, you tried the sample applications and they all just compiled happily. Now when the development started, you realized that you'll be developing the applications that use the library in Borland C++ Builder 6.0 rather than Visual C++ 6.0, in which all the samples were written and compiled. If you've been through such a situation you can surely appreciate the description of the problem above.

Let's now analyze and find the cause and solution to the problem. If you're reading this just to gain some knowledge, then you'll surely find the basic terminology discussion beneficial.

A Dynamic Link Library (DLL) is a file that contains C Style functions or whole C++ classes that you can use for developing your applications to lessen your development effort and to save time when testing problems in a normal SDLC.

Any function that is visible outside the DLL and is intended for use by external client applications is said to be an export from the DLL. Consequently any client using this function from the DLL is said to have an import from the DLL. Sometimes we call all exported C Style functions and Classes to be exported symbols as a collection.

An import library is a file with a .lib extension and provides the compiler with enough information to resolve function call references if you choose to link implicitly at compile time. You may choose to load the DLL at runtime (using LoadLibrary API) and resolve and call functions then, by locating them in the DLL (using GetProcAddress API).

The import library is the key element if you're linking implicitly, and in most cases this poses a greater problem, as we'll see shortly. Also required is the header file and the DLL you're trying to use.

So now let's recap all we've discussed until now and then we shall move to analysis of the problem. While using third party DLLs in an environment which poses certain problems in the usage, we start with three elements (please note that we're stretching only to implicit linking, and that too with DLLs exporting C style functions, and explicit linking is not in the scope of our discussion right now. But we shall discuss it in the next article of this series):

1. The DLL built with any tool and exporting C Style functions only.
   
   
2. A header file that provides prototypes to all the functions exported by the DLL.
   
   
3. The import library that is emitted by the compiler when the DLL is built.

With these three ingredients we can have all the functions and classes at our disposal and use them freely in our code. We shall have real life problems so that we can see it all happen; so we'll be using a DLL built with Microsoft Visual C++ 6.0 in Borland C++ Builder 6.0. This case will be similar to the problem mentioned above (the PDFLib case).

DLL Conventions: Issues and Solutions, Part I - Where is the actual problem?
(Page 3 of 4 )

In a world where several vendors compete to make fabulous developer tools, it is certainly impossible to have interoperability, especially when there are no standards in place. In short, calling a DLL produced by Visual C++ would be not much different than calling a DLL built with Borland C++ Builder. But to my surprise, Borland and Microsoft disagree on several issues regarding the techniques they follow and embed in the developer tools they produce.

For instance, Borland and Microsoft disagree on file formats for intermediate object files (.OBJ files) and import library files (Visual C++ uses the COFF library format while Borland uses OMF). COFF stands for Common Object File Format and OMF stands for Object Metafile Format. This means that you can't add a Microsoft generated import library to a Borland C++ Builder project. But we developers owe a big thanks to Borland for providing the IMPLIB (Import Library) utility, which means the file format differences are no longer an issue of great concern.

Further the two C++ compilers produced by these two companies also disagree on naming conventions of the exported symbols produced as a result of linking the final DLL or Libraries. This is the root cause of most problems which we developers face while we try to bridge the gap between the two.

Each exported function/symbol in a DLL or OBJect file bears a linker name. The linker uses this linker name to resolve function references that remained unresolved during compile time in order to make the final output executable or DLL. The linker generates an unresolved external error if it is unable to find a function with a linker name in the program's code itself or in the list of import libraries provided to it.

As far as the linker names are concerned, Borland and Microsoft compilers disagree on the scheme that is used to generate the linker name. For example, the Microsoft Visual C++ compiler sometimes decorates exported __stdcall functions, while the Borland C++ Builder expects that only imported __cdecl functions be decorated. If you don't understand what __stdcall and __cdecl mean, don't worry; we'll be discussing them shortly in detail.

So "How does that creates problems for us developers?" might be the question coming to your mind. Okay, let's talk about that in plain words. Say you've created a DLL in VC++ that exports a function Foo() which uses the __stdcall convention. The linker name generated for this function looks like _Foo@4. When you try using this DLL in the Borland environment, the Borland linker first expects the .lib file (import library) to be in OMF format, and then looks for a name Foo only because it is not expecting to see a decorated name for a function using the __stdcall calling convention. Okay I told you that __cdecl and __stdcall are calling conventions, but let's leave them here before we discuss them in detail. Next the Borland compiler reports an unresolved external for the name Foo.

DLL Conventions: Issues and Solutions, Part I - In the Next Article
(Page 4 of 4 )

We can try to solve these problems in different ways depending on how the Visual C++ DLL was developed. Don't worry if you don't know about that too, because we have a lot of tools at our disposal that make it visible to us.

So in the next part of this series we'll discuss what these Calling Conventions are and how can we identify exported symbols and the used calling convention. And later we will discuss how to solve the problem.

This article presented the roadblocks that developers encounter while a Visual C++ DLL is to be used from a C++ Builder project.

You might have noticed that we're discussing only "How to call C style functions in a DLL" and there is not a word about classes exported by DLLs, but in future articles we shall discuss them to the extent possible. This is done intentionally because Visual C++ DLLs present more problems, because linker names for classes' member functions are mangled. The name mangling scheme is employed in order to support function overloading.

But again, there are differences in the schemes that different vendors adopt to mangle member names. The ANSI C++ standard does not govern the specifications of how a compiler should mangle class member names. Today, in absence of a strict standard in place, all the vendors have each developed their own techniques for name mangling, and the resulting conventions are certainly not compatible. But surely we now have ways to eradicate these subtle issues.

In the previous part of this series we saw the basic issues encountered when we try to use DLLs in an environment different from the one in which they were developed. We discussed the problems with plain C style functions exported from the DLLs, and the cause of these problems.

In this part we will elaborate on the calling conventions and how to resolve these differences, after we learn how to identify which calling conventions are in use for a specific exported function. Near the end we'll also take a look at the different tools and utilities available to help accomplish these tasks easily.

We'll also write some dummy DLLs and analyze them to get a firm grasp and clear understanding of the problems and the way we solve them.

This is something that happens at a very low level, say at the time your C or C++ code is compiled to native CPU instructions, and comes into the picture then. A good understanding of calling conventions can help you write better code, debug the code better, and finally enable you to use a source level debugger like Softice. But this is not something we'd like to get into. So we'll limit the discussion to our purpose -- how calling conventions affect functions exported from a DLL, and what benefit we get if we can find the calling convention used for a specific export.

When a function is invoked, whether it resides in a DLL or in your own source files, it receives its arguments/parameters through the stack. These arguments/parameters are set up for the called function by the calling function. We can use these parameters, but how does this frame of stack values get cleaned up? Of course it'll be done by either the caller or the callee. And this, finally, is specified by the specific calling convention used. You may have seen a few in action, but never took the time to understand what they actually are. Never mind, below is a list of keywords that will refresh your memory.

__pascal, __fortran, and __syscall,  __cdecl, __stdcall, __fastcall

Out of these PASCAL, FORTRAN and SYSCALL are obsolete and their use, even if supported, is discouraged. When we use the CDECL calling convention, the calling function is responsible for cleaning up the stack that it set up before the call was made, whereas in STDCALL and FASTCALL, the callee or the called function is responsible for the cleanup of stack.

Apart from the listed ones THISCALL ( thiscall ) is another convention which is the default calling convention used by C++ member functions that do not use variable arguments. Under thiscall, the callee cleans the stack, and as this is not a keyword, unlike other calling conventions, you can't explicitly specify it. To put it simply, you can assume it to be used whenever a class member function is called which uses the this pointer (that's right, static members don't use thiscall). Essentially, the thiscall convention applies to C++ only.

So now that you understand what these __cdecl and __stdcall mean, let's move on to identifying the convention used in functions exported to a DLL. For the purpose of demonstration you may use the DemoDLL project (as well as the compiled binaries) provided as a download with the article.

DLL Conventions: Issues and Solutions - Part II - How to identify exported symbols
(Page 2 of 4 )

You can use any tool that dumps information about a DLL import/export, such as tdump from Borland, or Dependency Walker (which comes bundled with Visual Studio). I use Depends.exe most of the time; I used it to view the exports from the MSVC as well as the BCB DLL. Below are the screen shots for the same. Figure 1 shows the exports for the DLL built with Borland C++ builder and Figure 2 shows the exports for the DLL built with Visual C++ Compiler.

Fig 1: Exports from the BCB built DemoDLL.dll

Fig 2: Exports from the MSVC built DemoDLL.dll

In order to see the true linker names you should turn the undecorated feature to off, as shown in Figure 3.

Fig 3: Turn undercoating OFF in order to view true linker names

DLL Conventions: Issues and Solutions - Part II - Solving the problem
(Page 3 of 4 )

From the first two figures above, you can clearly make out the different linker names generated by the Visual C++ Compiler and the Borland C++ Compiler. From the first figure you may see the symbols that are exported by the Borland built DLL, and they're exactly the same ones that any application developed in Borland Builder that uses this DLL would expect in the import library at link time.

But why are they different even when they're built from the same source? Yes, you guessed right! It's due to the differences in name mangling/name decoration techniques that these different compilers follow. To start with the analysis I'll first state a few facts on which our analysis will be based. The facts are:

• The default calling convention for Visual C++ 6.0 is __cdecl, that is, if you don't specify or omit the calling convention (see figure 4 below) it is assumed to be __cdecl.

Fig 4: Where to look for calling convention

For instance, __declspec(dllexport) int MyFoo(void); uses __stdcall.

• VC++ compiler always decorates function names that use the __stdcall calling convention. In specific cases where a module definition file has explicitly been used while building the DLL, the functions with __stdcall calling convention are not decorated or mangled. Essentially, one has less work to do if a .def file is used while building the DLL.
  
  If you wish to provide the operability between different compilers, you can consider using a .def file to explicitly define the exported symbols. In figure 2 you can observe that the FooSTDCALL function name has been decorated to _FooSTDCALL@0. It can also be observed that the last number is usually the number of bytes that the caller have to push on the stack before calling this DLL method; but it's just an observation, and I'm not sure if it is the same in all cases. Had the DLL been built using a module definition file (.def), the exported name will appear to be FooSTDCALL without the leading underscore and the trailing @XX.

• Borland C++ builder expects __cdecl function names to be decorated by prep ending an underscore.

Now that we have seen the problem in action, let's talk about the solution. The aim is to be able to use the DLL created in VC++ (with or without a .def file), which is accompanied by the header file for using that DLL in Borland C++ builder.

Essentially, based on the observations you made, you can now identify the calling convention used (by using Depends.exe and the DLL header file) and whether a .def file was used during the DLL build process (that is, the __stdcall functions are not decorated). Based on this, your approach to use the DLL in Borland Builder will vary. To proceed further, let's make a table of a single function on which we'll base our analysis.

Table 1: Comparison of the generated/expected linker names

DLL Conventions: Issues and Solutions - Part II - Use IMPLIB
(Page 4 of 4 )

Next, if you find that all exported symbols use the __stdcall and a .def has indeed been used while building the VC++ DLL, you may directly use the IMPLIB utility to generate a Borland (OMF) compatible import library. We can now simply add the generated .lib file along with the DLL header file to the Borland builder project, and use the imported functions as we wish. To use IMPLIB, follow the syntax at command prompt as under:

:>IMPLIB <libfile name to generate> <VC++ DLL name>

But if any function uses the __cdecl calling convention, or the .def was not used while building the VC++ DLL, then you're bound to do some more work before using the DLL in Borland Builder project. This is due to the fact that __cdecl names are decorated and, in the absence of the .def file, the __stdcall names are also decorated.

Here you first need to resolve the naming issues manually, and then create the import library from this modified information. To do so, you'll need the IMPDEF utility and build. There are command line tools that ship with Borland C++ builder that ease this task for you, but still you'll need to do some manual work. If we see this process in two steps, the first one would be to generate a .def file from the VC++ DLL (using the IMPDEF utility) and edit it, and the second one would be to build the .lib file using the above mentioned IMPLIB utility. The syntax to use for IMPDEF is as shown below:

:> IMPDEF <def file name> <VC++ DLL File name>

In order to make the names compatible you'll need to edit them so that they're the same as expected by the Borland Linker. After you've generated a def file you can edit it using notepad or any standard editor. Just provide an alias name for each export/linker name you wish to modify. For example the line containing FooCDECL will be changed to _FooCDECL = FooCDECL.You may ignore the @1 @2 numbers at the end of lines. Here the name at the left of = sign is the name expected by the Borland Linker whereas the name at the right is the true linker name exported by the VC++ Compiler. By providing the alias manually you've just made them equivalent.

And finally to create an import library from this aliased definition file you again use the IMPLIB utility as shown below:

:> IMPLIB <lib file name to generate> <aliased .def file name>

When you add this generated lib to your Borland project, all unresolved externals will be found by the linker, and it will be happy to link them in your code.

This article explained the different calling conventions in use with the most prominent compilers available today, and the differences between them. We also observed the differences due to non-availability of standards for the naming conventions for symbols exported from DLLs, and finally learned how to use available tools to analyze and bridge these gaps in order to use the DLLs built with VC++ 6.0 in Borland C++ Builder projects.

Worth mentioning is the COFF2OMF utility, which does most of this work you. It is available with the command line tools in the Borland Bin directory. But again, there is no guarantee that it will do all the work for you. The points that you learned and observed here will surely help you handle similar problems with DLLs. In case you have any suggestions or questions, feel free to write to me at Digvijay.Chauhan@gmail.com.