I have studied many articles to understand whether Compiler generates Object Code or Assembly Code and there is conflict, even in stack overflow. Some says Compiler generates Object Code while other says Compiler generates Assembly Code which is then converted to Object Code by Assembler. Both answers has up-votes. Is there any clarification or justification for this.
Ultimately, the compiler has to somehow produce object files with the code that will end up in the application, and the linker combines the code from object files and libraries (which are just collections of object files) to produce the application. So it's correct to say that the compiler produces object files and the linker combines them.
On the other hand, there are various ways that the compiler can produce the object files. One way is to directly generate object files. Another way is to generate assembler code and run the assembler to produce the object files. That introduces some flexibility, because the compiler doesn't have to know the details of how object files are laid out; the assembler does that. Yet another way is to generate C code and run the C compiler (which could, in turn generate assembler code and run the assembler) on that to produce object files. That's how cfront worked back in the olden days of C++. It's also how some modern compiler front-ends work. For example, Edison Design Group sells a C++ front-end that vendors can hook up to their own back-end for code generation. They also provide a version that generates C code, for use during compiler development when the back-end isn't yet working. Typically in these cases, the compiler will have a switch to keep the C or assembler file around, so you can examine that to see what's going on.
Related
I have a C++ program that I want to compile to assembly, and then assembler will compile it to machine code.
Now, as far as I know, in order to transform assembly code to machine code the assembler needs some kind of table to map assembly instructions to the actual machine instructions.
Which table will the assembler use? Is there a chance that my C++ program won't run on all CPUs, because CPUs use different tables which means that the same machine code will do different things on different CPUs?
The assembler assembles for whatever architecture it has been told to/programed to assemble for. As the assembly language for each instruction set architecture (ISA) differs, you can only assemble an assembly program written for one architecture for that same architecture. It is generally not possible to accidentally or intentionally assemble the program for the wrong architecture.
When you use a compiler, the compiler invokes the correct assembler with the correct flags to assemble the assembly code it generated for the architecture you told it to compile for. The resulting program will only run on processors of the ISA your have compiled it for. If you want the program to run on processors of a different ISA, you have to compile it for that other ISA.
If your program is poorly written, it is possible that it won't compile or work when compiled for other architectures than the one(s) you developed it for. Such a program is called an unportable program. However, unless you do weird things or make assumptions about properties of the architecture you are programming for, this is unlikely to happen.
In general what is call assembly is roughly a human readable (text) form of machine code (binary).
As franji1 said in a comment, in general compilers emit an intermediate abstract machine code from the source. And this kind of code can easily (it is intended to) be translated to assembly/machine code.
I have a C++ program that I want to compile to assembly, and then
assembler will compile it to machine code.
This is what a compiler is designed to. Compiler is somehow misleading. Compiler can be the "compiler phase" or "compiler toolchain". compiler phase is the one that translate your source code to the intermediate abstract form, that then needs to be translated to target assembly/machine code by the assembler. Compilation is commonly what denotes the whole process from source code to executable machine code.
Now, as far as I know, in order to transform assembly code to machine
code the assembler needs some kind of table to map assembly
instructions to the actual machine instructions.
Roughly yes. This is what a document like Instruction Set Reference Manual is for: describing how textual form must be translated to byte form.
Which table will the assembler use?
See document...
Is there a chance that my C++ program won't run on all CPUs, because
CPUs use different tables which means that the same machine code will
do different things on different CPUs?
You have to generate a byte form of your program for each platform (machine/os). A compiler is designed to generate a machine code for a given platform that realizes exactly what your source code specifies. This is why compilers exist, to free you from writing program in assembly (that is very hard to do).
This question already has answers here:
Does a compiler always produce an assembly code?
(4 answers)
What do C and Assembler actually compile to? [closed]
(11 answers)
Closed 2 years ago.
I have started learning C++, and I have learned that a compiler turns source code from a program into machine code through compilation.
However, I've learned that C++ compilers actually translate the source code into Assembly as an interim step before translating the Assembly code into machine code. What is the purpose of this step?
Why don`t they translate it directly into the machine code?
First of all: There is no need to write an intermediate assembly language representation. Every compiler vendor is free to emit machine code directly.
But there are a lot of good reasons to "write" an intermediate assembly and pass it to an assembler to generate the final executable file. Important is, that there is no need to really write a file to some kind of media, but the output can directly piped to the assembler itself.
Some of the reasons why vendors are using intermediate assembly language:
The assembler is already available and "knows" how to generate some executable file formats ( elf for example ).
Some tasks can be postponed until assembly level is reached. Resolving jump targets for example. This is possible because the intermediate assembly is often not only 1:1 representation but some kind of "macro-assembler" which can do a lot more than simply creating bits from mnomics.
the assembler level is followed by executing the linker. This must also be done if a compiler directly wants to create executable file formats. A lot of duplicated jobs if this must be coded again. As an example all the relocation of before "unknown addresses" must be done on the way to an executable file. Simply use the assembler/linker and the job is done.
The intermediate assembly is always useful for debugging purpose. So there is a more or less hard requirement to be able to do this intermediate step, even if it can be omitted if no debug output is requested from the user.
I believe there are are lot more...
The bad side is:
"writing" a text representation and parsing the program from the text takes longer as directly passing the information to the linker.
Usually, compilers invoke the assembler (and the linker, or the archiver) on your behalf unless you ask it to do otherwise, because it is convenient.
But separating the distinct steps is useful because it allows you to swap the assembler (and linker and archiver) for another if you so desire or need to. And conversely, this assembler may potentially be used with other compilers.
The separation is also useful because assemblers already existed before the compiler did. By using a pre-existing assembler, there is no need to re-implement the machine code translation. This is still potentially relevant because occasionally there will be a need to boot-strap a new CPU architecture.
REQUIREMENT: For a certain project we have unique requirement. The application supports an expression language that allows the user to define their own complex expressions that can be evaluated at run time (many hundred times a second) and they need to be executed at machine level for performance.
WORKING: Our expression parser translates the script into corresponding assembly language routine perfectly. We checked it by statically linking the object files generated with our C test program and they produce correct result.
Since the client can change the script anytime, our program (at run time) detects the change, calls the parser which generates the corresponding assembly routine. We then call the assembler from back end to create the object code.
PROBLEM
How can we call this assembly routine dynamically from the C++ program
(Loader)?
We are not supposed to call the C++ compiler to link it with the loader because the loader already would have other subroutines running and we cannot take the loader off, recompile and then execute the new loader program.
I tried searching for a solution online but every time the results are littered with .NET assembly dynamic calling. Our app has nothing to do with .NET.
First, the "generated plugin" approach (on Linux; my answer focuses on Linux but could be adapted to Windows with some effort; you could use many-platform frameworks like Qt or POCO or Glib from GTK; then all wrap plugin loading abilities à la dlopen with a common API that you could use on Windows, on Linux, on MacOSX, on Android) :
generate C (or assembly) code in some file /tmp/generated01.c (you might even generate C++ code using standard C++ containers, but its compilation would be significantly slower; beware of name mangling so emit and use extern "C" functions; read the C++ dlopen mini HowTo). See this answer explaining why generating C is worthwhile (and could be better, and more portable, than generating assembler code).
run (using fork+execve+waitpid, or simply system) a compilation of that generated file into a shared object /tmp/genenerated01.so by running gcc -fPIC -Wall -O /tmp/generated01.c -shared -o /tmp/generated01.so command; you practically need to get position-independent code, hence the -fPIC flag. If using dlopen on your generated assembler code you'll need to improve your assembler generator to emit PIC code.
dlopen that new /tmp/generated01.so (so use the dynamic linker), see dlopen(3); you could even remove the now useless generated C file /tmp/generated01.c
dlsym the relevant symbols to get function pointers to the generated code, see dlsym(3); your application would simply call the generated code using these function pointers.
when you are sure that you don't need any functions from it and that no call frame uses it, you could dlclose that shared object library (but you might accept to leak some address space by not calling dlclose at all)
The above approach is worthwhile and can be used a big lot of times (my manydl.c demonstrates that you could dlopen a million different shared objects), and is practically even compatible (even when emitting C code!) with an interactive Read-Eval-Print-Loop -on most current desktops and laptops and servers-, since most of the time the generated /tmp/generated01.c would be quite small (e.g. a few hundred lines at most) to be very quickly generated and compiled (by gcc, etc...). I am even using this in MELT for its REPL mode. On Linux this plugin approach generally requires to link the main application with -rdynamic (so that dlopen-ed plugins can reference and call functions from the main application).
Then, other approaches could be to use some Just-In-Time compilation library, like
GNU lightning (which emits slow machine code very quickly - so very short JIT emission time, but the generated code is running slowly since it is very unoptimized)
asmjit; it is x86-64 specific, and enables you to generate individual x86-64 machine instructions
GNU libjit is available for several platforms, and offer an "interpreter" mode for other platforms
LLVM (part of Clang/LLVM compiler, usable as a JIT library)
GCCJIT (a new JIT library front-end to GCC)
Grossly speaking, the first elements of that list are able to emit JIT machine code fairly quickly, but that code won't run as fast as compiling with gcc -fPIC -O1 or -O2 the equivalent generated C code (but would run typically 2x to 5x slower!); the last two elements (LLVM & GCCJIT) are compiler based: so they are able to optimize and emit efficient code, at the expense of slower JIT code emission. All the JIT libraries are able (like dlsym does for plugins) to give function pointers to newly JIT-constructed functions.
Notice that there is a trade-off to be made: some techniques are able to generate quickly some machine code, if you accept that generated code to later run a bit slowly; other techniques (notably GCCJIT or LLVM) are spending time to optimize the generated machine code, so takes more time to emit the machine code, but that code would later run quickly. You should not expect both (small generation time, quick execution time), since there is no such thing as a free lunch.
I believe that generating manually some assembler code is practically not worthwhile. You won't be able to generate very optimized code (because optimization is a very difficult art, and both GCC and Clang have millions of source line code for optimization passes), unless you spend many years of work for that. Using some JIT library is easier, and "compiling" to C or C++ is also quite easy (you leave the burden of optimization to the C compiler you are calling).
You could also consider rewriting your application into some language with homoiconicity and metaprogramming abilities (e.g. multi-stage programming), such as Common Lisp (and many others, e.g. those providing eval). Its SBCL implementation is always emitting machine code...
You could also embed an interpreter like Lua -perhaps even LuaJit- or Guile in your application. The main advantage of embedding an existing language is that there are resources (books, modules, ...) and community of people knowing them (designing a good language is difficult!). Also, the embedded interpreter library is well designed and probably well debugged (since used a lot), and some of them are fast enough (since using bytecode techniques).
As the comments already suggest, LoadLibrary (Windows) and dlopen (Linux/POSIX) are by far the easiest solution. These are specifically intended to dynamically load code. Equally important, they both allow unloading as well, and there are functions to then get a function entry point by name.
You can dynamically do it. I will take linux case as an example. Since your parser working fine and generates machine code, you should be able to generate .so (for linux) or .dll for windows.
Next, load the library as
handle = dlopen(so_file_name, RTLD_LAZY);
Next get function pointer
func = dlsym(handle, "function_name");
Then you should be able to execute it as func()
One thing you need to experiment (in case you do not get desired result) is close and open the so file or dll file (you need to do only if required, else it may reduce performance)
It sounds like you can generate the proper byte code. So you could just ensure that you generate position independent code, write it into an executable piece of memory, and then call or create thread upon the code. The simplest way would just be to cast the pointer to the base of the memory you wrote the code into as a function pointer, and then call it.
If you write your bytecode to avoid referencing different sections, and instead reference offsets from its loaded base, 'loading' the code is as easy as writing it to executable memory. You could do a call/pop/jmp to find the base of the code once it begins executing.
Conversely, and probably the easiest solution, would be to just write the code out as function expecting arguments, that way you could pass the code's base and any other arguments to it, as you would with any other function, as long as you use the proper typedef for your function pointer, and the generated assembly handles the arguments properly. As long as you avoid creating absolute jumps or data references to absolute addresses, you shouldn't have any issue.
too late but I think it would help someone else.
in case you want to dynamically execute a piece of code, you can create an interpreter for this.
compile your expressions into some byte code then write the interpreter for executing this.
here is a tutorial about writing interpreters, but in python.
https://ruslanspivak.com/lsbasi-part1/
you can write it using c/c++
How to see the added code in C++ by the compiler?
E.g., we know that
when an object of some class goes out of scope, the destructor for that object is called, but how do you see the specific code that does the destructor call? Is that code still written in C++?
Its compiler-dependent and in assembly language. For example, with the Microsoft compiler, compiling with /FAsc will generate a .cod file for each object file containing the assembly code along with the original C++ lines as comments. It will show the calls to constructors/destructors as well.
There's not necessarily any "code" that gets added. C++ is pretty clear on when such things happen, and for the compiler, making a new object clearly means calling its constructor -- no additional "code" anywhere.
You're right, however, things like calls to the constructor or destructor must end up somewhere in the assembly -- but there's absolutely no guarantee that having a look at the assembly reveals much more than what you'd have known without. C++ compilers are pretty mature in these aspects, and inline a lot of things in cases where that makes sense, making the same code look different in different places.
The closest thing you'll get is adding debug symbols to your build and using a debugger to get a call graph -- that will make sure that you notice when what you see as code gets called.
You can add flags to the compile command which will let you see the file in various stages of operations done by the compiler. For e.g., the -S flag will produce a file which would have had the preprocessor done and the initial compilation done, but before the assembler runs. However, this code will not be written in C++.
Does a C++ compiler generate machine code via assembly language code (i.e., c++ compiler first converts C++ code to assembly language code and then uses assembly language compiler to convert it to machine code), or is assembly language output generation just an option for reference or debugging purposes?
It doesn't have to, but most do it anyway, as the same assembler (program) can be used for the output of the C/C++/whatever-to-assembler compiler.
g++ for example generates assembler code first (you can see the generated assembler using the -S switch).
MSVC does it too (/FAs).
They used to, a long time ago, although that was typical only for C compilers. The first one I used worked that way, a long time ago. Not unusual for ones that generated code for unusual hardware and operating systems, they saved the cost of writing the object file generator and leveraged existing linkers.
Modern compilers don't bother with the extra step of generating machine code as text and running an assembler afterward, they generate the machine code directly in binary form. The compile speed advantage is fairly significant, text isn't cheap. The option to generate it in textual form from binary is pretty simple to implement.