I have a code in fortran which I compiled and then sent to run as a pbs job in a
supercomputer. I want to modify the source code and compile it again while keeping
the already running program. My question is what happens if I modify the source code
and make a new executable if I have a pbs job running with a different executable.
Would the results from the original executable be modified?
Thanks.
If the replacement of the executable happens atomically with the rename() system call, then it should work such that the running program keeps using the old executable. OTOH if the replacement happens by writing bits and pieces into the existing executable, the running executable might be affected.
FWIW, this has nothing to do with fortran, pbs, nor torque, but rather to do with POSIX filesystem semantics. POSIX filesystems are essentially reference-counted object stores, with delete-on-last-close. When you replace the executable with rename(), the reference count of the old executable is reduced by one since the directory no longer has a reference to it. However, the executing program still holds a reference and can keep using it. The new executable is a separate object with it's own reference count.
Related
At the end of the compilation process, the program is in a .exe file in machine code. So shouldn't the machine be able to run it without having to install something like MS Visual Studio C++? Basically, I am making a program with mingw and want to share it with someone else. I do not understand why I can not just send them the .exe file. Clarification will be appreciated.
C++ compiles your code to machine code. If your program is self-contained, that is all you need. However, more complex running programs often relies on additional compiled code, which is made available to your program through a library.
Generally, libraries come in two "flavors" - static and dynamic. Static libraries are "baked into" your compiled code. This is not ideal, because multiple programs include identical code, leading to code duplication. Dynamic libraries, on the other hand, are shared among all programs using them, leading to more efficient use of space.
Installing runtime adds dynamic libraries for use by all programs compiled with C++.
Your program likely calls many functions from the standard library that you didn't write yourself. You need the runtime libraries for that. Your code probably also needs code run before main to setup the basic environment that's expected for a C++ program - the runtime libs do that for you. Also after main ends, various cleanup needs to happen according to the standard (and your program probably also depends on this) and the compilers runtime libraries take care of this.
Your code does not exist in a vacuum (it can, but then it's no longer a standard hosted C++ program). It depends on and relies on the standard runtime libs to provide the environment the C++ standard says you can expect.
I've implemented a reasonably good optimizing compiler (for a toy language), and have come across a rather puzzling scenario. I can take an input file and produce assembly from it, so in that respect the "compiler" is finished. If I take that assembly file assemble it with NASM and link it to my runtime support library with G++ (the runtime needs libstdc++), I get a working executable, with no errors. However, I'd like to be able to compile to an executable in one command, so I added some calls to system to my compiler, passing the EXACT SAME COMMANDS as I was using in bash. When I run the compiler, it seems to assembly properly, but the linking step (again, using g++) fails with an undefined reference to main. Confused, I attempted to link manually (without reassembling, so I was using the object file produced by the NASM run with system, and received the same error. If I reassemble using the new compiler's assembly output, I have no problems, which has led be to believe that NASM is the problem. Like I said, the commands are exactly the same (I literally copied and pasted them just to make sure after the first time). Is this environment variables or something? What's going on?
EDIT:
I manually assembled an object file, again using the same command as the one in the compiler, and I did a vim diff between the two. The compiler-generated one seems to only contain the ELF header.
EDIT 2:
A screenshot of the diff
EDIT 3:
I tried using system to call a Perl script that in turn would call NASM and G++, still no luck.
Fixed it! It was the file not being flushed due to a race condition. Thanks for all the help, much appreciated.
I was wondering why executables (written in c++) have to be recompiled from sources on every linux machine, even if the machines are software and hardware the same?
I had a search engine, written in c++, and I have to recompile it every time I want to move it on a new linux machine to make it work.
Any ideas?
If you are asking why an executable compiled on Linux-X won't run on Linux-Y, then the reason is probably that dynamic libraries (.so) are missing or could not be found.
EDIT: oh sorry, looks like I didn't read your question well enough. Removed the sarcasm.
It normally shouldn't be necessary to recompile. Many applications are distributed as executables and they work fine.
What errors do you get when you just copy the executable and run it on a different machine?
Maybe the problem is with the way you're copying the executable, it might be corrupting it.
The recompilation is ensuring you get optimal performance on your machine, because each time the configuration script is running to find dependencies and settings. This also ensures the openness of software as its source is always available and is modifiable by an appropriate agents - that is us.
This is not necessarily the case. Consider how Ubuntu packages are installed for example - https://askubuntu.com/questions/162477/how-are-packages-actually-installed-via-apt-get-install
Apparently these are not compiled from source on the destination machine, but installed as pre-built binaries. Having said that, it is generally a good idea to build binaries from source on the machine that they will run on as you will avoid potential problems such having incompatible shared libraries (such as libc) which can occur when building something on Linux X and running on Linux Y.
I want to compile a bunch of C++ files into raw machine code and the run it with a platform-dependent starter written in C. Something like
fread(buffer, 1, len, file);
a=((*int(*)(int))buffer)(b);
How can I tell g++ to output raw code?
Will function calls work? How can I make it work?
I think the calling conventions of Linux and Windows differ. Is this a problem? How can I solve it?
EDIT: I know that PE and ELF prevent the DIRECT starting of the executable. But that's what I have the starter for.
There is one (relatively) simple way of achieving some of this, and that's called "position independent code". See your compiler documentation for this.
Meaning you can compile some sources into a binary which will execute no matter where in the address space you place it. If you have such a piece of x86 binary code in a file and mmap() it (or the Windows equivalent) it is possible to invoke it from both Linux and Windows.
Limitations already mentioned are of course still present - namely, the binary code must restrict itself to using a calling convention that's identical on both platforms / can be represented on both platforms (for 32bit x86, that'd be passing args on the stack and returning values in EAX), and of course the code must be fully self-contained - no DLL function calls as resolving these is system dependent, no system calls either.
I.e.:
You need position-independent code
You must create self-contained code without any external dependencies
You must extract the machine code from the object file.
Then mmap() that file, initialize a function pointer, and (*myblob)(someArgs) may do.
If you're using gcc, the -ffreestanding -nostdinc -fPIC options should give you most of what you want regarding the first two, then use objdump to extract the binary blob from the ELF object file afterwards.
Theoretically, some of this is achievable. However there are so many gotchas along the way that it's not really a practical solution for anything.
System call formats are totally incompatible
DEP will prevent data executing as code
Memory layouts are different
You need to effectively dynamically 'relink' the code before you can run it.
.. and so forth...
The same executable cannot be run on both Windows and Linux.
You write your code platform independently (STL, Boost & Qt can help with this), then compile in G++ on Linux to output a linux-binary, and similarly on a compiler on the windows platform.
EDIT: Also, perhaps these two posts might help you:
One
Two
Why don't you take a look at wine? It's for using windows executables on Linux. Another solution for that is using Java or .NET bytecode.
You can run .NET executables on Linux (requires mono runtime)
Also have a look at Agner's objconv (disassembling, converting PE executable to ELF etc.)
http://www.agner.org/optimize/#objconv
Someone actually figured this out. It’s called αcτµαlly pδrταblε εxεcµταblε (APE) and you use the Cosmopolitan C library. The gist is that there’s a way to cause Windows PE executable headers to be ignored and treated as a shell script. Same goes for MacOS allowing you to define a single executable. Additionally, they also figured out how to smuggle ZIP into it so that it can incrementally compress the various sections of the file / decompress on run.
https://justine.lol/ape.html
https://github.com/jart/cosmopolitan
Example of a single identical Lua binary running on Linux and Windows:
https://ahgamut.github.io/2021/02/27/ape-cosmo/
Doing such a thing would be rather complicated. It isn't just a matter of the cpu commands being issued, the compiler has dependencies on many libraries that will be linked into the code. Those libraries will have to match at run-time or it won't work.
For example, the STL library is a series of templates and library functions. The compiler will inline some constructs and call the library for others. It'd have to be the exact same library to work.
Now, in theory you could avoid using any library and just write in fundamentals, but even there the compiler may make assumptions about how they work, what type of data alignment is involved, calling convention, etc.
Don't get me wrong, it can work. Look at the WINE project and other native drivers from windows being used on Linux. I'm just saying it isn't something you can quickly and easily do.
Far better would be to recompile on each platform.
That is achievable only if you have WINE available on your Linux system. Otherwise, the difference in the executable file format will prevent you from running Windows code on Linux.
Is there any particular reason that the linking phase when building a project with distcc is done locally rather than sent off to other computers to be done like compiling is? Reading the distcc whitepages didn't give a clear answer but I'm guessing that the time spent linking object files is not very significant compared to compilation. Any thoughts?
The way that distcc works is by locally preprocessing the input files until a single file translation unit is created. That file is then sent over the network and compiled. At that stage the remote distcc server only needs a compiler, it does not even need the header files for the project. The output of the compilation is then moved back to the client and stored locally as an object file. Note that this means that not only linking, but also preprocessing is performed locally. That division of work is common to other build tools, like ccache (preprocessing is always performed, then it tries to resolve the input with previously cached results and if succeeds returns the binary without recompiling).
If you were to implement a distributed linker, you would have to either ensure that all hosts in the network have the exact same configuration, or else you would have to send all required inputs for the operation in one batch. That would imply that distributed compilation would produce a set of object files, and all those object files would have to be pushed over the network for a remote system to link and return the linked file. Note that this might require system libraries that a referred and present in the linker path, but not present in the linker command line, so a 'pre-link' would have to determine what set of libraries are actually required to be sent. Even if possible this would require the local system to guess/calculate all real dependencies and send them with a great impact in network traffic and might actually slow down the process, as the cost of sending might be greater than the cost of linking --if the cost of getting the dependencies is not itself almost as expensive as linking.
The project I am currently working on has a single statically linked executable of over 100M. The static libraries range in size but if a distributed system would consider that the final executable was to be linked remotely it would require probably three to five times as much network traffic as the final executable (templates, inlines... all these appear in all translation units that include them, so there would be multiple copies flying around the network).
Linking, almost by definition, requires that you have all of the object files in one place. Since distcc drops the object files on the computer that invoked distcc in the first place, the local machine is the best place to perform the link as the objects are already present.
In addition, remote linking would become particularly hairy once you start throwing libraries into the mix. If the remote machine linked your program against its local version of a library, you've opened up potential problems when the linked binary is returned to the local machine.
The reason that compliation can be sent to other machines is that each source file is compiled independently of others. In simple terms, for each .c input file there is a .o output file from the compilation step. This can be done on as many different machines as you like.
On the other hand, linking collects all the .o files and creates one output binary. There isn't really much to distribute there.