We are developing C++ applications for Windows, Mac and Linux. The application have many binary resources, and for some reason we need to pack them within executable binary, instead of laying at directories (or Apple App bundle).
At current, we use a script to convert these resources into C++ array constants, then compile and link them. However this approach have so many deficiencies:
You have to compile the resource source code, it takes time and is unnecessary in essential.
The resource source codes would be parsed by IDE. As they are large, code analytic is greatly slowed down.
MSVC have limit on source code size, so large resources (several MB) must be separated into many parts then concatenated at run-time.
After some study, I found some solutions:
In Windows, I can use .rc files and related WinAPI.
In Linux, I can directly convert arbitrary binary file into obj file via objcopy.
However, there are still some questions remaining:
The use of WinAPI to fetch resources needs many functions to access one resource. Is there any simpler ways in Windows?
How to do it in Mac?
A quite common trick, most notably used for self-extracting archives or scripting language to executable compilers, is to append the resources at the end of the executable file.
Windows:
copy app.exe+all-resources app-with-resources.exe
Linux:
cp executable executable-with-resources
cat all-resources >>executable-with-resources
Then you can read your own executable using fopen(argv[0]) for example.
In order to jump at the correct position, i.e. beginning of resources, a possible solution is to store the size of the executable without resources as the last word of the file.
FILE* fp = fopen(argv[0], "rb");
fseek(fp, -sizeof(int), SEEK_END);
int beginResourcesOffset;
fread(&beginResourcesOffset, 1, sizeof(int), fp);
fseek(fp, beginResourcesOffset, SEEK_SET);
Be careful with this solution though, anti-virus on windows sometimes don't like it. There probably are better solutions.
Related
My question is, when you compile your c++ program, why is it all put into one exe file? The file could become to large. Would you use dll libraries to shrink the size, or are there other files you can make? I just want to know how to make a program that uses separate files to run.
(EDIT) I just don't want it all in a single file. Files could become too large eventually for the computer to handle it, right? There must be a way to separate the files. Like in java, everything is in a class file, which just seems easier and more efficient. Some drives like FAT32 can't have a file bigger than 4 gigabytes, so they need a more broken down program. I looked at my game called portal, its exe is 100KB and it has about 100 dll files!
To answer your question, yes. You absolutely can split your program into separate DLL files if you'd like.
I've seen some developers compile utility functions into a separate common DLL files which can be included in other projects as references. This way its objects and methods can be be called from it.
In hindsight, compiled code is relatively small. Binary data is really what consumes the most space: videos, images, models, sounds, etc. Although it is possible and common for smaller programs to pack these resources directly into the executable, it generally isn't a good idea for many obvious reasons.
Finally, large executables aren't a huge problem with today's technology. For smaller programs, I wouldn't sweat it. It's more about the design development the larger the project gets.
Too large for what? If it was due to storage space restrictions, splitting it into multiple files wouldn't buy you anything. Unless you are somehow overflowing the maximum size for a file on a platform (like a 2GB limit on some 32-bit platforms), which seems very unlikely, you are probably worrying about a non-issue.
You can reduce the size of the generated executable by turning off debug options in the compiler, "stripping" it on various platforms, setting optimization settings to optimize for code size rather than execution speed, etc.
The way to split an exe into several files (on Windows) is, as the OP suggested, to use dlls.
The commenters are correct that this will not actually take any less space on disk, nor save any memory, but there are other reasons to split an application into multiple files. For example, to share code (a single dll can be used by several applications), or to help an application load more quickly (only load the dll when it is actually needed).
I want to create a cross-platform installer, in c++. It can be any compression type, eg zip or gzip, embedded inside the program itself like an average installer. I don't want to create many changes on different platforms, linux and windows. How do I embed and extract files to a c++ program, cross-platform?
C++ is a poor choice for a cross-platform installer, because there's no such thing as cross-platform machine code.
C++ code can be extremely portable, but it needs to be compiled for each platform, and then you get a distinct output executable for each platform.
If you want to build installers for many platforms from a single source file, you can use C++. But if you want to build ONE installer that works on many platforms, you'll need to use an interpreted or JIT-compiled language with runtime support available on all your targets. Of those, the only one likely to already be installed on a majority of computers of each platform is Java.
Ok, assuming that you're building many single-platform installers from machine code, this is what is needed:
You need to get the compressed code into the program. You want to do this in a way that doesn't affect the load time badly, nor cause compilation to take a few months. So using an initialized global array is a bad idea.
One way is to link your data in as an additional section. There are tools to help with that, e.g. Binary to COFF converter, I've seen an ELF version as well, maybe this. But this might still cause the runtime library to try to lead the entire file into memory before execution begins.
Another way is to use platform-specific resource APIs. This is efficient, but platform specific.
The most straightforward solution is to simply append the compressed archive to your executable, then append eight more bytes with the file offset where the compressed archive begins. Unpacking is then as simple as opening the executable in read-only mode, fseek(-8, SEEK_END), reading the correct offset, then seeking to the beginning of the compressed data and passing that stream to your decompressor.
Of course, now I find a website listing pretty much the same methods.
And here's a program which implements the last option, with additional ability to store multiple files. I wouldn't recommend doing that, let the compression library take care of storing the file metadata.
The only way I know of to portably embed data (strings or raw, binary
data) in a C++ program is to convert it into a data table, then compile
that. For raw data, this would look something like:
unsigned char data[] =
{
// raw data here.
};
It should be fairly trivial to write a small program which reads your
binary data, and writes it out as a C++ table, like the above. Compile
it, and link it into your program, and there you are.
Use zlib.
Have your packing program generate a list of exe's in the program. i,e.
unsigned char x86_windows_version[] = { 0xff,...,0xff};
unsigned char arm_linux_version[] = { 0xff,...,0xff};
unsigned char* binary_files[MAX_BINARIES] = {x86_windows_version,arm_linux_version};
somewhere in your excitable:
enflate(x86_windows_version);
And thats about it. Look at the zlib docs for the parameters for enflate() and deflate() and that's about it.
It's a pattern used a lot on embedded platforms (that are not linux) mostly for string _tables and other image binaries. It should work for your needs.
How can I include my programs dependency DLLs inside the EXE file (so I only have to distribute that one file)? I am using C++ so I can't use ILMerge like I usually do for C#, but is there an easier way to automatically do this in Visual Studio?
I know this is possible (thats why installers work), I just need some help being pointed to the best way to this.
Thank you for your time.
There are many problems with this approach. For one example, see this post from REAL Software. Their “REALbasic” product used to do this and had problems including:
When writing the DLLs out at run-time, it would trigger anti-virus warnings.
Problems with machines where the user doesn’t have write permissions or is low on disk space.
Their attempt to fix the problem caused more problems, including crashes. Eventually they relented and now distribute DLLs side-by-side with apps.
If you really need a single-EXE deployment, and can’t use an installer for some reason, the reliable way is to static-link all dependencies. This assumes that you have the correct .libs (and not just .libs that link in the DLL).
There exist two options, both of which are far from ideal:
write a temporary file somewhere
load the DLL to memory "by hand", i.e. create a memory block, put DLL image to memory, then process relocations and external references.
The downside of the first approach is described above by Nate. Second approach is possible, but is complicated (requires deep knowledge of certain low-level things) and doesn't allow the DLL code to access DLL resources (this is obvious - there's no image of the DLL so the OS doesn't know where to take resources).
One more option usable in some scenarios: create a virtual disk whose contents are stored in your EXE file resources, and load the DLL from there. This is possible using our SolFS product (OS edition), but creation of the virtual disk itself requires use of kernel-mode drivers which must be written to disk before use.
Most installers use a zip file (or something similar) to hold whatever files are needed. When you run the installer, it decompresses the data and puts the individual files where needed (and typically adds registry entries, registers any COM controls it installed, etc.)
I am developing a library that uses one or more helper executable in the course of doing business. My current implementation requires that the user have the helper executable installed on the system in a known location. For the library to function properly the helper app must be in the correct location and be the correct version.
I would like to removed the requirement that the system be configured in the above manner.
Is there a way to bundle the helper executable in the library such that it could be unpacked at runtime, installed in a temporary directory, and used for the duration of one run? At the end of the run the temporary executable could be removed.
I have considered automatically generating an file containing an unsigned char array that contains the text of the executable. This would be done at compile time as part of the build process. At runtime this string would be written to a file thus creating the executable.
Would it be possible to do such a task without writing the executable to a disk (perhaps some sort of RAM disk)? I could envision certain virus scanners and other security software objecting to such an operation. Are there other concerns I should be worried about?
The library is being developed in C/C++ for cross platform use on Windows and Linux.
"A clever person solves a problem. A
wise person avoids it." — Albert Einstein
In the spirit of this quote I recommend that you simply bundle this executable along with the end-application.
Just my 2 cents.
You can use xxd to convert a binary file to a C header file.
$ echo -en "\001\002\005" > x.binary
$ xxd -i x.binary
unsigned char x_binary[] = {
0x01, 0x02, 0x05
};
unsigned int x_binary_len = 3;
xxd is pretty standard on *nix systems, and it's available on Windows with Cygwin or MinGW, or Vim includes it in the standard installer as well. This is an extremely cross-platform way to include binary data into compiled code.
Another approach is to use objcopy to append data on to the end of an executable -- IIRC you can obtain objcopy and use it for PEs on Windows.
One approach I like a little better than that is to just append raw data straight onto the end of your executable file. In the executable, you seek to the end of the file, and read in a number, indicating the size of the attached binary data. Then you seek backwards that many bytes, and fread that data and copy it out to the filesystem, where you could treat it as an executable file. This is incidentally the way that many, if not all, self-extracting executables are created.
If you append the binary data, it works with both Windows PE files and *nix ELF files -- neither of them read past the "limit" of the executable.
Of course, if you need to append multiple files, you can either append a tar/zip file to your exe, or you'll need a slightly more advance data structure to read what's been appended.
You'll also probably want to UPX your executables before you append them.
You might also be interested in the LZO library, which is reportedly one of the fastest-decompressing compression libraries. They have a MiniLZO library that you can use for a very lightweight decompressor. However, the LZO libraries are GPL licensed, so that might mean you can't include it in your source code unless your code is GPLed as well. On the other hand, there are commercial licenses available.
Slightly different approach than using an unsigned char* array is to put the entire executable binary as resource of the dll. At runtime, you can save the binary data as a local temp file and execute the app. I'm not sure if there is a way to execute an executable in memory, though.
For the library to function properly
the helper app must be in the correct
location
On Windows, would that be the Program Files directory or System32 directory?
This might be a problem. When an application is installed, particularly in a corporate environment, it usually happens in an context with administrative rights. On Vista and later with UAC enabled (the default), this is necessary to write to certain directories. And most Unix flavours have had sensible restrictions like that for as long as anyone can remember.
So if you try to do it at the time the host application calls into your library, that might not be in a context with sufficient rights to install the files, and so your library would put constraints on the host application.
(Another thing that will be ruled out is Registry changes, or config file updates on the various Unices, if the host application doesn't have the ability to elevate the process to an administrative level.)
Having said all that, you say you're considering unpacking the helpers into a temporary directory, so maybe this is all moot.
Qt has an excellent method of achieving this: QResource
"The Qt resource system is a platform-independent mechanism for storing binary files in the application's executable."
You don't say if you are currently using Qt, but you do say "C++ for cross platform use on Windows and Linux", so even if you aren't using it, you may want to consider starting.
There is a way in Windows to run an executable from within memory without writing it to disk. The problem is that due to modern security systems (DEP) this probably won't work on all systems and almost any anti-malware scanner will detect it and warn the user.
My advice is to simply package the executable into your distribution, it's certainly the most reliable way to achieve this.
Well, my first thought would be: what does this helper executable do that couldn't be done within your library's code itself, perhaps using a secondary thread if necessary. This might be something to consider.
But as for the actual question... If your "library" is actually bundled up as a dll (or even an exe) then at least Windows has relatively simpe support for embedding files within your library.
The resource mechanism that allows things like version information and icons to be embedded within executables can also allow arbitrary chunks of data. Since I don't know what development environment you're using, I can't say exactly how to do this. But roughly speaking, you'd need to create a custom resource with a type of "FILE" or something sensible like that and point it at the exe you want to embed.
Then, when you want to extract it, you would write something like
HRSRC hResource = FindResource(NULL, MAKEINTRESOURCE(IDR_MY_EMBEDDED_FILE), "FILE");
HGLOBAL hResourceData = LoadResource(NULL, hResource);
LPVOID pData = LockResource(hResourceData);
HANDLE hFile = CreateFile("DestinationPath\\Helper.exe", GENERIC_WRITE, 0, NULL, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL);
DWORD dwBytesWritten = 0;
WriteFile(hFile, pData, SizeofResource(NULL, hResource), &dwBytesWritten, NULL);
CloseHandle(hFile);
(filling in your own desired path, filename, and any appropriate error checking of course)
After that, the helper exe exists as a normal exe file and so you can execute it however you normally would.
For removing the file after use, you should investigate the flags for CreateFile, particularly FILE_FLAG_DELETE_ON_CLOSE. You might also look at using MoveFileEx when combining the MOVEFILE_DELAY_UNTIL_REBOOT flag with NULL passed for the new file name. And of course, you could always delete it in your own code if you can tell when the executable has ended.
I don't know enough about Linux executables, so I don't know if a similar feature is available there.
If Linux doesn't provide any convenient mechanism and/or if this idea doesn't suit your needs in Windows, then I suppose your idea of generating an unsigned char array from the contents of the helper exe would be the next best way to embed the exe in your library.
I've inherited a fairly large C++ project in VS2005 which compiles to a DLL of about 5MB. I'd like to cut down the size of the library so it loads faster over the network for clients who use it from a slow network share.
I know how to do this by analyzing the code, includes, and project settings, but I'm wondering if there are any tools available which could make it easier to pinpoint what parts of the code are consuming the most space. Is there any way to generate a "profile" of the DLL layout? A report of what is consuming space in the library image and how much?
When you build your DLL, you can pass /MAP to the linker to have it generate a map file containing the addresses of all symbols in the resulting image. You will probably have to do some scripting to calculate the size of each symbol.
Using a "strings" utility to scan your DLL might reveal unexpected or unused printable strings (e.g. resources, RCS IDs, __FILE__ macros, debugging messages, assertions, etc.).
Also, if you're not already compiling with /Os enabled, it's worth a try.
If your end goal is only to trim the size of the DLL, then after tweaking compiler settings, you'll probably get the quickest results by running your DLL through UPX. UPX is an excellent compression utility for DLLs and EXEs; it's also open-source with a non-viral license, so it's okay to use in commercial/closed-source products.
I've only had it turn up a virus warning on the highest compression setting (the brute-force option), so you'll probably be fine if you use a lower setting than that.
While i don't know about any binary size profilers, you could alternatively look for what object files (.obj) are the biggest - that gives you at least an idea of where your problematic spots are.
Of course this requires a sufficiently modularized project.
You can also try to link statically instead of using a dll. Indeed, when the library is linked statically the linker removes all unused functions from the final exe. Sometime the final exe is only slightly bigger and you don't have any more dll.
If your DLL is this big because it's exporting C++ function with exceptionally long mangled names, an alternative is to use a .DEF file to export the functions by ordinal, without name (using NONAME in the .DEF file). Somewhat brittle, but it reduces the DLL size, EXE size and load times.
See e.g. http://home.hiwaay.net/~georgech/WhitePapers/Exporting/Exp.htm
Given that all your .obj files are about the same size, assuming that you're using precompiled headers, try creating an empty obj file and see how large it is. That will give you an idea of the proportion of each .obj that's due to the PCH compilation. The linker will be able to remove all the duplicates there, incidentally. Alternatively you could try disabling PCH so that the obj files will give you a better indication of where the main culprits are.
All good suggestions. What I do is get the map file and then just eyeball it. The kind of thing I've found in the past is that a large part of the space is taken by one or more class libraries brought in by the fact that some variable somewhere was declared as having a type that sounded like it would save some coding effort but wasn't really necessary.
Like in MFC (remember that?) they have a wrapper class to go around every thing like controls, fonts, etc. that Win32 provides. Those take a ton of space and you don't always need them.
Another thing that can take a ton of space is collection classes you could manage without. Another is cout I/O routines you don't use.
i would recommend one of the following:
coverage - you can run a coverage tool in the hope of detecting some dead code
caching - cache the dll on the client side on the initial activatio
splitting - split the dll into several smaller dlls, start the application with the bootstrap dll and download the other dlls after the application starts
compilation and linking - use smaller run time library, compile with size optimization, etc. see this link for more suggestions.
compression - if you have data or large resources within the dll, you can compress them and decompress only after the download or at runtime.