Is there any consistency when generating a .so file between two builds? When we perform clean and build?
Basically, I wanted the .so file for an app for a previous state of the code (C++), the files changed were very few, which i reverted back, if i build now will the so file be same as the one i got before?
I can replicate the code state to exactly what it was before, I needed that to map the stack trace to the code using this file as we can map hex values to function names.
Thanks
Assuming you are building from the same source code with the same build options, your output product should be the same (with the possible exception of some timestamps embedded into the code). Any compiler/kernel/library upgrades might break this guarantee.
This is exactly what version control (especially tagged snapshots) are for.
Related
I ask my question in such a specific way because I am afraid that a more generic form could lead to excessively theoretic discussions of how the things should be done best and in the most appropriate way (like a question about pre and post-process actions in SCons).
WPP incorporation actually requires execution of an additional command (commands) before compilation of a file and only even if the build process finds necessity to compile the file without any regard to WPP.
I would remark that this is easily achieved with few lines of definitions in a shared Visual Studio property page file making this work for multiple files in multiple projects, folders, etc. in an absolutely transparent for developers way.
Thus I am wondering whether this can be done in a similarly simple way with SCons? I do not have any deep knowledge of either SCons or MSBuild frameworks; I work with them for simple practical use so I would truly appreciate a practical and useful advise.
Here's what I'd suggest.
SCons builds command lines from Environment() variables.
For example the compile command line for building shared object for c++ is stored in SHCXXCOM (and the variable for what is displayed to user when the command is run defaults to SHCXXCOM, but can be changed by modifying SHCXXCOMSTR).
Back to the problem at hand.
Assuming you have a limited number of build steps you want to wrap, you can do something like.
env['SHCXXCOM'] = [ 'MPP PRE COMMAND LINE', env['SHCXXCOM'], 'MPP POST COMMAND LINE']
You'll have to figure out which variables you need to do this with, but take a look at the manpage to figure that out.
https://scons.org/doc/production/HTML/scons-man.html
p.s. I've not tried this, but in theory it should work. Let us know if not.
I'm currently developing an application (Windows) that needs internal modifications upon download time.
Also, I'm delivering it using a Linux host, so, can't compile on demand as proposed.
How does Ninite deal with it?
In Ninite.com, each time you select different options, you get the same .exe, however, with minor modifications inside.
Option 1
Compile the program with predefined data (in Windows).
Use PHP to fseek the file and replace my custom strings.
Option 2
Append the original .EXE with a different resource file
Other?
Has someone developed something like this? What would be the best approach?
Thank you.
You can just append data to the back of your original executable. The Windows PE file format is robust enough that this does not invalidate the executable itself. (It will however invalidate any existing digital signatures.)
Finding the start of this data can be a challenge if its size isn't known up front. In that case, it may be necessary to append the variable-length data, and then append the data length (itself a fixed length field - 4 bytes should do). To read the extra data, read the last 4 bytes to get the data length. Get the file length, subtract 4 for the length field, then subtract the variable length to get the start of the data.
The most portable way could be to have a plugin (whose path in wired inside your main program) inside your application. That plugin would be modified (e.g. on Linux by generating C++ code gencod.cc, forking a g++ -Wall -shared -fPIC -O gencod.cc -o gencod.so compilation, then dlopen-ing the ./gencod.so) and your application could have something to generate the C++ source code of that plugin and to compile it.
I guess that the same might be doable on Windows (which I don't know). Probably the issue is to compile it (the compilation command would be different on Windows and on Linux). Beware that AFAIK on Windows a process cannot modify its own executable (but you should check).
Qt has a portable layer for plugins. See QPluginLoader & Qt Plugins HowTo
Alternatively, don't modify the application, but use some persistent file or data (at a well defined place, -whose location or filepath is wired in the executable- preferably in textual format like JSON, or maybe using sqlite, or a real database) keeping the changing information. Read also about application checkpointing.
If you need to implement your specific application checkpointing, you'll better design your application very early with this concern. Study garbage collection algorithms (a checkpointing procedure is similar to a precise copying GC) and read more about continuations. See also this answer to a very similar question.
We have a bit of a mess of a situation on our build machine...
I finally managed to get one matrix build working, a "first check that everything compiles" type of task which just compiles everything on the current platform it's running on. It runs on multiple platforms fine (about the only problem it might have is that it's compiling the java code multiple times when it could probably be optimised to do that once.)
I imagine that setting up a matrix build for "build installers" would be not too hard, either.
But there are two problems which will definitely hit.
There's one zip file we redistribute which ideally would contain all platform-dependent binaries in a single zip file to reduce duplication (essentially it's a library we hand out to others.)
The process we have for copying the actual releases up to the server relies on every single generated file for the same version number of the same product being ready before the build starts. No single-OS builds would have a complete enough view of the produced files to be able to do the release and it doesn't seem to be possible to add build steps which run in the parent job.
We're using Archive for Clone Workspace SCM as a post-build step for this initial matrix build, but it looks like that runs independently on each OS and no attempt is made to merge the results together.
How do other people get around all these issues?
I know I can just ditch matrix builds entirely and do everything via configuring multiple of each job, but we have three platforms now and the number of jobs would skyrocket.
Options which involve alternatives to Jenkins will be looked at seriously as well, as lately... the number of problems we have been having with it is enormous.
Here's what we eventually set up which is working:
Project "platform-releases" is still a "Matrix Build".
Slaves are each labeled with their relevant platform and we use the slave label as the parameters of the matrix.
The "Archive Artifacts" is used to archive the useful files. ("Clone Workspace SCM" seems to be a dead end when working with matrix builds.)
Project "unified-releases" is a normal build which copies in the artifacts from platform-releases. Since we copy the artifacts without specifying a specific platform, we get the artifacts for all platforms, which appear in directories named /os/windows, /os/macosx, etc.
The Ant build knows about the location of the artifacts (they are outside the working copy) and pulls all the files into a single directory structure before uploading.
"Parameterized Trigger" plugin is set up for platform-releases to trigger unified-releases so that the svn version it's working with matches the actual files produced. Unfortunately, we have to check out the entire repository despite only using a tiny portion of it, because there is a bug (suspected to be in the subversion plugin) which prevents subdirectories being checked out with the correct revision otherwise.
I am writing a program that produces a formatted file for the user, but it's not only producing the formatted file, it does more.
I want to distribute a single binary to the end user and when the user runs the program, it will generate the xml file for the user with appropriate data.
In order to achieve this, I want to give the file contents to a char array variable that is compiled in code. When the user runs the program, I will write out the char file to generate an xml file for the user.
char* buffers = "a xml format file contents, \
this represent many block text \
from a file,...";
I have two questions.
Q1. Do you have any other ideas for how to compile my file contents into binary, i.e, distribute as one binary file.
Q2. Is this even a good idea as I described above?
What you describe is by far the norm for C/C++. For large amounts of text data, or for arbitrary binary data (or indeed any data you can store in a file - e.g. zip file) you can write the data to a file, link it into your program directly.
An example may be found on sites like this one
I'll recommend using another file to contain data other than putting data into the binary, unless you have your own reasons. I don't know other portable ways to put strings into binary file, but your solution seems OK.
However, note that using \ at the end of line to form strings of multiple lines, the indentation should be taken care of, because they are concatenated from the begging of the next lineļ¼
char* buffers = "a xml format file contents, \
this represent many block text \
from a file,...";
Or you can use another form:
char *buffers =
"a xml format file contents,"
"this represent many block text"
"from a file,...";
Probably, my answer provides much redundant information for topic-starter, but here are what I'm aware of:
Embedding in source code: plain C/C++ solution it is a bad idea because each time you will want to change your content, you will need:
recompile
relink
It can be acceptable only your content changes very rarely or never of if build time is not an issue (if you app is small).
Embedding in binary: Few little more flexible solutions of embedding content in executables exists, but none of them cross-platform (you've not stated your target platform):
Windows: resource files. With most IDEs it is very simple
Linux: objcopy.
MacOS: Application Bundles. Even more simple than on Windows.
You will not need recompile C++ file(s), only re-link.
Application virtualization: there are special utilities that wraps all your application resources into single executable, that runs it similar to as on virtual machine.
I'm only aware of such utilities for Windows (ThinApp, BoxedApp), but there are probably such things for other OSes too, or even cross-platform ones.
Consider distributing your application in some form of installer: when starting installer it creates all resources and unpack executable. It is similar to generating whole stuff by main executable. This can be large and complex package or even simple self-extracting archive.
Of course choice, depends on what kind of application you are creating, who are your target auditory, how you will ship package to end-users etc. If it is a game and you targeting children its not the same as Unix console utility for C++ coders =)
It depends. If you are doing some small unix style utility with no perspective on internatialization, then it's probably fine. You don't want to bloat a distributive with a file no one would ever touch anyways.
But in general it is a bad practice, because eventually someone might want to modify this data and he or she would have to rebuild the whole thing just to fix a typo or anything.
The decision is really up to you.
If you just want to keep your distributive in one piece, you might also find this thread interesting: Store data in executable
Why don't you distribute your application with an additional configuration file? e.g. package your application executable and config file together.
If you do want to make it into a single file, try embed your config file into the executable one as resources.
I see it more of an OS than C/C++ issue. You can add the text to the resource part of your binary/program. In Windows programs HTML, graphics and even movie files are often compiled into resources that make part of the final binary.
That is handy for possible future translation into another language, plus you can modify resource part of the binary without recompiling the code.
I'd like to get scons to read a previous version number from a file, update a source file with a new version number and current date and then write the number back to the original file ready for the next build.
This needs to happen only when the target is out of date. IOW the version number doesn't change if no build takes place. The original file is source controlled and isn't a source file else it could trigger another build on check-in (due to CI). CLARIFICATION From scons' point of view the code will always be out of date due to the auto-generated source file but scons will only be run from a Continuous Integration job (Jenkins) when a SCM change is detected.
I've looked into AddPostMethod, but this seems to fire for all files within the list of source files.
Command and Builder methods use the VARIANT_DIR so I can't edit these files and then check them back in as they no longer map to the repo.
I'm hoping I'm just misunderstanding some of the finer details of scons else I'm running out of ideas!
Update
Thinking this through some more, Tom's comment is correct. Although I have two files, one version controlled text file (non-source code) and one non-source controlled source file there is no way to check one file in and prevent a continuous build/check-in cycle. Jenkins will see the new text file and spin off a build, and scons will see the new generated file. So unless I delete the generated file at some point, although this seems to go against the workflow of both tools.
Does anyone have any method for achieving this? It seems pretty straightforward. Ultimately I just want to generate build numbers each time a build is started.
From SCons User Guide section 8, Order-Only Dependencies, you can use the Requires method:
import time
# put whatever text you want in your version.c; this is just regular python
version_c_text = """
char *date = "%s";
""" % time.ctime(time.time())
open('version.c', 'w').write(version_c_text)
version_obj = Object('version.c')
hello = Program('hello.c',
LINKFLAGS = str(version_obj[0]))
Requires(hello, version_obj)
Two things to note: first you have to add the explicit Requires dependency. Second, you can't make version_obj a source of the Program builder, you have to cheat (here we pass it as a linkflag), otherwise you'll get an automatic full dependency on it.
This will update the version.c always, but won't rebuild just because version.c changed.