I've been iterating on a bazel rule for a tool that is dependent on a "custom" (verilator if you're familiar). This tool is supposed to read arguments and inputs and generate cpp files. The action that invokes verilator is defined below
ctx.actions.run(
arguments = [args],
executable = verilator_toolchain.verilator_bin,
inputs = inputs,
outputs = [verilator_output],
progress_message = "[Verilator] Compiling {}".format(ctx.label),
)
The problem is that the executable given to this action is not /exactly/ the same across platforms -- it is slightly larger, has a different hash when comparing mac and linux executables here.
I can trust that the output can be the same, and I'd like to share a remote cache for this action for both platforms; is there a "best practice" where I can rewrite this action to be non-hermetic so the toolchain binary isn't considered as an "input" to the cache? I think the cpp rules do something similar to this.
No, outside of writing an incorrect, non-hermetic rule, there's no way to prevent Bazel for putting all action inputs into the hash key.
Related
I'm trying to understand source code of SDL-1.2.15, and to find out how it renders stuff on windows. But I can't find where the rendering is happening. I looked inside SDL-1.2.15/src/video folder, and there is a ton of subfolders, and I don't know what any of these stands for. See for yourself.
aalib/ directfb/ ipod/ os2fslib/ quartz/ windib/
ataricommon/ dummy/ maccommon/ photon/ riscos/ windx5/
bwindow/ fbcon/ macdsp/ picogui/ svga/ wscons/
caca/ gapi/ macrom/ ps2gs/ symbian/ x11/
dc/ gem/ nanox/ ps3/ vgl/ xbios/
dga/ ggi/ nds/ qtopia/ wincommon/ Xext/
Is this documented somewhere? This is a pretty popular library, so it probably is documented, right? Right? What's the point of having source code if you can't even understand it, if you can't find functions you are using.
While not all the names are self-explanatory, they contain some hints.
directfb, fbcon (framebuffer console) and X (x11, Xext) are output layers on Linux (unix).
The ones starting with win indicate they are for Windows. More specifically, windib should be about device independent bitmaps (DIBs), dx5 about DirectX 5, and wincommon about some common stuff. Indeed, using grep shows that (only) these folders contain Windows-specific code:
grep -r windows.h src/video/*
[ lists files in the win* folders ]
You could also just compile the package on Windows and see which files were compiled (which folders contain object files)
However, to find out what it actually does, you should rather study the function you're interested in (e.g. SDL_BlitSurface), look at it's implementation, and then look at the implementation of the functions it uses. Start in SDL_video.h (and notice that SDL_BlitSurface is just a define).
You should use some tool to search the code base. Grep or some IDE. Or both.
First of all, why not SDL2?
These are different SDL's video drivers. You can get what driver is used by your program by calling SDL_VideoDriverName. Which driver will be used determined by target platform (e.g. operating system - most drivers are platform-specific), environment variable SDL_VIDEODRIVER, or calling side.
I am trying to use C++ application with FreeRTOS.
I come to know about this post :- https://sourceforge.net/p/freertos/discussion/382005/thread/5d5201c0 but I am not sure how and where to add this TaskCPP.h file.
Right now I have very simple main.cpp file something like this.
int main(void)
{
//Set priority bits to preempt priority
NVIC_PriorityGroupConfig(NVIC_PriorityGroup_4);
for( ;; );
return 0;
}
And this gives me an error :-
/usr/bin/../lib/gcc/arm-none-eabi/4.7.4/../../../../arm-none-eabi/bin/ld: error: STM32F4_FreeRTOS.axf uses VFP register arguments, /usr/bin/../lib/gcc/arm-none-eabi/4.7.4/libgcc.a(unwind-arm.o) does not
/usr/bin/../lib/gcc/arm-none-eabi/4.7.4/../../../../arm-none-eabi/bin/ld: failed to merge target specific data of file /usr/bin/../lib/gcc/arm-none-eabi/4.7.4/libgcc.a(unwind-arm.o)
I am not sure what is wrong with settings.
That error is related to your tool chain. Your target triple indicates, a more generic tool chain, but FreeRTOS seems to use more specific ARM features. You may want to read this question: ARM compilation error, VFP registered used by executable, not object file
As workaround: call your compiler with -print-multi-lib and check whether the libraries required by FreeRTOS are available. If they are, you'll have to enable them. If they are not, you'll have to use another tool chain.
I have a program in C++ that is designed to run a simulation for a summer project I'm doing. It is pretty computationally intensive, but I have gotten permission to use a cluster computer's resources to run it, but I test it and develop it on my own laptop. This program generates text files as output, and this is where I run into trouble.
I need the text files to be saved in different paths depending on whether I'm running the program on my own computer or on the cluster computer. My solution for now has been to use $(shell hostname) in my makefile to check which machine the code is being compiled on and, from that output, use conditional compilation with macros defined from that operation in the makefile. At one time, I was using two different versions of a header that defined macros differently on my computer versus the cluster, but I'm using a git repository to transfer changes back and forth, and I was having a very difficult time excluding one file like this.
I was just wondering what is the most preferable practice to set paths at compile time on different computers with the same source.
It doesn't sound to me like it needs to compile differently on different machines. It sounds like it needs to take some paths at run-time from either the command line, or from some sort of config file.
One suggestion would be to use the boost program options library which in one simple setup allows you to read the same params either from the command line or from a config file. This is what I used when running similar jobs on a big cluster or on my laptop and it worked nicely.
Below is a simple example from their docs:
// Declare the supported options.
po::options_description desc("Allowed options");
desc.add_options()
("help", "produce help message")
("compression", po::value<int>(), "set compression level")
;
po::variables_map vm;
po::store(po::parse_command_line(ac, av, desc), vm);
po::notify(vm);
if (vm.count("help")) {
cout << desc << "\n";
return 1;
}
if (vm.count("compression")) {
cout << "Compression level was set to "
<< vm["compression"].as<int>() << ".\n";
} else {
cout << "Compression level was not set.\n";
}
I agree with Alex, the easiest solution will not be at compile time, but at runtime either via a config file or command line arguments. All other things being equal, it may be easier for you to just try passing it via command line arguments using argv and argc.
I am not very experienced with this, but I can think of one simple way of doing this.
Set up an environment variable that points to the appropriate directory on each machine,
and use that environment variable in your makefile.
For example,
in machine 1's ~/.bashrc
export MY_DIRECTORY = ~/Foo
in machine 2's ~/.bashrc
export MY_DIRECTORY = ~/Bar
your Makefile will use the environment variable of the machine it is running on.
Eg. $(MY_DIRECTORY)
And (~/.bashrc is not a part of your repository, so different copies can exist on the two machines)
If you can stomach a dependency on QtCore (between 750K and 4MB library depending on compile options and platform), you can use QSettings to conveniently store the directory path without having to set the directory path each time. You can pass it on the command line at runtime once and have the program store the result into the settings file, and then that setting will become the new default for future invocations of the program without the command line argument.
Other dependency-free alternatives would involve writing your own configuration file parsing routines or using existing ones, but I always like to rely on well-tested and open source code.
Good luck!
You could continue with the route of using separate headers. Include both in your Git repository as clusterHeader.h and laptopHeader.h and use your existing Makefile script to build with a different header on each system. To ease linking troubles, perhaps temporarily rename or copy the file within the script from clusterHeader.hpp or laptopHeader.hpp to just plain old header.h while building, and change the name back at the end of the script.
If you want to keep changes in your header consistent between builds, use this method as another header file, and #include that header within your OS independent header.
i.e.
source.cpp
|
-> header.hpp
|
-> clusterHeader.hpp OR laptopHeader.hpp
Alternatively, as long as the systems aren't exactly the same OS (which I'm assuming they aren't since one is a cluster), you could probably quite easily get it to work with some simple #ifdef statements.
Finally, CMake or qmake are always options.
I've got a series of cpp source file and I want to write another program to JUDGE if they can run correctly (give input and compare their output with standart output) . so how to:
call/spawn another program, and give a file to be its standard input
limit the time and memory of the child process (maybe setrlimit thing? is there any examples?)
donot let the process to read/write any file
use a file to be its standard output
compare the output with the standard output.
I think the 2nd and 3rd are the core part of this prob. Is there any way to do this?
ps. system is Linux
To do this right, you probably want to spawn the child program with fork, not system.
This allows you to do a few things. First of all, you can set up some pipes to the parent process so the parent can supply the input to the child, and capture the output from the child to compare to the expected result.
Second, it will let you call seteuid (or one of its close relatives like setreuid) to set the child process to run under a (very) limited user account, to prevent it from writing to files. When fork returns in the parent, you'll want to call setrlimit to limit the child's CPU usage.
Just to be clear: rather than directing the child's output to a file, then comparing that to the expected output, I'd capture the child's output directly via a pipe to the parent. From there the parent can write the data to a file if desired, but can also compare the output directly to what's expected, without going through a file.
std::string command = "/bin/local/app < my_input.txt > my_output_file.txt 2> my_error_file.txt";
int rv = std::system( command.c_str() );
1) The system function from the STL allows you to execute a program (basically as if invoked from a shell). Note that this approach is inherenly insecure, so only use it in a trusted environment.
2) You will need to use threads to be able to achieve this. There are a number of thread libraries available for C++, but I cannot give you recommendation.
[After edit in OP's post]
3) This one is harder. You either have to write a wrapper that monitors read/write access to files or do some Linux/Unix privilege magic to prevent it from accessing files.
4) You can redirect the output of a program (that it thinks goes to the standard output) by adding > outFile.txt after the way you would normally invoke the program (see 1)) -- e.g. otherapp > out.txt
5) You could run diff on the saved file (from 3)) to the "golden standard"/expected output captured in another file. Or use some other method that better fits your need (for example you don't care about certain formatting as long as the "content" is there). -- This part is really dependent on your needs. diff does a basic comparing job well.
How can I statically link the intel's TBB libraries to my application?
I know all the caveats such as unfair load distribution of the scheduler, but I don't need the scheduler, just the containers, so it's ok.
Anyways I know this can be done, although its undocumented, however I just can't seem to find the way to do it right now (although I've seen it before somewhere).
So does anyone know or have any clues?
thanks
This is strongly not recommended:
Is there a version of TBB that provides statically linked libraries?
TBB is not provided as a statically linked library, for the following reasons*:
Most libraries operate locally. For example, an Intel(R) MKL FFT transforms an array. It is irrelevant how many copies of the FFT there are. Multiple copies and versions can coexist without difficulty. But some libraries control program-wide resources, such as memory and processors. For example, garbage collectors control memory allocation across a program. Analogously, TBB controls scheduling of tasks across a program. To do their job effectively, each of these must be a singleton; that is, have a sole instance that can coordinate activities across the entire program. Allowing k instances of the TBB scheduler in a single program would cause there to be k times as many software threads as hardware threads. The program would operate inefficiently, because the machine would be oversubscribed by a factor of k, causing more context switching, cache contention, and memory consumption. Furthermore, TBB's efficient support for nested parallelism would be negated when nested parallelism arose from nested invocations of distinct schedulers.
The most practical solution for creating a program-wide singleton is a dynamic shared library that contains the singleton. Of course if the schedulers could cooperate, we would not need a singleton. But that cooperation requires a centralized agent to communicate through; that is, a singleton!
Our decision to omit a statically linkable version of TBB was strongly influenced by our OpenMP experience. Like TBB, OpenMP also tries to schedule across a program. A static version of the OpenMP run-time was once provided, and it has been a constant source of problems arising from duplicate schedulers. We think it best not to repeat that history. As an indirect proof of the validity of these considerations, we could point to the fact that Microsoft Visual C++ only provides OpenMP support via dynamic libraries.
Source: http://www.threadingbuildingblocks.org/faq/11#sthash.t3BrizFQ.dpuf
EDIT - Changed to use extra_inc. Thanks Jeff!
Build with the following parameter:
make extra_inc=big_iron.inc
The static libraries will be built. See the caveats in build/big_iron.inc.
Build static libraries from source
After acquiring the source code from https://www.threadingbuildingblocks.org/, build TBB like this:
make extra_inc=big_iron.inc
If you need extra options, then instead build like this:
make extra_inc=big_iron.inc <extra options>
Running multiple TBB programs per node
If you run a multiprocessing application, e.g. using MPI, you may need to explicitly initialize the TBB scheduler with the appropriate number of threads to avoid oversubscription.
An example of this in a large application can be found in https://github.com/m-a-d-n-e-s-s/madness/blob/master/src/madness/world/thread.cc.
Comment on documentation
This feature has been available for many years (since at least 2013), although it is not documented for the reasons described in other answers.
Historical note
This feature was originally developed because IBM Blue Gene and Cray supercomputers either did not support shared libraries or did not perform well when using them, due to the lack of a locally mounted filesystem.
Using the opensource version:
After running "make tbb"，go to the build/linux_xxxxxxxx_release folder.
Then run:
ar -r libtbb.a concurrent_hash_map.o concurrent_queue.o concurrent_vector.o
dynamic_link.o itt_notify.o cache_aligned_allocator.o pipeline.o queuing_mutex.o
queuing_rw_mutex.o reader_writer_lock.o spin_rw_mutex.o spin_mutex.o critical_section.o
task.o tbb_misc.o tbb_misc_ex.o mutex.o recursive_mutex.o condition_variable.o
tbb_thread.o concurrent_monitor.o semaphore.o private_server.o rml_tbb.o
task_group_context.o governor.o market.o arena.o scheduler.o observer_proxy.o
tbb_statistics.o tbb_main.o concurrent_vector_v2.o concurrent_queue_v2.o
spin_rw_mutex_v2.o task_v2.o
And you should get libtbb.a as output.
Note that your program should build both with "-ldl" and libtbb.a
Although not officially endorsed by the TBB team, it is possible to build your own statically linked version of TBB with make extra_inc=big_iron.inc.
I have not tested it on Windows or MacOS, but on Linux, it worked (source):
wget https://github.com/01org/tbb/archive/2017_U6.tar.gz
tar xzfv 2017_U6.tar.gz
cd tbb-2017_U6
make extra_inc=big_iron.inc
The generated files are in tbb-2017_U6/build/linux*release.
When you link your application to the static TBB version:
Call g++ with the -static switch
Link against tbb (-ltbb) and pthread (-lpthread)
In my test, I also needed to explicitely reference all .o files from the manually build TBB version. Depending on your project, you might also need to pass -pthread to gcc.
I have created a toy example to document all the steps in this Github repository:
tbb-static-linking-tutorial
It also contains test code to make sure that the generated binary is portable on other Linux distributions.
Unfortunately it does not appear to be possible: From TBB site..
One suggestion on the Intel forum was to compile it manually if you really need the static linkage: From Intel Forum.
Just link the files, I just did it and works. Here's the SConscript file. There's two minor things, a symbol which has the same name in tbb and tbbmalloc which I had to prevent to be multiply defined, and I prevented the usage of ITT_NOTIFY since it creates another symbol with the same name in both libs.
Import('g_CONFIGURATION')
import os
import SCutils
import utils
tbb_basedir = os.path.join(
g_CONFIGURATION['basedir'],
'3rd-party/tbb40_233oss/')
#print 'TBB base:', tbb_basedir
#print 'CWD: ', os.getcwd()
ccflags = []
cxxflags = [
'-m64',
'-march=native',
'-I{0}'.format(tbb_basedir),
'-I{0}'.format(os.path.join(tbb_basedir, 'src')),
#'-I{0}'.format(os.path.join(tbb_basedir, 'src/tbb')),
'-I{0}'.format(os.path.join(tbb_basedir, 'src/rml/include')),
'-I{0}'.format(os.path.join(tbb_basedir, 'include')),
]
cppdefines = [
# 'DO_ITT_NOTIFY',
'USE_PTHREAD',
'__TBB_BUILD=1',
]
linkflags = []
if g_CONFIGURATION['build'] == 'debug':
ccflags.extend([
'-O0',
'-g',
'-ggdb2',
])
cppdefines.extend([
'TBB_USE_DEBUG',
])
else:
ccflags.extend([
'-O2',
])
tbbenv = Environment(
platform = 'posix',
CCFLAGS=ccflags,
CXXFLAGS=cxxflags,
CPPDEFINES=cppdefines,
LINKFLAGS=linkflags
)
############################################################################
# Build verbosity
if not SCutils.has_option('verbose'):
SCutils.setup_quiet_build(tbbenv, True if SCutils.has_option('colorblind') else False)
############################################################################
tbbmallocenv = tbbenv.Clone()
tbbmallocenv.Append(CCFLAGS=[
'-fno-rtti',
'-fno-exceptions',
'-fno-schedule-insns2',
])
#tbbenv.Command('version_string.tmp', None, '')
# Write version_string.tmp
with open(os.path.join(os.getcwd(), 'version_string.tmp'), 'wb') as fd:
(out, err, ret) = utils.xcall([
'/bin/bash',
os.path.join(g_CONFIGURATION['basedir'], '3rd-party/tbb40_233oss/build/version_info_linux.sh')
])
if ret:
raise SCons.Errors.StopError('version_info_linux.sh execution failed')
fd.write(out);
#print 'put version_string in', os.path.join(os.getcwd(), 'version_string.tmp')
#print out
fd.close()
result = []
def setup_tbb():
print 'CWD: ', os.getcwd()
tbb_sources = SCutils.find_files(os.path.join(tbb_basedir,'src/tbb'), r'^.*\.cpp$')
tbb_sources.extend([
'src/tbbmalloc/frontend.cpp',
'src/tbbmalloc/backref.cpp',
'src/tbbmalloc/tbbmalloc.cpp',
'src/tbbmalloc/large_objects.cpp',
'src/tbbmalloc/backend.cpp',
'src/rml/client/rml_tbb.cpp',
])
print tbb_sources
result.append(tbbenv.StaticLibrary(target='libtbb', source=tbb_sources))
setup_tbb()
Return('result')