QBS build system, can't initialize environment with vcvars64.bat - c++

I'm trying to implement my own module to build C++ on Windows with clang-cl toolchain as there's no built-in support in QBS right now.
I chose to use lld-link instead of microsoft linker, so I have to supply it with all the MS library include paths manually. With these paths hardcoded, I manage to build my apps fine. But I'd like to make my module more flexible and use %LIB% environment variable set by vcvars32.bat|vcvars64.bat
As far as I understand, this could (should?) be done inside module's setupBuildEnvironment script. Here's what I try to read the %LIB% and fail:
import qbs.Environment
import qbs.Process
Module
{
setupBuildEnvironment:
{
var p = new Process();
p.exec("vcvars64.bat", [], true);
// makes no difference
// p.exec("cmd", ["/c", "vcvars64.bat"], true);
var lib = p.getEnv("LIB");
// this fails too
// var lib = Environment.getEnv("LIB");
console.info("LIB = " + lib);
p.close();
}
...
}
This gives me LIB = so I'm getting nowhere. My guess is that the process is already terminated at the moment of querying the variable (p.getEnv("LIB")), hence the empty result. The QBS docs for Process.getEnv() state nothing in this regard.
What is the correct QBS way to initialize environment with vcvars64.bat, and more broadly, what is the correct way to get environment of a process inside setupBuildEnvironment?
[update]
Well, embarassingly, this was easy to work around by creating a simple batch and getting rid of setupBuildEnvironment script altogether:
#echo off
call vcvars64 && qbs
But I'd like to avoid batch scripting as much as possible, so the question still stands.

The vars batch files just dump some information onto the console. That does not set an environment on the calling process in any way. You would need to parse the process output. I suggest you take a look at the MsvcProbe item in the qbs sources to see how that is implemented for MSVC. You might be able to adapt the code for clang-cl.

Related

C++ - I need a macro or an environment variable to detect when I am doing bazel run on Windows

As a workaround, I have to append "../../../../../../"s to the file paths I want to open that are part of my data dependencies when I run my executable using bazel on Windows. See also the comment linked here: C++ Bazel project with a Data repository
I would like to have something like this below, which would simplify my application testing a lot:
auto basePath = "dataDir/subDir/"s;
if( isRunningFromBazelOnWindows() ) basePath = "../../../../../../"s + basePath;
auto file = std::ifstream{basePath + "dataFile"};
...
how can I code isRunningFromBazelOnWindows()?
Thanks!
Bazel now (maybe not yet released in stable bazel?) provides runfiles libraries simplifying access to runfiles. I guess the design doc is best place to start.

Running a compiled executable in Xcode gives a different result from directly invoking the executable in the shell

The program uses OpenSceneGraph to generate an .osg file.
int main( int, char** )
{
osg::ref_ptr<osg::Node> root = createSceneGraph();
if (!root.valid())
{
osg::notify(osg::FATAL) << "Failed in createSceneGraph()." << endl;
}
bool result = osgDB::writeNodeFile( *(root.get()), "Simple.osg" );
if ( !result )
{
osg::notify(osg::FATAL) << "Failed in osgDB::writeNode()." << endl;
}
}
The code excerpt is from openscenegraph Quick Start Guide http://www.lulu.com/items/volume_51/767000/767629/3/print/OSGQSG.pdf
page 40
The program can be compiled and run in XCode.
But, it shows the following error:
Error writing file Simple.osg: Warning: Could not find plugin to
write nodes to file "Simple.osg".
Failed in osgDB::writeNode().
However, the compiled binary can be executed without error by directly running in the shell.(./program) And the program correctly generates the Simple.osg file.
This is not the end of story. It is most likely because xcode invokes the compile program differently from invoking it in the shell. So, I copy and paste two dependent libraries into the directory where the compiled binary resides. (In case who knows OSG, the two libraries are osgdb_deprecated_osg.so and osgdb_osg.so)
This time, the Xcode correctly runs the program without errors, generating the Simple.osg file.
Therefore, there are two questions.
1) Why the shell is able to locate the dependent libraries automatically whereas the Xcode cannot?
2) How to make Xcode work without manually copying the dependent libraries or via post compile copy options?
This is simply a matter of the shell PATH being different in the shell and in Xcode.
A lot of the time an IDE will let you specify what PATH is in use at executable run-time. Make sure it contains all the important entries the normal shell has.
OK. After several hours I finally get it resolved.
The reason why the libraries are loaded under the shell is because of the shell's default environmental variable settings in ~/.profile, so that the program can search the libraries with these variables.
As XenonofArcticus pointed out, xcode doesn't inherit the environmental variable settings in the ~/.profile so the program running under xcode cannot resolve the actual path of the libraries.
However, at first I copied those variables in the environmental variable settings in the xcode but it didn't work.
It turns out that the environmental variable settings in xcode is not expandable across different ones. e.g:
HELLO=123/hi
WORLD=$HELLO/enter
The second WORLD variable does not replace the HELLO defined previously into the path.
Therefore, I need to manually write like this:
HELLO=123/hi
WORLD=123/hi/enter

Py_Initialize fails - unable to load the file system codec

I am attempting to put together a simple c++ test project that uses an embedded python 3.2 interpreter. The project builds fine but Py_Initialize raises a fatal error:
Fatal Python error: Py_Initialize: unable to load the file system codec
LookupError: no codec search functions registered: can't find encoding
Minimal code:
#include <Python.h>
int main (int, char**)
{
Py_Initialize ();
Py_Finalize ();
return 0;
}
The OS is 32bit Vista.
The python version used is a python 3.2 debug build, built from sources using VC++ 10.
The python_d.exe file from the same build runs without any problems.
Could someone explain the problem and how to fix it? My own google-fu fails me.
EDIT 1
After going through the python source code I've found that, as the error says, no codec search functions have been registered. Both codec_register and PyCodec_Register are as they should be. It's just that nowhere in the code are any of these functions called.
I don't really know what this means as I still have no idea when and from where these functions should have been called. The code that raises the error is entirely missing from the source of my other python build (3.1.3).
EDIT 2
Answered my own question below.
Check the PYTHONPATH and PYTHONHOME environment variables and make sure they don't point to Python 2.x.
http://bugs.python.org/issue11288
Parts of this have been mentioned before, but in a nutshell this is what worked for my environment where I have multiple Python installs and my global OS environment set-up to point to a different install than the one I attempt to work with when encountering the problem.
Make sure your (local or global) environment is fully set-up to point to the install you aim to work with, e.g. you have two (or more) installs of, let's say a python27 and python33 (sorry these are windows paths but the following should be valid for equivalent UNIX-style paths just as well, please let me know about anything I'm missing here (probably the DLLs path might differ)):
C:\python27_x86
C:\python33_x64
Now, if you intend to work with your python33 install but your global environment is pointing to python27, make sure you update your environment as such (while PATH and PYTHONHOME may be optional (e.g. if you temporarily work in a local shell)):
PATH="C:\python33_x64;%PATH%"
PYTHONPATH="C:\python33_x64\DLLs;C:\python33_x64\Lib;C:\python33_x64\Lib\site-packages"
PYTHONHOME=C:\python33_x64
Note, that you might need/want to append any other library paths to your PYTHONPATH if required by your development environment, but having your DLLs, Lib and site-packages properly set-up is of prime importance.
Hope this helps.
The core reason is quite simple: Python does not find its modules directory, so it can of course not load encodings, too
Python doc on embedding says "Py_Initialize() calculates the module search path based upon its best guess" ... "In particular, it looks for a directory named lib/pythonX.Y"
Yet, if the modules are installed in (just) lib - relative to the python binary - above guess is wrong.
Although docs says that PYTHONHOME and PYTHONPATH are regarded, we observed that this was not the case; their actual presence or content was completely irrelevant.
The only thing that had an effect was a call to Py_SetPath() with e.g. [path-to]\lib as argument before Py_Initialize().
Sure this is only an option for an embedding scenario where one has direct access and control over the code; with a ready-made solution, special steps may be necessary to solve the issue.
Ran into the same thing trying to install brew's python3 under Mac OS! The issue here is that in Mac OS, homebrew puts the "real" python a whole layer deeper than you think. You would think from the homebrew output that
$ echo $PYTHONHOME
/usr/local/Cellar/python3/3.6.2/
$ echo $PYTHONPATH
/usr/local/Cellar/python3/3.6.2/bin
would be correct, but invoking $PYTHONPATH/python3 immediately crashes with the abort 6 "can't find encodings." This is because although that $PYTHONHOME looks like a complete installation, having a bin, lib etc, it is NOT the actual Python, which is in a Mac OS "Framework". Do this:
PYTHONHOME=/usr/local/Cellar/python3/3.x.y/Frameworks/Python.framework/Versions/3.x
PYTHONPATH=$PYTHONHOME/bin
(substituting version numbers as appropriate) and it will work fine.
From python3k, the startup need the encodings module, which can be found in PYTHONHOME\Lib directory.
In fact, the API Py_Initialize () do the init and import the encodings module.
Make sure PYTHONHOME\Lib is in sys.path and check the encodings module is there.
I had this issue with python 3.5, anaconda 3, windows 7 32 bit. I solved it by moving my pythonX.lib and pythonX.dll files into my working directory and calling
Py_SetPythonHome(L"C:\\Path\\To\\My\\Python\\Installation");
before initialize so that it could find the headers that it needed, where my path was to "...\Anaconda3\". The extra step of calling Py_SetPythonHome was required for me or else I'd end up getting other strange errors where python import files.
I just ran into the exact same problem (same Python version, OS, code, etc).
You just have to copy Python's Lib/ directory in your program's working directory ( on VC it's the directory where the .vcproj is )
There appears to be something going wrong with the release build either failing to include the appropriate codecs or else misidentifying the codec to use for system APIs. Since the python_d executable is working, what does that return for os.getfsencoding()? (Use the C API to call that between your Initialize/Finalize calls)
I had the same issue and found this question. However from the answers here I was not able to solve my problem. I started debugging the cpython code and thought that I might be discovered a bug. Therefore I opened a issue on the python issue tracker.
My mistake was that I did not understand that Py_SetPath clears all inferred paths.
So one needs to set all paths when calling this function.
Link to the issue conversation
For completion I also copied the most important part of the conversation below.
My original issue text
I compiled the source of CPython 3.7.3 myself on Windows with Visual Studio 2017 together with some packages like e.g numpy. When I start the Python Interpreter I am able to import and use numpy. However when I am running the same script via the C-API I get an ModuleNotFoundError.
So the first thing I did, was to check if numpy is in my site-packages directory and indeed there is a folder named numpy-1.16.2-py3.7-win-amd64.egg. (Makes sense because the python interpreter can find numpy)
The next thing I did was to get some information about the sys.path variable created when running the script via the C-API.
#### sys.path content ####
C:\Work\build\product\python37.zip
C:\Work\build\product\DLLs
C:\Work\build\product\lib
C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO\2017\PROFESSIONAL\COMMON7\IDE\EXTENSIONS\TESTPLATFORM
C:\Users\rvq\AppData\Roaming\Python\Python37\site-packages
Examining the content of sys.path I noticed two things.
C:\Work\build\product\python37.zip has the correct path 'C:\Work\build\product\'. There was just no zip file. All my files and directory were unpacked. So I zipped the files to an archive named python37.zip and this resolved the import error.
C:\Users\rvq\AppData\Roaming\Python\Python37\site-packages is wrong it should be C:\Work\build\product\Lib\site-packages but I dont know how this wrong path is created.
The next thing I tried was to use Py_SetPath(L"C:/Work/build/product/Lib/site-packages") before calling Py_Initialize(). This led to
Fatal Python Error 'unable to load the file system encoding'
ModuleNotFoundError: No module named 'encodings'
I created a minimal c++ project with exact these two calls and started to debug Cpython.
int main()
{
Py_SetPath(L"C:/Work/build/product/Lib/site-packages");
Py_Initialize();
}
I tracked the call of Py_Initialize() down to the call of
static int
zipimport_zipimporter___init___impl(ZipImporter *self, PyObject *path)
inside of zipimport.c
The comment above this function states the following:
Create a new zipimporter instance. 'archivepath' must be a path-like
object to a zipfile, or to a specific path inside a zipfile. For
example, it can be '/tmp/myimport.zip', or
'/tmp/myimport.zip/mydirectory', if mydirectory is a valid directory
inside the archive. 'ZipImportError' is raised if 'archivepath'
doesn't point to a valid Zip archive. The 'archive' attribute of the
zipimporter object contains the name of the zipfile targeted.
So for me it seems that the C-API expects the path set with Py_SetPath to be a path to a zipfile. Is this expected behaviour or is it a bug?
If it is not a bug is there a way to changes this so that it can also detect directories?
PS: The ModuleNotFoundError did not occur for me when using Python 3.5.2+, which was the version I used in my project before. I also checked if I had set any PYTHONHOME or PYTHONPATH environment variables but I did not see one of them on my system.
Answer
This is probably a documentation failure more than anything else. We're in the middle of redesigning initialization though, so it's good timing to contribute this feedback.
The short answer is that you need to make sure Python can find the Lib/encodings directory, typically by putting the standard library in sys.path. Py_SetPath clears all inferred paths, so you need to specify all the places Python should look. (The rules for where Python looks automatically are complicated and vary by platform, which is something I'm keen to fix.)
Paths that don't exist are okay, and that's the zip file. You can choose to put the stdlib into a zip, and it will be found automatically if you name it the default path, but you can also leave it unzipped and reference the directory.
A full walk through on embedding is more than I'm prepared to type on my phone. Hopefully that's enough to get you going for now.
I had the problem and was tinkering with different solutions mentioned here. Since I was running my project from Visual Studio, apparently, I needed to set the environment path inside Visual Studio and not the system path.
Adding a simple PYTHONHOME=PATH\TO\PYTHON\DIR in the project solution\properties\environment solved the problem.
For me this happened when I updated Python 64 bit from 3.6.4 to 3.6.5. It threw some error like "unable to extract python.dll. Do you have permissions."
Pycharm also failed to load interpreter, even though I reloaded it in settings. Running python command gave same error, with and without administrator mode.
Reason
There was error in installation of Python, include folder in python installation directory C:\Users\USERNAME\AppData\Local\Programs\Python\Python36 was missing
Reinstalling Python also dint solve the issue.(Not removal and install)
Solution
Uninstall Python and Install of Python again.
Because running installer was just extracting same files excluding include folder
In my cases, for windows, if you have multiple python versions installed, if PYTHONPATH is pointing to one version the other ones didn't work. I found that if you just remove PYTHONPATH, they all work fine
For those working in Visual Studio simply add the include, Lib and libs directories to the Include Directories and Library Directories under
Projects Properties -> Configuration Properties > VC++ Directories :
For example I have Anaconda3 on my system and working with Visual Studio 2015 This is how the settings looks like (note the Include and Library directories) :
Edit:
As also pointed out by bossi setting PYTHONPATH in your user Environment Variables section seems necessary.
a sample input can be like this (in my case):
C:\Users\Master\Anaconda3\Lib;C:\Users\Master\Anaconda3\libs;C:\Users\Master\Anaconda3\Lib\site-packages;C:\Users\Master\Anaconda3\DLLs
is necessary it seems.
Also, you need to restart Visual Studio after you set up the PYTHONPATH in your user Environment Variables for the changes to take effect.
Also note that :
Make sure the PYTHONHOME environment variable is set to the Python
interpreter you want to use. The C++ projects in Visual Studio rely on
this variable to locate files such as python.h, which are used when
creating a Python extension.
So, for some reason the python dll fails to locate the encodings module. The python.exe executable apparently finds it because it has the expected relative path. Modifying the search path works.
The reason for all of this? Don't know but at least it works. I highly suspect a typo on my part somewhere, that's usually the reason for odd bugs it seems.

scons - how to run something /after/ all targets have been built

I've recently picked up scons to implement a multi-platform build framework for a medium sized C++ project. The build generates a bunch of unit-tests which should be invoked at the end of it all. How does one achieve that sort of thing?
For example in my top level sconstruct, I have
subdirs=['list', 'of', 'my', 'subprojects']
for subdir in subdirs:
SConscript(dirs=subdir, exports='env', name='sconscript',
variant_dir=subdir+os.sep+'build'+os.sep+mode, duplicate=0)
Each of the subdir has its unit-tests, however, since there are dependencies between the dlls and executables built inside them - i want to hold the running of tests until all the subdirs have been built and installed (I mean, using env.Install).
Where should I write the loop to iterate through the built tests and execute them? I tried putting it just after this loop - but since scons doesn't let you control the order of execution - it gets executed well before I want it to.
Please help a scons newbie. :)
thanks,
SCons, like Make, uses a declarative method to solving the build problem. You don't want to tell SCons how to do its job. You want to document all the dependencies and then let SCons solve how it builds everything.
If something is being executed before something else, you need to create and hook up the dependencies.
If you want to create dmy touch files, you can create a custom builder like:
import time
def action(target, source, env):
os.system('echo here I am running other build')
dmy_fh = open('dmy_file','w')
dmy_fh.write( 'Dummy dependency file created at %4d.%02d.%02d %02dh%02dm%02ds\n'%time.localtime()[0:6])
dmy_fh.close()
bldr = Builder(action=action)
env.Append( BUILDERS = {'SubBuild' : bldr } )
env.SubBuild(srcs,tgts)
It is very important to put the timestamp into the dummy file, because scons uses md5 hashes. If you have an empty file, the md5 will always be the same and it may decide to not do subsequent build steps. If you need to generate different tweaks on a basic command, you can use function factories to modify a template. e.g.
def gen_a_echo_cmd_func(echo_str):
def cmd_func(target,source,env):
cmd = 'echo %s'%echo_str
print cmd
os.system(cmd)
return cmd_fun
bldr = Builder(action = gen_a_echo_cmd_func('hi'))
env.Append(BUILDERS = {'Hi': bldr})
env.Hi(srcs,tgts)
bldr = Builder(action = gen_a_echo_cmd_func('bye'))
env.Append(BUILDERS = {'Bye': bldr})
env.Bye(srcs,tgts)
If you have something that you want to automatically inject into the scons build flow ( e.g. something that compresses all your build log files after everything else has run ), see my question here.
The solution should be as simple as this.
Make the result of the Test builders depend on the result of the Install builder
In pseudo:
test = Test(dlls)
result = Install(dlls)
Depends(test,result)
The best way would be if the Test builder actually worked out the dll dependencies for you, but there may be all kinds of reasons it doesn't do that.
In terms of dependencies, what you want is for all the test actions to depend on all the program-built actions. A way of doing this is to create and export a dummy-target to all the subdirectories' sconscript files, and in the sconscript files, make the dummy-target Depends on the main targets, and have the test targets Depends on the dummy-target.
I'm having a bit of trouble figuring out how to set up the dummy target, but this basically works:
(in top-level SConstruct)
dummy = env.Command('.all_built', 'SConstruct', 'echo Targets built. > $TARGET')
Export('dummy')
(in each sub-directory's SConscript)
Import('dummy')
for target in target_list:
Depends(dummy, targe)
for test in test_list:
Depends(test, dummy)
I'm sure further refinements are possible, but maybe this'll get you started.
EDIT: also worth pointing out this page on the subject.
Just have each SConscript return a value on which you will build dependencies.
SConscript file:
test = debug_environment.Program('myTest', src_files)
Return('test')
SConstruct file:
dep1 = SConscript([...])
dep2 = SConscript([...])
Depends(dep1, dep2)
Now dep1 build will complete after dep2 build has completed.

How to make XCode put required resources in "build" folder?

I am trying out lua script with C++ in Mac OS X. I was finding a way to make the program returning the current working directory. That's no problem with getcwd, but then I came one thing:
My foo.lua stays at its initial path only. When I compile program, it is not being copied over to the build/Debug directory. Sure, I can grab my script there, but that's just impractical. XCode or any IDE should carry resources to the build zone. XCode does this automatically with iPhone app, but this seems to be a different case. For this case, how to command XCode to put the respective resources in the build directories?
int main (int argc, char * const argv[]) {
...
...
luaL_dofile(luaVM,"/Users/yourNameHere/Desktop/LuaSandbox/LetsTryLua/foo.lua");
//typing the whole absolute path here is just ugly and impractical.
...
...
printf("working directory: %s", buffer);
//output is: working directory: /Users/yourNameHere/Desktop/LuaSandbox/LetsTryLua/build/Debug
...
...
Rather than hard code the path to your Lua script you may want to use the NSBundle API's to find it:
NSBundle * mainNSBundle = [NSBundle mainBundle];
NSString * luaFilePath = [mainNSBundle pathForResource:#"foo"
ofType:#"lua"
inDirectory:NULL
forLocalization:NULL];
luaL_dofile(luaVM,[luaFilePath UTF8String]);
This will find it in the bundle's folder (if you added the "Copy Bundle Resources" build step to your target as the above poster suggested.
Because you're using a .lua file as a resource, I suspect that isn't recognised as a standard resource type and hence it hasn't been automatically copied. You should be able to do this though by adding an extra Copy Bundle Resources build step to your target and then add your file to it in the project view.
If you're creating a command line tool that is not a bundle, then there's never going to be a good solution. If you're creating a regular app then the aforementioned solution will work, but you're going to have to stop assuming that your working directory is set to anything even remotely meaningful at any point in time and use the appropriate methods for finding resources stored within your bundle.