embedding OpenCL code into executable - c++

This is a followup to Using #include to load OpenCL code
I've noticed that when you use the method described by grrussel (and used in Bullet Physics), the created string has all newlines stripped (comments seem to be stripped, too, but I'm not too worried about that). Now for the most part this is fine if the included opencl code doesn't have any preprocessor definitions in it, but if there is the code will fail to compile with the OpenCL compiler.
Is there a way to get the #include to keep the newlines in there, or is there a better method to embed the opencl code into my executable (other than copying the string into a cpp file and putting quotes around everything)?
I tested this in Visual Studio 2010, I'm not sure if other compilers exhibit the same behavior. I would prefer a method which doesn't require any external tools and works with a variety of compilers/platforms.
Copied code from other answer:
In C++ / C source
#define MSTRINGIFY(A) #A
char* stringifiedSourceCL =
#include "VectorAddKernels.cl"
In the OpenCL source
MSTRINGIFY(
__kernel void VectorAdd(__global float8* c)
{
// snipped out OpenCL code...
return;
}
);

I'd write a small tool that generates a cpp with a constant string from the text file, which you would then compile along with the other stuff.
This is what Qt's resource compiler moc does (and more, since it has a whole API around it), and there's also the option of Windows's resource kit (see this answer below).
Anyway, a small build tool that walks a files in a dir and builds a cpp from them should be enough, and something you can put together in 15 minutes (given suitable scripting skills).
Update: As per Henry's comment, on Linux (or with Msys/Cygwin), you could use xxd with the -i (i.e. xxd -i ) flag, which reads a file and generates a c-include file with the exact contents defined properly.

Excellent question!
One thing that works on Windows is to treat your OpenCL C files as Windows resources. Then the source code gets built into your executable. In your reasource (.rc) file, you would add something like this:
IDR_OPENCL_FILE_1 OPENCL_SOURCE "mydir\\myfile.cl"
Then, in the host source, something like this:
// Load the resource memory
HRSRC resource = FindResource(NULL, L"IDR_OPENCL_FILE_1", L"OPENCL_SOURCE");
HGLOBAL resMem = LoadResource(NULL, resource);
char* source = (char*)LockResource(resMem);
// Build a STL string out of it
DWORD resSize = SizeofResource(NULL, resource);
std::string cl_source = string(source, resSize);
On Linux, objcopy will let you do something similar. You can build an elf .o with the source file exported as a symbol. Sorry I don't have a code snippet handy for that right now.
I wish there were a better platform-independent answer. If you come across it, let me know.

Related

MSVC C++ compiler option to prevent replacing LF by CR LF in ostream/ofstream

I'm using Microsoft Visual Studio 2017's command line tools such as cl.exe to compile C++ codes.
Instructions to output line breaks like
std::cout << "abc" << std::endl;
or
std::cout << "abc" << '\n';
or
printf("abc\n");
result in "abc\r\n" instead of "abc\n".
How can I change this behavior to get "abc\n" output?
If possible, I want it to be realized without making a big change to the core C++ source code like adding "binary mode" flag to everywhere an output stream is created. (I already have a relatively large C++ project which is fine with GCC and trying to build it with MSVC)
There's no way to achieve this with Standard C++. Apparently there's no way to achieve this with implementation specific documented functions. If it is possible, it will rely on implementation details.
You may hook CRT functions fopen, _wfopen, fopen_s, wfopen_s, in hooked function you would alter open mode. C++ streams internally use those functions as well.
How exactly to do it will depend on whether you link CRT statically or dynamically and which version of Visual Studio you use.
I want it to be realized without making a big change to the core C++ source code like adding "binary mode" flag to everywhere an output stream is created.
There's no correct way other than fixing your buggy code. There shouldn't be a lot of places that open files so just fix that manually. You don't open tens of thousands of files at a time, do you?
That said, if you really want to use LF everywhere in the C++ streams then the Microsoft STL is open source. You can download the STL, patch it and use that one instead of the shipped version in MSVC. The C standard libraries aren't open sourced so you'll have to fix the C part manually, if any
However even if you patch the STL, the source code or the output binary to make it always use LF instead of CRLF then lots of other functionalities will fail, for example console output won't work properly. This is also highly fragile because new code or new 3rd party libraries added in the future might break because they don't expect LF, so beware of that
I already have a relatively large C++ project which is fine with GCC and trying to build it with MSVC
Line ending has nothing to do with the compiler. Even if you build with GCC on Windows then it'll still use CRLF. It's a platform property
You can prevent this via _setmode which you can call at the beginning of your program (watch out for global constructors which might already print something):
#ifdef _WIN32
#include <fcntl.h>
#include <io.h>
#endif
int main() {
#ifdef _WIN32
_setmode(_fileno(stdout), _O_BINARY);
#endif
// ...
}

Assistance including a basic SDK into a C++ program Dev C++

I have been making some applications that I would like to link to discord, specifically with Discord's SDK. The SDK comes with 64 and 32 bit x84 lib files, and a C++ folder full of the includes it needs, so I would assume C++ is supported.
However, I am very terrible at linking libraries or anything at that, and always run into issues when linking. I am using Dev C++ as my IDE, and my code is as follows:
#include <iostream>
#include "Discord/discord.h"
using namespace std;
void InitDiscord()
{
auto discid = 772671910668133376; //Not my actuall discord app ID, but real one does not make a difference
discord::Core* core{};
discord::Core::Create(discid, DiscordCreateFlags_Default, &core);
}
int main(){
InitDiscord();
cout << "Discord active";
while(1){
}
return 0;
}
and I am getting the error:
C:\TDM-GCC-64\x86_64-w64-mingw32\bin\ld.exe Discord Testing.o:Discord Testing.cpp:(.text+0x32): undefined reference to `discord::Core::Create(long long, unsigned long long, discord::Core**)'
for only the line discord::Core::Create(discid, DiscordCreateFlags_Default, &core); and not discord::Core* core{};
I am using C++17 and a newer TDM-GCC compiler, the same one that works for all of my other applications. I am including the .lib files and .dll files in the program's directory, and in the linker the only thing I am using is -discord_game_sdk.dll.lib which is a valid directory. I have also tried discord_game_sdk.dll.lib and putting the library in the same directory as the includes: Discord/discord_game_sdk.dll.lib. I have tried using both 32 bit and 64 bit libraries in all project and compiler directories with no change, and im sure this is something probably really simple, but nowhere have I found any example C++ discord programs or how to include their SDK.
If anyone could figure out what the problem is and how I can fix it, that would be very helpful and appreciated.
EDIT:
It appears that user4581301 was right, TDM-GCC and other Mingw compilers do not support .lib files, and will ignore them despite being linked. The SDK did not come with any other formats other than .dylib, .so, and .bundle.
This creates a somewhat new issue, I already have my compiler set up and cannot really switch to Visual Studio, so I need a way to convert .lib to .a somehow. A post here recommends http://code.google.com/p/lib2a/ , which requires a .def file, another file that did not come with the SDK, but apparently a program called gendef.exe that came with my compiler can create .def files from .dll files. That is indeed the case, however when attempting it I get the error:
C:\Program Files (x86)\Dev-Cpp\MinGW64\x86_64-w64-mingw32\bin>gendef.exe discord_game_sdk.dll
* [discord_game_sdk.dll] Found PE image
* failed to create discord_game_sdk.def ...
with no other warnings. Now I need to know if I am converting wrong, if there is an easier workaround, or if one of the other file types can be converted or used. Any suggestions at this point are welcome and appreciated!

Using a function defined in a DLL from C++ code

I built Qt from source (dlls) and am trying to build an application that uses the Qt dlls. I don't have a lot of experience with C++ so I'm running into what I'm sure is a very basic issue.
My builds are failing on the includes with errors like so:
Fatal error: QNetworkProxy: No such file or directory
Here is the g++ command I am using (I also used -L to add the correct folder to the lib path, but that also didn't work):
g++ -l..\..\wkqt\bin\QtCore4.dll -l..\..\wkqt\bin\QtNetwork4.dll -l..\..\wkqt\bin\QtWebKit4.dll -I..\include -Ishared -Ipdf -Ilib -Iimage -o ..\bin\wkhtmltopdf.exe pdf\*.cc lib\*.cc image\*.cc shared\*.cc
I tried in Visual Studio as well (assuming it wouldn't build, but I wanted to see if I could at least include the Qt dlls from there properly) and I am getting the same errors. Am I doing something wrong with the way I am compiling with g++? If I am linking with the Dlls properly then what is the proper way to use Qt functions from my code?
To clarify, I am not looking for how to properly use Qt. My question is: what is the proper way to use functions defined in any Dll from native C++ code? I apologize if this is a very basic question, but I'm unable to find a clear answer on Google and I don't have any experience with C++ and including third party libraries for use from C++ code.
DLLs can be used by dynamicly loading them and calling their used functions.
to call the exposed functions first define their syntax in the begining
suppose function is syntax is
BOOL MyFunction(int a,char* pszString)
then define syntax
#typedef BOOL (WINAPI *PMYFUNCTION)(int a,char* pszString)
then make object
PMYFUNCTION pfnMyFunction;
and get valid pointer by calling GetProcaddress after loadlibrarycall
HMODULE hlib= Loadlibrary("c:\\Mylib.dll");
if(hlib)
{ pfnMyFunction = (PMYFUNCTION)Getprocaddress(hlib,"MyFunction"); }
Hope this helps...

How should I use g++'s -finput-charset compiler option correctly in order to compile a non-UTF-8 source file?

I'm trying to compile a UTF-16BE C++ source file in g++ with -finput-charset compiler option but I'm always getting a bunch of errors. More details follow.
My environment(in CentOS Linux):
g++: 4.1.2
iconv: 2.5
Linux language(in Terminal): LANG="en_US.UTF-8"
My sample source file(stored in UTF-16BE encoding):
// main.cpp:
#include <iostream>
int main()
{
std::cout << "Hello, UTF-16" << std::endl;
return 0;
}
My steps:
I read the manual of g++ about the -finput-charset option. The g++ manual says:
-finput-charset=charset
Set the input character set, used for translation from the character set of the input file to the source character set used by
GCC. If the locale does not specify, or GCC cannot get this
information from the locale, the default is UTF-8. This can be
overridden by either the locale or this command line option.
Currently the command line option takes precedence if there’s a
conflict. charset can be any encoding supported by the system’s
"iconv" library routine.
Thus I entered the command as follows:
g++ -finput-charset=UTF-16BE main.cpp
and I got these errors:
In file included from main.cpp:1:
/usr/lib/gcc/i386-redhat-linux/4.1.2/../../../../include/c++/4.1.2/iostream:1:
error: stray ‘\342’ in program
/usr/lib/gcc/i386-redhat-linux/4.1.2/../../../../include/c++/4.1.2/iostream:1:
error: stray ‘\274’ in program
...(repeatedly, A LOT, around 4000+)...
/usr/lib/gcc/i386-redhat-linux/4.1.2/../../../../include/c++/4.1.2/iostream:1:
error: stray ‘\257’ in program
main.cpp: In function ‘int main()’:
main.cpp:5: error: ‘cout’ is not a member of ‘std’
main.cpp:5: error: ‘endl’ is not a member of ‘std’
The manual text suggests that the charset can be any encoding supported by 'iconv' routine, thus I guessed the compilation errors might be caused by my iconv library. I then tested the iconv:
iconv --from-code=UTF-16BE --to-code=UTF-8 --output=main_utf8.cpp main.cpp
A "main_utf8.cpp" file is generated as expected. I then tried to compile it:
g++ -finput-charset=UTF-8 main_utf8.cpp
Note that I specified the input-charset explicitly to see if I did anything wrong, but this time a "a.out" was generated without any errors. When I ran it, it could produce the correct output.
Finally...
I couldn't figure out where I did wrong. I searched in the web trying to find out some examples for this compiler option but I couldn't.
Please advise! Thanks!
Further edits:
Thanks, guys! Your replies are quick! Some updates:
When I said "UTF-16" I meant "UTF-16 + BOM". In fact I used UTF-16BE. I have updated the text above.
Some answers say the errors are caused by the non-UTF-16 header files. Here are my thoughts if this is the case: We'll always include some standard header files when writing a C/C++ project, right? Such as stdio.h or iostream. If the G++ compiler only deals with the encoding of the source files created by us but never with the source files in the standard library, then what does this -finput-charset option exist for??
Final edit:
At last, my solution is like this:
At the beginning, I changed the encoding of my source files to GB2312, as "Mr Lister" said below. This worked fine for a while, but later I found it not suitable for my situation because most of the other parts in the system still use UTF-8 for communication and interfaces, thus I must convert the encoding in many places... Not only an overhead of my work, it may also result in some performance decrease in my program.
Later I tried to convert all my source files to UTF-8 + BOM. In this way, Visual Studio in Windows could compile them happily but GCC in Linux would complain. I then wrote a shell script to remove the BOM, and before I want to compile my code with GCC, I run this script first.
Luckily, I don't have to build the code in Linux manually because TeamCity the continuous integration tool is used in my project to generate the build automatically. I could change the build steps in TeamCity to help me run this script before the daily build starts.
With this UTF-8 + BOM + script method, I decide not to edit my source code in Linux, because if I want to do so, I must make sure my code could build successfully before I commit it, which means I must run the script to remove the BOM before I build the code, which means SVN would report EVERY file is modified(BOM removed) thus make it very easy to mistakenly commit a wrong file. To solve this problem, I wrote another shell script to add the BOM back to the source files. Though I still don't edit my code very often in Linux, but when I really need to, I don't have to face the terribly long change list in the commit dialog.
Encoding Blues
You cannot use UTF-16 for source code files; because the header you are including, <iostream>, is not UTF-16-encoded. As #include includes the files verbatim, this means that you suddenly have an UTF-16-encoded file with a large chunk (approximately 4k, apparently) of invalid data.
There is almost no good reason to ever use UTF-16 for anything, so this is just as well.
Edit: Regarding problems with encoding support: The OSes themselves are not responsible for providing encoding support, this comes down to the compilers used.
g++ on Windows supports absolutely all of the same encodings as g++ on Linux, because it's the same program, unless whatever version of g++ you are using on Windows relies on a deeply broken iconv library.
Inspect your toolchain and ensure that all your tools are in working order.
As an alternative; don't use Chinese in the source files, but write them in English, using English-language literals, or simple TOKEN_STYLE_PLACEHOLDERs, using l10n and i18n to replace these in the running executable.
Threedit: -finput-charset is almost certainly a holdover from the days of codepages and other nonsense of the kind; however; an ISO-8859-n file will almost always be compatible with UTF-8 standard headers, however, see the reedit below.
Reedit: For next time; remember a simple mantra: "N'DUUH!"; "Never Don't Use UTF-8!"
I18N
A common solution to this kind of problem is to remove the problem entirely, by way of, for instance, gettext.
When using gettext, you usually end up with a function loc(char *) that abstracts away most of the translation tool specific code. So, instead of
#include <iostream>
int main () {
std::cout << "瓜田李下" << std::endl;
}
you would have
#include <iostream>
#include "translation.h"
int main () {
std::cout << loc("DEEPER_MEANING") << std::endl;
}
and, in zh.po:
msgid DEEPER_MEANING
msgstr "瓜田李下"
Of course, you could also then have a en.po:
msgid DEEPER_MEANING
msgstr "Still waters run deep"
This can be expanded upon, and the gettext package has tools for expansion of strings with variables and such, or you could use printf, to account for different grammars.
The Third Option
Instead of having to deal with multiple compilers with different requirements for file encodings, file endings, byte order marks, and other problems of the kind; it is possible to cross-compile using MinGW or similar tools.
This option requires some setup, but may very well reduce future overhead and headaches.
The error message says the problem is in the include files, so I presume what happens is that the include files are normal UTF-8, but the compiler wants to treat them as UTF-16 because of the compiler switch.
So I'm afraid the solution is to always convert the source to UTF-8 first; perhaps in the makefile. Or to find a solution that doesn't contain include files in other encodings...
Edit:
Maybe a GB encoding would work, if and only if none of the system source files contain any non-ASCII characters. Then you could tell the compiler they were GB encoded without problem.
This does not work because the compiler will also try to read the header files as UTF-16, which they are not.
UTF-16 is not an encoding for bytes. It's an encoding where your basic storage unit is 16 bits large.
When you want to store UTF-16 in a byte sequence you have to choose between UTF-16BE and UTF-16LE.

Open file in TagLib with Unicode chars in filename

I am quite new to the C++ 'area' so I hope this will not be just another silly 'C++ strings' question.
Here is my problem. I want to integrate TagLib (1.5, 1.6 as soon as I manage to build it for Windows) into an existing Windows MFC VS2005 project. I need it to read audio files metadata (not write).
The problem is that the program stores the input file names using CString(), and it has the Unicode option turned on (so the default chars are "wchar_t"). The reason for this (I think, the project was started by someone else) is that some of the 'input' file names could contain Unicode charaters (for example, Japanse or Arabic characters).
For example, the file path is something like "d:\docs\audio_test\stragecharڝhere.mp3", but I get it with:
CString fpath = tmpFile->GetFilePath();
Now.. if I try to do:
TagLib::FileRef f(fpath.GetBuffer(0));
fpath.ReleaseBuffer();
I get something like:
unresolved external symbol
"__declspec(dllimport) public:
__thiscall TagLib::FileName::FileName(wchar_t
const *)"
If I try something like:
TagLib::FileRef f(reinterpret_cast<char*>(fpath.GetBuffer(0)));
fpath.ReleaseBuffer();
I get rid of the compilation errors, but "f" is an invalid pointer/object.. and when I try reading a tag, I receive an assert failed.
So, can anyone give me some pointers on how should I pass that CString, in it's Unicode form, to the TagLib ?
Update: TagLib address: http://developer.kde.org/~wheeler/taglib.html
Thank you,
Alex
It's possible that the problem is caused by the issue described here. Basically MSVC has an option to treat the wchar_t type differently, which causes the compiled library is not binary compatible with an application compiled without that option. Unfortunately the CMakeLists.txt build file enables the /Zc:wchar_t- option by default. I'd try to edit the file, remove the option and re-compile TagLib. Ideally version 1.6, as it contains a lot of bug fixes.
I missed something essential when I first read your post, so here is another, new and improved answer:
The error comes from the linker, not the compiler. It thus seems that TagLib::FileName does have a ctor taking wchar_t const*, but the problem is that you don't link with the library implementing it, or link with a version of the library that does not include it.
IIUC, this library comes from the Linux world (where file names are expressed as char arrays), and was later ported to Windows (where file names are expressed as wchar_t arrays). The FileName ctor taking a wchar_t array is thus probably conditionally compiled on Windows (i.e., inside #ifdef _WIN32 or something similar), and the library you are linking with (if you are linking the library) was not compiled with the same preprocessor defines.