Embedding cg shaders in C++ GPGPU library - c++

I'm writing a GPGPU Fluid simulation, which runs using C++/OpenGL/Cg. At the moment, the library requires that the user specify a path to the shaders, which is will then read it from.
I'm finding it extremely annoying to have to specify that in my own projects and testing, so I want to make the shader contents linked in with the rest.
Ideally, my .cg files would still be browsable seperately, but a post-build step or pre-processor directive would include it in the source when required.
To make things slightly more annoying, I have a "utils" shader file, which contains functions that are shared among things (like converting 3d texture coords to the 2d atlas equivalent).
I'd like a solution that's cross platform if possible, but it's not so big a deal, as it is currently windows-only. My searches have only really turned up objcopy for linux, but using that for windows is less than ideal.
If it helps, the project is available at http://code.google.com/p/fluidic

You mean you want the shaders embedded as strings in your binary? I'm not aware of any cross-platform tools/libraries to do that, which isn't that surprising because the binaries will be different formats.
For Windows it sounds like you want to store them as a string resource. You can then read the string using LoadString(). Here's how to add them, but it doesn't look like you can link them to a file.
A particularly hacky but cross-platform solution might be to write a script to convert your shaders into a string constant in a header. Then you can just #include it in your code.
I.e. you have the file myshader.shader which contains:
int aFunction(int i, int j)
{
return i/j;
}
And you have a build step that creates the file myshader.shader.h which looks like:
const char[] MYSHADER_SHADER =
"int aFunction(int i, int j)"
"{"
" return i/j;"
"}";
Then add #include "myshader.shader.h" to your code.
Very hacky, but I see no reason why it wouldn't work (except for maybe length/space limits on string literals).
Update: With the release of G++ 4.5 it supports C++0x raw string literals. These can contain new lines 4. I haven't tested it but you should be able to do something like this:
const char[] MY_SHADER = R"qazwsx[
#include "my_shader.c"
]qazwsx";
I haven't tested it though.

Related

How to properly navigate directory paths in C++

I'm working on a solution within Visual Studio. It currently has two projects.
I will represent Directories or folders with capitals letters, and filenames will be all lower case. My solution structure is as follows:
SolutionDir
ProjectLib
source files
Shaders
shader files
ProjectApp
source files
x64
Debug
app.exe // debug build
Release
app.exe // release build
Within ProjectLib I have a function to open and read my Shader files. Here is what my function looks like:
std::vector<char> VRXShader::readFile(std::string_view shadername) {
std::string filename = std::string("Shaders/");
filename.append(shadername);
std::ifstream file(filename.data(), std::ios::ate | std::ios::binary);
if (!file.is_open()) {
throw std::runtime_error("failed to open file!");
}
size_t fileSize = static_cast<size_t>(file.tellg());
std::vector<char> buffer(fileSize);
file.seekg(0);
file.read(buffer.data(), fileSize);
file.close();
return buffer;
}
This function is being called within my VRXDevices::createPipeline function and here is the relevant code:
void VRXDevices::createPipeline(
VkDevice device, VkExtent2D swapChainExtent, VkRenderPass renderPass,
const std::vector<std::string_view>& shaderNames,
VkPipelineLayout& pipelineLayout, VkPipeline& pipeline
) {
std::vector<std::vector<char>> shaderCodes;
shaderCodes.resize(shaderNames.size());
for (auto& name : shaderNames) {
auto shaderCode = VRXShader::readFile(name.data());
}
// .... more code
}
The names are being created and passed to this function from my VRXEngine::initVulkan function which can be seen here:
void VRXEngine::initVulkan(
std::string_view app_name, std::string_view engine_name,
glm::ivec3 app_version, glm::ivec3 engine_version
) {
//... code
std::vector<std::string_view> shaderFilenames{ "vert.spv", "frag.spv" };
VRXDevices::createPipeline(device_, swapChainExtent_, renderPass_, shaderFilenames, pipelineLayout_, graphicsPipeline_);
}
I'm using just the name of the shader files such as vert.spv, frag.spv, geom.spv etc. I'm not including the paths here because these will be used as the key to a std::map<string_view, object>. So I'm passing a vector of these names from my ::initVulkan function into ::createPipeline().
Within ::createPipeline() is where ::readFile() is being called passing in the string_view.
Now as for my question... within ::readFile() I'm creating a local string and trying to initialize it with the appropriate path... then append to it the string_view for the shader's filename as can be seen from these two lines...
std::string filename = std::string("Shaders/");
filename.append(shadername);
I'm trying to figure out the appropriate string to initialize filename with... Shaders/ will be a part of the name, but it's not finding the file and I'm not sure what the appropriate prefix should be...
My working directories within both projects are as follows:
ProjectApp -> $(SolutionDir)x64/Release AND $(SolutionDir)x64/Debug
ProjectLib -> $(SolutionDir)x64/Release AND $(SolutionDir)x64/Debug
So I need to go back 2 directories then into VRX Engine/Shader...
What is the correct string value for navigating back directories?
Would I initialize filename with "../../VRX Engine/Shaders/" or is it "././" also, should I have quotes around VRX Engine since there is a space in the folder name? What do I need to initialize filename with before I append the shader name to it?
How to properly navigate directory paths in C++
It depends on which C++ standard your implementation claims to be compliant with.
Or else which additional libraries can you use.
C++ is useful on computers without directories (e.g. inside some operating system kernel coded in C++ and compiled with GCC, see OSDEV for examples).
Look on en.cppreference.com for details.
Licensing constraints could matter when using extra open source libraries.
If your implementation is C++17 compliant (in a "hosted" not "freestanding" way), use the std::filesystem part of the standard library.
If your operating system supports the Qt or POCO frameworks and you are allowed to use them (e.g. on C++11), you could use appropriate APIs. So QDir and related classes with Qt, Poco::Path and related classes with POCO.
Perhaps you want to code just for the WinAPI. Then read its documentation (I never coded on Windows myself, just on POSIX or Unix -e.g. Linux- and MSDOS....).
I was originally initializing my local temp string properly with "../../VRX Engine/Shaders/" before appending the string_view to it to be able to open the file. This was actually correct, but because it didn't initially work, I was assuming that it was wrong.
The correct string value for going back one directory should be "../" at least on Windows, I'm not sure about Linux, Mac, Android, etc...
My problem wasn't with the string at all, it pertained to settings within my projects. Within my project that builds into an executable, I had its working directory set to $(SolutionDir)x64/Debug and $(SolutionDir)x64/Release respectively which is correct for my solutions structure.
The issue was within my Engine project that is being built as a static library. Within its settings for its working directory, I had forgotten to modify both of the Debug and Release build options... These were still set to the default values of Visual Studio which I believe is (ProjectDir). Once I changed these to $(SolutionDir)x64/Debug and $(SolutionDir)x64/Release to match that of my ApplicationProject, I was able to open and read the contents of the files.

Initialize tesseract without any external resources (languages/dictionaries)

I am currently writing a C++ program that should read hex data from JPEG images. I have to compile it into one single windows executable without any external resources (like the "tessdata" directory or config files). As I am not reading any words or sentences, I don't need any dictionaries or languages.
My problem is now that I could not find a way to initialize the API without any language files. Every example uses something like this:
tesseract::TessBaseAPI api;
if (api.Init(NULL, "eng")) {
// error handling
return -1;
}
// do stuff
I also found that I can call the init function without language argument and with OEM_TESSERACT_ONLY:
if(api.Init(NULL, NULL, tesseract::OcrEngineMode::OEM_TESSERACT_ONLY)) {
// ...
}
This should disable the language/dictionary, but NULL just defaults to "eng". It seems like tesseract still wants a language file to initialize and will disable it afterwards.
This also seems to be the case for any other solutions I found so far: I always need .traineddata files to initialize the api and can disable them afterwards or using config files.
My question is now:
Is there any way to initialize the tesseract API in C++ using just the executable and no other resource files?
No. Tesseract always needs some language (default is eng) + osd (.traineddata) files. Without language data file tesseract is useless.
Your post seems that you made several wrong assumptions (e.g. about OEM_TESSERACT_ONLY), so maybe if you describe what you try to achieve with tesseract you can get better advice.

SDL embed image inside program executable

Is it possible to embed an image within a program using SDL which can be used at run time.
For example, I have a program which brings up a splash screen on startup containing the logo and copyright information. Rather than having this image in a bitmap file and using SDL_LoadBMP to load it to a SDL_Surface. I would like to have the image embedded in the program binary, to stop someone potentially changing the splash image and copyright name.
Does anyone have any suggestions on ways to do this? Example code would be great.
Embedding a file in an executable is easy but there are some gotchas, there are several ways to do it including some portable and non-portable ways.
Using #embed
This will reportedly be part of C23. It may be on track to appear in C++26 as well. Check whether your compiler supports this feature. In the future, this may be the most portable and straightforward way to embed binary data.
static const unsigned char IMAGE_DATA[] = {
#embed "myimage.bmp
};
See WG14 n2592 for the feature proposal.
Advantages: simplest, easiest
Disadvantages: your compiler probably doesn’t support this yet
Convert the image to C code
Write a script to convert the image to a constant array in C. The script would look something like this in Python:
#!/usr/bin/env python3
print("static const unsigned char IMAGE_DATA[] = {{{}}};".format(
",".join(str(b) for b in open("myimage.bmp", "rb").read())))
Just pipe the output to a *.h file and include that file from one other file. You can get the size of the file with sizeof(IMAGE_DATA).
Advantages: portable
Disadvantages: requires Python to be installed, does not work if array is too large for compiler, requires adding a custom step to the build system
Convert the image to an object file
This is more platform-dependent. On platforms with GNU binutils toolchains (e.g. Linux) you can use objcopy, I think bin2obj works on Microsoft toolchains.
Advantages: works everywhere
Disadvantages: non-portable, requires adding a custom step to the build system, the custom step might be tricky to get right
On GNU binutils toolchains, with objcopy
The objcopy program lets you specify binary as the input format, but then you need to specify the architecture explicitly... so you will have to modify the command for i386 and x64 versions of your executable.
$ objcopy --input binary --output elf32-i386 --binary-architecture i386 \
myimage.bmp myimage.o
You can get the data from C by using the following declarations:
// Ignore the fact that these are char...
extern char _binary_myimage_bmp_start, _binary_myimage_bmp_end;
#define MYIMAGE_DATA ((void *) &_binary_myimage_bmp_start)
#define MYIMAGE_SIZE \
((size_t) (&_binary_myimage_bmp_end - &_binary_myimage_bmp_start))
Use an assembler directive
Paradoxically, embedding a static file is fairly easy in assembler. Assemblers often have directives like .incbin (which works with GAS and YASM).
Advantages: works everywhere
Disadvantages: non-portable, assembler syntax is different between platforms
(Windows) Embed the file as a resource
On Windows, you can embed resources in an EXE and then get the resources using library calls.
Advantages: probably easiest if you are on Windows
Disadvantages: only works on Windows
You can export the image as .xpm format (in gimp) and include it to your code. But you will need SDL_Image.h to load it as SDL_Surface.
As it is in this doc, is really simple:
//To create a surface from an XPM image included in C source, use:
SDL_Surface *IMG_ReadXPMFromArray(char **xpm);
A example in C/C++:
#include <SDL/SDL.h>
#include "test.xpm"
#include <SDL/SDL_image.h>
SDL_Surface *image;
SDL_Surface *screen;
int main(int argc, char **argv)
{
SDL_Init(SDL_INIT_EVERYTHING);
screen = SDL_SetVideoMode(800,600,32,SDL_SWSURFACE);
image = IMG_ReadXPMFromArray(test_xpm); //the .xpm image is a char array. "test_xpm" is the name of the char array
SDL_Rect offset;
offset.x = 0;
offset.y = 0;
SDL_BlitSurface(image,NULL,screen,&offset);
SDL_Flip(screen);
SDL_Delay(5000);
return 0;
}
I hope this helps.
With gimp you can save a image as c code.

Accessing resources from program in Debian package structure

I've made a DEB package of an C++ app that I've created. I want this app to use resources in the "data" directory, which, in my tests (for convenience), is in the same location that the program binary, and I call it from inside the code by its relative path. In the Debian OS there are standard locations to put the data files in (something like /usr/share/...), and other location to put the binaries in (probably /usr/bin). I'd not like to put the paths hard-coded in my program, I think its a better practice to access an image by "data/img.png" than "/usr/share/.../data/img.png". All the GNU classic programs respect the directories structure, and I imagine they do it in a good manner. I tried to use dpkg to find out the structure of the apps, but that didn't help me. Is there a better way that I'm doing to do this?
PS: I also want my code to be portable to Windows (cross-platform) avoiding using workarounds like "if WIN32" as much as possible.
In your Debian package you should indeed install your data in /usr/share/. When accessing your data, you should use the XDG standard, which states that $XDG_DATA_DIRS is a colon-separated list of data directories to search (also, "if $XDG_DATA_DIRS is either not set or empty, a value equal to /usr/local/share/:/usr/share/ should be used.").
This is not entirely linux specific or debian specific. I think is has something to do with Linux Standard Base or POSIX specifications maybe. I were unable to discover any specification quickly enough.
But you should not use some "base" directory and subdirectories in it for each type of data. Platform dependent code should belong into /usr/lib/programname, platform independent read-only data into /usr/share/programname/img.png. Data changed by application in /var/lib/programname/cache.db. Or ~/.programname/cache.db, depends what kind of application it is and what it does. Note: there is no need to "data" directory when /usr/share is already there for non-executable data.
You may want check http://www.debian.org/doc/manuals/developers-reference/best-pkging-practices.html if packaging for Debian. But it is not resources like in adroid or iphone, or windows files. These files are extracted on package install into target file system as real files.
Edit: see http://www.debian.org/doc/packaging-manuals/fhs/fhs-2.3.html
Edit2: As for multiplatform solution, i suggest you make some wrapper functions. On windows, it depends on installer, usually programs usually have path in registry to directory where they are installed. On unix, place for data is more or less given, you may consider build option for changing target prefix, or use environment variable to override default paths. On windows, prefix would be sufficient also, if it should not be too flexible.
I suggest some functions, where you will pass name of object and they will return path of file. It depends on toolkit used, Qt library may have something similar already implemented.
#include <string>
#ifdef WIN32
#define ROOT_PREFIX "c:/Program Files/"
const char DATA_PREFIX[] = ROOT_PREFIX "program/data";
#else
#define ROOT_PREFIX "/usr/"
/* #define ROOT_PREFIX "/usr/local/" */
const char DATA_PREFIX[] = ROOT_PREFIX "share/program";
#endif
std::string GetImageBasePath()
{
return std::string(DATA_PREFIX) + "/images";
}
std::string GetImagePath(const std::string &imagename)
{
// multiple directories and/or file types could be tried here, depends on how sophisticated
// it should be.
// you may check if such file does exist here for example and return only image type that does exist, if you can load multiple types.
return GetImageBasePath() + imagename + ".png";
}
class Image;
extern Image * LoadImage(const char *path);
int main(int argc, char *argv[])
{
Image *img1 = LoadImage(GetImagePath("toolbox").c_str());
Image *img2 = LoadImage(GetImagePath("openfile").c_str());
return 0;
}
It might be wise to make class Settings, where you can initialize platform dependent root paths once per start, and then use Settings::GetImagePath() as method.

Is it possible to store binary files inside an exe

Is it possible to do this: (for educational purpose).
suppose I have a image file "image.jpg"
I want to create a program when it executes it should create this image. That means the data of the image is stored in the exe. Is this possible to do?
Something like this: link the image file from resource.rc then tell the compiler to get the data and store it (something like this unsigned char data_buffer[]="binary data of the image" then when the program is executed I can write this data to a file)
(I'm using C++ with mingw compiler)
Any help is highly appreciated.
There are several options:
1) Add it as a byte array in a source file. It is trivial to write an auxiliary program that reads the bytes from the files and writes the C source. E.g.:
data_jpg.c:
unsigned char data_jpg[] = {1,2,3... };
data_jpg.h:
extern char data_jpg[];
const size_t data_jpg_size = 1000;
2) Add it as a binary resource to the executable. You said "exe", did you? So you are likely on Windows. Window EXE files can have binary resources, that can be located using the resource API. See the FindResource, LoadResource and GlobalLock, functions.
resource.rc
ID_DATA_JPG FILE "data.jpg"
3) Convert the binary file directly into a OBJ file and link it into the executable. In the old good days of turbo-c used to be a BINOBJ tool for that. And GNU tools can do it, AFAIk, but with MS tools, I really cannot tell.
With a PE file, you can add data(include bin data) to the PE file's tail as your resource. You just remember the PE file's size. But I'm not sure of that whether you need change the PE's checksum. And use VC++ Compiler to embed resources would be pretty much easy.