I need to access a one dimensional big(~2MB) buffer from a shader. However, I don't know which type of OpenGL buffer I should use. I'm going to store floats(16F) and unsigned integers (16UI). My data will be like an struct:
struct{
float d; //16F
int a[7]; //or a1,a2,a3,a4,a5,a6,a7; //7x16UI
}
I read about buffer texture and other kind of buffers(Passing a list of values to fragment shader), but it only works for one type of data (float or int), not both. I could use two buffers, but I think this won't be cache friendly nor easy.
Related
Im trying to pass a 2D float array to a constant buffer:
//In the shader:
cbuffer myBuffer
{
other buffer elements
.
.
float myArray[16][16];
};
//In the CPU:
struct myBuffer_struct
{
other buffer elements
.
.
float myArray[16][16];
};
But im having a lot of problems dealing with the padding. I tried using
float4[size/4][size]
in my cbuffer and a lot of other type combinations but I cant access to my array by indexation in any way. What is the proper way to do this?
Thank you.
I've had this issue and it comes down to basically the alignment of the buffer. Your HLSL cbuffer definition most definitely will be padding differently to what you have defined in your struct.
The alignment probably along 16 byte (4 floats) alignment. In my code, I was writing 4 floats out into a buffer. Like this below, as the array alignment was different in the cbuffer.
for (int i = 0; i < 8; i++)
{
stream.Write<float>(m_waveLengths[i] );
stream.Write<float>(m_waveSpeeds[i] );
stream.Write<float>(m_amplitudes[i] );
stream.Write<float>(m_steepness[i]);
}
To read this, I used a float4 array definition.
// hlsl definition
float4 Wave[8];
I then referenced the relevant item as Wave[0].x, Wave[0].y, Wave[0].z, Wave[0].w
The memory alignment would make the buffer 4 times bigger if I didn't pack it like this. This is because in the HLSL code, the buffer definition seems to of aligned each element of the array along 16 byte boundries (4 x floats). So instead, I interweaved my 4 arrays into 1 array and used the properties of float4 to reference it.
because the alignment of float waveLengths[8] would of meant that I would have to write it into the buffer like this:
for (int i = 0; i < 8; i++)
{
stream.Write<float>(m_waveLengths[i] );
stream.Write<float>(0.0f);
stream.Write<float>(0.0f);
stream.Write<float>(0.0f);
}
For some reason (and I am probably not setting a certain HLSL compiler directive), using arrays in the Cbuffer had some quirks where it would pad each element to a 16 byte boundary.
So, for your float myArray[16][16], I would assume that you look at the alignment, you may have to write the buffer for this out in a similar manner, padding out 12 bytes after each element in the array. I'm sure someone will respond with correct compiler directive to get rid of this quirk, I just solved this a while ago and your problem looks similar to what I had.
I'm making a voxel engine and I can render a chunk. I'm using instanced rendering, meaning that I can render all of the chunk with a single draw call. Every blocks of a chunk has a single int (From 0 to 4095) that defines his block type (0 for air, 1 for dirt, etc...). I wanna be able to render my block by applying the good texture in my fragment shader. My chunk contains a tri-dimensionnal array :
uint8_t blocks[16][16][16]
The problem is that I can't find a way to send my array of int to the shader. I tried using a VBO but it makes no-sense (I didn't get any result). I also tried to send my array with glUniform1iv() but I failed.
Is it possible to send an array of int to a shader with glUniformX() ?
In order to prevent storing big data, can I set a byte (uint8_t) instead of int with glUniformX() ?
Is there a good way to send that much data to my shader ?
Is instanced drawing a good way to draw the same model with different textures/types of blocks.
For all purposes and intents, data of this type should be treated like texture data. This doesn't mean literally uploading it as texture data, but rather that that's the frame of thinking you should be using when considering how to transfer it.
Or, in more basic terms: don't try to pass this data as uniform data.
If you have access to OpenGL 4.3+ (which is a reasonably safe bet for most hardware no older than 6-8 years), then Shader Storage Buffers are going to be the most laconic solution:
//GLSL:
layout(std430, binding = 0) buffer terrainData
{
int data[16][16][16];
};
void main() {
int terrainType = data[voxel.x][voxel.y][voxel.z];
//Do whatever
}
//HOST:
struct terrain_data {
int data[16][16][16];
};
//....
terrain_data data = get_terrain_data();
GLuint ssbo;
GLuint binding = 0;//Should be equal to the binding specified in the shader code
glGenBuffers(1, &ssbo);
glBindBuffer(GL_SHADER_STORAGE_BUFFER, ssbo);
glBufferData(GL_SHADER_STORAGE_BUFFER, GLsizeiptr size, data.data, GLenum usage);
glBindBufferBase(GL_SHADER_STORAGE_BUFFER, binding, ssbo);
glBindBuffer(GL_SHADER_STORAGE_BUFFER, 0);
Any point after this where you need to update the data, simply bind ssbo, call glBufferData (or your preferred method for updating buffer data), and then you're good to go.
If you're limited to older hardware, you do have some options, but they quickly get clunky:
You can use Uniform Buffers, which behave very similarly to Shader Storage Buffers, but
Have limited storage space (65kb in most implementations)
Have other restrictions that may or may not be relevant to your use case
You can use textures directly, where you convert the terrain data to floating point values (or use as integers, if the hardware supports integer formats internally), and then convert back inside the shader
Compatible with almost any hardware
But requires extra complexity and calculations in your shader code
I do second the approach as laid out in #Xirema's answer, but come to a slightly different recommendation. Since your original data type is just uint8_t, using an SSBO or UBO directly will require to either waste 3 bytes per element or to manually pack 4 elements into a single uint. From #Xirema's answer:
For all purposes and intents, data of this type should be treated like texture data. This doesn't mean literally uploading it as texture data, but rather that that's the frame of thinking you should be using when considering how to transfer it.
I totally agree to that. Hence I recommend the use of a Texture Buffer Object (TBO), (a.k.a. "Buffer Texture").
Using glTexBuffer() you can basically re-interpret a buffer object as a texture. In your case, you can just pack the uint8_t[16][16][16] array into a buffer and interpret it as GL_R8UI "texture" format, like this:
//GLSL:
uniform usamplerBuffer terrainData;
void main() {
uint terrainType = texelFetch(terrainData, voxel.z * (16*16) + voxel.y * 16 + voxel.x).r
//Do whatever
}
//HOST:
struct terrain_data {
uint8_t data[16][16][16];
};
//....
terrain_data data = get_terrain_data();
GLuint tbo;
GLuint tex;
glGenBuffers(1, &tbo);
glBindBuffer(GL_TEXTURE_BUFFER, tbo);
glBufferData(GL_TEXTURE_BUFFER, sizeof(terrain_data), data.data, usage);
glGenTextures(1, &tex);
glBindTexture(GL_TEXTURE_BUFFER, tex);
glTexBuffer(GL_TEXTURE_BUFFER, GL_R8UI, tbo);
Note that this will not copy the data to some texture object. Accessing the texture means directly accessing the memory of the buffer.
TBOs also have the advantage that they are available since OpenGL 3.1.
I am trying to use a buffer in a compute shader like this:
layout (binding = 1, std430) writeonly buffer bl1
{
uint data[gl_WorkGroupSize.x * gl_NumWorkGroups.x * gl_NumWorkGroups.y];
};
but I get the following error (because of using gl_NumWorkGroups for the size):
Array size must be a constant integer expression
How can I work around this?
Stop putting in a length at all:
layout (binding = 1, std430) writeonly buffer bl1
{
uint data[];
};
This is a feature unique to SSBOs. And you can only have one unsized array in an SSBO, and it must be the last member in the interface block. The size of data will be computed based on the size of the buffer object range you bind to that binding point. So if you bind 32KB of buffer space, you will get 8K of items (the size of a uint is 4 bytes).
At runtime, your shader can use gl_WorkGroupSize.x * gl_NumWorkGroups.x * gl_NumWorkGroups.y to compute the length of data. Alternatively, just use data.length() to get the length of the buffer that the user gave you. Alternatively... you don't need to explicitly know the length, depending on how you use it.
As long as your OpenGL buffer binding code uses a buffer with enough memory for your dispatch count and work group size, you're fine.
I'm trying to visualize very large point cloud (700 mln points) and on glDrawArrays call debugger throws access violation writing location exception. I'm using the same code to render smaller clouds (100 mln) and everything works fine. I also have enough RAM memory (32GB) to store the data.
To store point cloud I'm using std::vector<Point3D<float>> where Point3D is
template <class T>
union Point3D
{
T data[3];
struct{
T x;
T y;
T z;
};
}
Vertex array and buffer initialization:
glBindVertexArray(pxCloudHeader.uiVBA);
glBindBuffer(GL_ARRAY_BUFFER, pxCloudHeader.xVBOs.uiVBO_XYZ);
glBufferData(GL_ARRAY_BUFFER, pxCloudHeader.iPointsCount * sizeof(GLfloat) * 3, &p3DfXYZ->data[0], GL_STREAM_DRAW);
glVertexAttribPointer((GLuint)0, 3, GL_FLOAT, GL_FALSE, 0, 0);
glEnableVertexAttribArray(0);
glBindVertexArray(0);
Drawing call:
glBindVertexArray(pxCloudHeader.uiVBA);
glDrawArrays(GL_POINTS, 0, pxCloudHeader.iPointsCount); // here exception is thrown
glBindVertexArray(0);
I also checked if there was OpenGL error thrown but I haven't found any.
I suspect your problem is due to the size of GLsizeiptr.
This is the data type used to represent sizes in OpenGL buffer objects, and it is typically 32-bit.
700 million vertices * 4-bytes per-component * 3-components = 8,400,000,000 bytes
There is a serious issue with trying to allocate that many bytes in GL if it is using 32-bit pointers:
8400000000 & 0xFFFFFFFF = 4,105,032,704 (half as many bytes as you actually need)
If sizeof (GLsizeiptr) on your implementation is 4 then you will have no choice but to split your array up. A 32-bit GLsizeiptr only allows you to store 4 contiguous GiB of memory, but you can work around this if you use 3 single-component arrays instead. Using a vertex shader you can reconstruct these 3 separate (small enough) arrays like so:
#version 330
layout (location = 0) in float x; // Vertex Attrib Ptr. 0
layout (location = 1) in float y; // Vertex Attrib Ptr. 1
layout (location = 2) in float z; // Vertex Attrib Ptr. 2
void main (void)
{
gl_Position = vec4 (x,y,z,1.0);
}
Performance is going to be awful, but that is one way to approach the problem with minimal effort.
By the way, the amount of system memory here (32 GiB) is not your biggest issue. You should be thinking in terms of the amount of VRAM on your GPU because ideally buffer objects are designed to be stored on the GPU. Any part of the buffer object that is too large to be stored in GPU memory will have to be transferred over the PCIe (these days) bus when it is used.
You could draw the data in smaller batches. While there is no predefined upper limit for the size of a buffer, storing 8 GBytes of data in a single buffer is a lot. I'm not very surprised that something would blow up.
I would probably start with storing something like 1 million, or at most a few million, points in each buffer. Then use a pool of buffers with this fixed size, enough to accommodate all your data points.
This might even be beneficial for you performance, because it allows you start submitting draw calls before copying all your data into buffers. This will give you better overlap between CPU and GPU work.
With the amount of data you are shuffling around, you may also want to look into using glMapBuffer()/glUnmapBuffer() instead of glBufferData(). This generally avoids one copy operation for the data.
i'm approaching c++ with some basic computer graphics.
pixels data is usually represented as :
unsigned char *pixels
and an unsigned char is good because is a value between 0 and 255 (256 = 2^8 because a char is 2 byte and 1 byte is 8 bit?). and this is good because in RGB color are represented with a number between 0 and 255.
but.. i understand this as a monchromatic image, in a normal image i have RGB, i would have 3 array of unsiged char, one for red, one for green, one for blue. something like:
unsigned char *pixels[3]
but i never found something similar for RGB pixels data
RGB images are usually stored in interleaved order (R1, G1, B1, R2, G2, B2, ...), so one pointer (to R1) is enough.
This makes it a bit harder to address individual pixels: pixel with index N is stored at pixels[3*N+0], pixels[3*N+1] and pixels[3*N+2] instead of just red[N], green[N], blue[N].
However, this has the advantage of allowing faster access: less pointers lead to easier programs, improving their speed; interleaved order also makes memory caching more effective.
unsigned char *pixels[3];
declares an array of three pointers to unsigned char. I'm not sure if that's what you wanted.
There are several different ways to represent pixels. The simplest is probably something like:
struct Pixel
{
unsigned char red;
unsigned char green;
unsigned char blue;
};
But you may have to (or want to) conform to some external format. Another frequent possibility is to put all three colors in a uint32_t. Also, in some graphic systems, there may be a fourth element, and alpha, representing transparency.
Really whenever you refer to a block of bytes, it's going to be of type unsigned char* because of the fact that unsigned char by the C-specification has no padding in the type itself (i.e., every bit is used for a value in the byte, and there are no padded bits that are not used), and pixel-data is going to be some block of X bytes with no padding (at least not internal padding ... there may be padding at the end of the buffer for alignment purposes). It will also most likely be allocated on the heap somewhere. So no matter if it's going to be monochrome, color-data, etc., you will often find that a pixel buffer will be pointed to via an unsigned char pointer, and you may then cast it to some struct like James mentioned in order to easily access the pixel information. Other times you may have to index into the buffer like anatolyg mentions. But in the end, a buffer of pixels is just a buffer of data, and a general buffer of data bytes should be accessed in C/C++ using type unsigned char*.
With *pixels[3] you've got separate arrays for the three colour components, whereas in files the three colour components for a single pixel are stored together. It also means you can use a single fread()/fwrite() for the whole block of image data,