OpenGL convert RGBA byte pixels to image integers array - c++

I read pixels from OpenGL texture 2D into byte array (unsigned char) as it is usually done.But now I need to convert it into image array (of Integers I suppose) to have the layout and pixel range of the images loaded from CPU for reverse process.
My question is - is it enough just to to do :
glGetTexImage(GL_TEXTURE_2D,0,GL_RGBA,GL_UNSIGNED_INT,bytes);
instead of :
glGetTexImage(GL_TEXTURE_2D,0,GL_RGBA,GL_UNSIGNED_BYTE,bytes);
and then iterate over each integer and convert it from 0-1 range to 0-255?
I haven't really found any example doing such a conversion without using 3 party image libs.
If I do this :
size_t lenght=_viewWidth * _viewHeight ;
GLubyte *bytes=(GLubyte*)malloc(lenght);
/////////////// read pixels from tex /////////////////////
glBindTexture(GL_TEXTURE_2D,tex);
glGetTexImage(GL_TEXTURE_2D,0,GL_BGR,GL_UNSIGNED_BYTE,bytes);
uint8_t Rc, Gc, Bc;
for(x = 0; x < lenght; x+=3)
{
Bc = *bytes + x;
Gc = *bytes + x + 1;
Rc = *bytes + x + 2;
}
Is Rc , Gc and Bc going to be in the 0-255 range ?

When OpenGL loads a texture, it will convert the incoming pixels into the format provided as the internal format as specified in the glTexImage*() call. This operation may include a mapping step from the pixel format (glTexImage*()'s third parameter) to the internal format, and often includes mapping into one of the ranges [0,1] or [-1,1], and then onto the range for the internal format for each component. For example, a pixel format of GL_FLOAT, and an internal format of GL_RGBA8, will cause the input values to be mapped from the range [0,1] into the range [0,255].
When you retrieve the texels using glGetTexImage(), the process is done in reverse, and so the output pixel values (per component) will be in the range of the specified output type (e.g., GL_UNSIGNED_INT in your case). The range for unsigned ints is [0,232-1], so that will be the range of values returned in your integer image array. If you need those values in a different range (e.g., GL_UNSIGNED_BYTES), then you would need to manually convert values into the range you need.
Personally, if one of the data types OpenGL can return matches the range of values you need, try to use that type.

Related

Unable to create image from compressed texture data (S3TC)

I've been trying to load compressed images with S3TC (BC/DXT) compression in Vulkan, but so far I haven't had much luck.
Here is what the Vulkan specification says about compressed images:
https://www.khronos.org/registry/dataformat/specs/1.1/dataformat.1.1.html#S3TC:
Compressed texture images stored using the S3TC compressed image formats are represented as a collection of 4×4 texel blocks, where each block contains 64 or 128 bits of texel data. The image is encoded as a normal 2D raster image in which each 4×4 block is treated as a single pixel.
https://www.khronos.org/registry/vulkan/specs/1.0/xhtml/vkspec.html#resources-images:
For images created with linear tiling, rowPitch, arrayPitch and depthPitch describe the layout of the subresource in linear memory. For uncompressed formats, rowPitch is the number of bytes between texels with the same x coordinate in adjacent rows (y coordinates differ by one). arrayPitch is the number of bytes between texels with the same x and y coordinate in adjacent array layers of the image (array layer values differ by one). depthPitch is the number of bytes between texels with the same x and y coordinate in adjacent slices of a 3D image (z coordinates differ by one). Expressed as an addressing formula, the starting byte of a texel in the subresource has address:
// (x,y,z,layer) are in texel coordinates
address(x,y,z,layer) = layerarrayPitch + zdepthPitch + yrowPitch + xtexelSize + offset
For compressed formats, the rowPitch is the number of bytes between compressed blocks in adjacent rows. arrayPitch is the number of bytes between blocks in adjacent array layers. depthPitch is the number of bytes between blocks in adjacent slices of a 3D image.
// (x,y,z,layer) are in block coordinates
address(x,y,z,layer) = layerarrayPitch + zdepthPitch + yrowPitch + xblockSize + offset;
arrayPitch is undefined for images that were not created as arrays. depthPitch is defined only for 3D images.
For color formats, the aspectMask member of VkImageSubresource must be VK_IMAGE_ASPECT_COLOR_BIT. For depth/stencil formats, aspect must be either VK_IMAGE_ASPECT_DEPTH_BIT or VK_IMAGE_ASPECT_STENCIL_BIT. On implementations that store depth and stencil aspects separately, querying each of these subresource layouts will return a different offset and size representing the region of memory used for that aspect. On implementations that store depth and stencil aspects interleaved, the same offset and size are returned and represent the interleaved memory allocation.
My image is a normal 2D image (0 layers, 1 mipmap), so there's no arrayPitch or depthPitch. Since S3TC compression is directly supported by the hardware, it should be possible to use the image data without decompressing it first. In OpenGL this can be done using glCompressedTexImage2D, and this has worked for me in the past.
In OpenGL I've used GL_COMPRESSED_RGBA_S3TC_DXT1_EXT as image format, for Vulkan I'm using VK_FORMAT_BC1_RGBA_UNORM_BLOCK, which should be equivalent.
Here's my code for mapping the image data:
auto dds = load_dds("img.dds");
auto *srcData = static_cast<uint8_t*>(dds.data());
auto *destData = static_cast<uint8_t*>(vkImageMapPtr); // Pointer to mapped memory of VkImage
destData += layout.offset(); // layout = VkImageLayout of the image
assert((w %4) == 0);
assert((h %4) == 0);
assert(blockSize == 8); // S3TC BC1
auto wBlocks = w /4;
auto hBlocks = h /4;
for(auto y=decltype(hBlocks){0};y<hBlocks;++y)
{
auto *rowDest = destData +y *layout.rowPitch(); // rowPitch is 0
auto *rowSrc = srcData +y *(wBlocks *blockSize);
for(auto x=decltype(wBlocks){0};x<wBlocks;++x)
{
auto *pxDest = rowDest +x *blockSize;
auto *pxSrc = rowSrc +x *blockSize; // 4x4 image block
memcpy(pxDest,pxSrc,blockSize); // 64Bit per block
}
}
And here's the code for initializing the image:
vk::Device device = ...; // Initialization
vk::AllocationCallbacks allocatorCallbacks = ...; // Initialization
[...] // Load the dds data
uint32_t width = dds.width();
uint32_t height = dds.height();
auto format = dds.format(); // = vk::Format::eBc1RgbaUnormBlock;
vk::Extent3D extent(width,height,1);
vk::ImageCreateInfo imageInfo(
vk::ImageCreateFlagBits(0),
vk::ImageType::e2D,format,
extent,1,1,
vk::SampleCountFlagBits::e1,
vk::ImageTiling::eLinear,
vk::ImageUsageFlagBits::eSampled | vk::ImageUsageFlagBits::eColorAttachment,
vk::SharingMode::eExclusive,
0,nullptr,
vk::ImageLayout::eUndefined
);
vk::Image img = nullptr;
device.createImage(&imageInfo,&allocatorCallbacks,&img);
vk::MemoryRequirements memRequirements;
device.getImageMemoryRequirements(img,&memRequirements);
uint32_t typeIndex = 0;
get_memory_type(memRequirements.memoryTypeBits(),vk::MemoryPropertyFlagBits::eHostVisible,typeIndex); // -> typeIndex is set to 1
auto szMem = memRequirements.size();
vk::MemoryAllocateInfo memAlloc(szMem,typeIndex);
vk::DeviceMemory mem;
device.allocateMemory(&memAlloc,&allocatorCallbacks,&mem); // Note: Using the default allocation (nullptr) doesn't change anything
device.bindImageMemory(img,mem,0);
uint32_t mipLevel = 0;
vk::ImageSubresource resource(
vk::ImageAspectFlagBits::eColor,
mipLevel,
0
);
vk::SubresourceLayout layout;
device.getImageSubresourceLayout(img,&resource,&layout);
auto *srcData = device.mapMemory(mem,0,szMem,vk::MemoryMapFlagBits(0));
[...] // Map the dds-data (See code from first post)
device.unmapMemory(mem);
The code runs without issues, however the resulting image isn't correct. This is the source image:
And this is the result:
I'm certain that the problem lies in the first code snipped I've posted, however, in case it doesn't, I've written a small adaption of the triangle demo from the Vulkan SDK which produces the same result. It can be downloaded here. The source-code is included, all I've changed from the triangle demo are the "demo_prepare_texture_image"-function in tri.c (Lines 803 to 903) and the "dds.cpp" and "dds.h" files. "dds.cpp" contains the code for loading the dds, and mapping the image memory.
I'm using gli to load the dds-data (Which is supposed to "work perfectly with Vulkan"), which is also included in the download above. To build the project, the Vulkan SDK include directory has to be added to the "tri" project, and the path to the dds has to be changed (tri.c, Line 809).
The source image ("x64/Debug/test.dds" in the project) uses DXT1 compression. I've tested in on different hardware as well, with the same result.
Any example code for initializing/mapping compressed images would also help a lot.
Your problem is actually quite simple - in the demo_prepare_textures function, the first line, there is a variable tex_format, which is set to VK_FORMAT_B8G8R8A8_UNORM (which is what it is in the original sample). This eventually gets used to create the VkImageView. If you just change this to VK_FORMAT_BC1_RGBA_UNORM_BLOCK, it displays the texture correctly on the triangle.
As an aside - you can verify that your texture loaded correctly, with RenderDoc, which comes with the Vulkan SDK installation. Doing a capture of it, the and looking in the TextureViewer tab, the Inputs tab shows that your texture looks identical to the one on disk, even with the incorrect format.

How do you convert a 16 bit unsigned integer to a larger 8 bit unsigned integer?

I have a function that needs to return a 16 bit unsigned int vector, but for another from which I also call this one, I need the output in 8 bit unsigned int vector format. For example, if I start out with:
std::vector<uint16_t> myVec(640*480);
How might I convert it to the format of:
std::vector<uint8_t> myVec2(640*480*4);
UPDATE (more information):
I am working with libfreenect and its getDepth() method. I have modified it to output a 16 bit unsigned integer vector so that I can retrieve the depth data in millimeters. However, I would also like to display the depth data. I am working with some example code c++ from the freenect installation, which uses glut and requires an 8 bit unsigned int vector to display the depth, however, i need the 16 bit to retrieve the depth in millimeters and log it to a text file. Therefore, i was looking to retrieve the data as a 16 bit unsigned int vector in glut's draw function, and then convert it so that I can display it with the glut function that's already written.
As per your update, assuming the 8-bit unsigned int is going to be displayed as a gray scale image, what you need is akin to a Brightness Transfer Function. Basically, your output function is looking to map the data to the values 0-255, but you don't necessarily want those to correspond directly to millimeters. What if all of your data was from 0-3mm? Then your image would look almost completely black. What if it was all 300-400mm? Then it'd be completely white because it was clipped to 255.
A rudimentary way to do it would be to find the minimum and maximum values, and do this:
double scale = 255.0 / (double)(maxVal - minVal);
for( int i = 0; i < std::min(myVec.size(), myVec2.size()); ++i )
{
myVec2.at(i) = (unsigned int)((double)(myVec.at(i)-minVal) * scale);
}
depending on the distribution of your data, you might need to do something a little more complex to get the most out of your dynamic range.
Edit: This assumes your glut function is creating an image, if it is using the 8-bit value as an input to a graph then you can disregard.
Edit2: An update after your other update. If you want to fill a 640x480x4 vector, you are clearly doing an image. You need to do what I outlined above, but also the 4 dimensions that it is looking for are Red, Green, Blue, and Alpha. The Alpha channel needs to be 255 at all times (this controls how transparent it is, you don't want it to be transparent), as for the other 3... that value you got from the function above (the scaled value) if you set all 3 channels (channels being red, green, and blue) to the same value it will appear as grayscale. For example, if my data ranged from 0-25mm, for a pixel who's value is 10mm, I would set the data to 255/(25-0)* 10 = 102 and therefore the pixel would be (102, 102, 102, 255)
Edit 3: Adding wikipedia link about Brightness Transfer Functions - https://en.wikipedia.org/wiki/Color_mapping
How might I convert it to the format of:
std::vector myVec2; such that myVec2.size() will be twice as
big as myVec.size()?
myVec2.reserve(myVec.size() * 2);
for (auto it = begin(myVec); it!=end(myVec); ++it)
{
uint8_t val = static_cast<uint8_t>(*it); // isolate the low 8 bits
myVec2.push_back(val);
val = static_cast<uint8_t>((*it) >> 8); // isolate the upper 8 bits
myVec2.push_back(val);
}
Or you can change the order of push_back()'s if it matters which byte come first (the upper or the lower).
Straightforward way:
std::vector<std::uint8_t> myVec2(myVec.size() * 2);
std::memcpy(myVec2.data(), myVec.data(), myVec.size());
or with the use of the standard library
std::copy( begin(myVec), end(myVec), begin(myVec2));

Luminance values clipped to [0, 1] during texture transfer?

I am uploading a host-side texture to OpenGL using something like:
GLfloat * values = new [nRows * nCols];
// initialize values
for (int i = 0; i < nRows * nCols; ++i)
{
values[i] = (i % 201 - 100) / 10.0f; // values from -10.0f .. + 10.0f
}
glTexSubImage2D(GL_TEXTURE_2D, 0, 0, 0, nRows, nCols, GL_LUMINANCE, GL_FLOAT, values);
However, when I read back the texture using glGetTexImage(), it turns out that all values are clipped to the range [0..1].
First, I cannot find where this behavior is documented (I am using the Red Book for OpenGL 2.1).
Second, is it possible to change this behavior and let the values pass unchanged? I want to access the unscaled, unclipped data in an GLSL shader.
I cannot find where this behavior is documented
In the actual specification, it's in the section on Pixel Rectangles, titled Transfer of Pixel Rectangles.
Second, is it possible to change this behavior and let the values pass unchanged?
Yes. If you want to use "unscaled, unclamped" data, you have to use a floating point image format. The format of your texture is defined when you created the storage for it, probably by a call to glTexImage2D. The third parameter of that function defines the format. So use a proper floating-point format instead of an integer one.

How can I create a 3d texture with negative values and read it from a shader

I have written a volume rendering program that turns some 2d images into a 3d volume that can be rotated around by a user. I need to calculate a normal for each point in the 3d texture (for lighting) by taking the gradient in each direction around the point.
Calculating the normal requires six extra texture accesses within the fragment shader. The program is much faster without these extra texture access, so I am trying to precompute the gradients for each direction (x,y,z) in bytes and store it in the BGA channels of the original texture. My bytes seem to contain the right values when I test on the CPU, but when I get to the shader it comes out looking wrong. It's hard to tell why it fails from the shader, I think it is because some of the gradient values are negative. However, when I specify the texture type as GL_BYTE (as opposed to GL_UNSIGNED_BYTE) it is still wrong, and that screws up how the original texture should look. I can't tell exactly what's going wrong just by rendering the data as colors. What is the right way to put negative values into a texture? How can I know that values are negative when I read from it in the fragment shader?
The following code shows how I run the operation to compute the gradients from a byte array (byte[] all) and then turn it into a byte buffer (byteBuffer bb) that is read in as a 3d texture. The function 'toLoc(x,y,z,w,h,l)' simply returns (x+w*(y+z*h))*4)--it converts 3d subscripts to a 1d index. The image is grayscale, so I discard gba and only use the r channel to hold the original value. The remaining channels (gba) store the gradient.
int pixelDiffxy=5;
int pixelDiffz=1;
int count=0;
Float r=0f;
byte t=r.byteValue();
for(int i=0;i<w;i++){
for(int j=0;j<h;j++){
for(int k=0;k<l;k++){
count+=4;
if(i<pixelDiffxy || i>=w-pixelDiffxy || j<pixelDiffxy || j>=h-pixelDiffxy || k<pixelDiffz || k>=l-pixelDiffz){
//set these all to zero since they are out of bounds
all[toLoc(i,j,k,w,h,l)+1]=t;//green=0
all[toLoc(i,j,k,w,h,l)+2]=t;//blue=0
all[toLoc(i,j,k,w,h,l)+3]=t;//alpha=0
}
else{
int ri=(int)all[toLoc(i,j,k,w,h,l)+0] & 0xff;
//find the values on the sides of this pixel in each direction (use red channel)
int xgrad1=(all[toLoc(i-pixelDiffxy,j,k,w,h,l)])& 0xff;
int xgrad2=(all[toLoc(i+pixelDiffxy,j,k,w,h,l)])& 0xff;
int ygrad1=(all[toLoc(i,j-pixelDiffxy,k,w,h,l)])& 0xff;
int ygrad2=(all[toLoc(i,j+pixelDiffxy,k,w,h,l)])& 0xff;
int zgrad1=(all[toLoc(i,j,k-pixelDiffz,w,h,l)])& 0xff;
int zgrad2=(all[toLoc(i,j,k+pixelDiffz,w,h,l)])& 0xff;
//find the difference between the values on each side and divide by the distance between them
int xgrad=(xgrad1-xgrad2)/(2*pixelDiffxy);
int ygrad=(ygrad1-ygrad2)/(2*pixelDiffxy);
int zgrad=(zgrad1-zgrad2)/(2*pixelDiffz);
Vec3f grad=new Vec3f(xgrad,ygrad,zgrad);
Integer xg=(int) (grad.x);
Integer yg=(int) (grad.y);
Integer zg=(int) (grad.z);
//System.out.println("gs are: "+xg +", "+yg+", "+zg);
byte gby= (byte) (xg.byteValue());//green channel
byte bby= (byte) (yg.byteValue());//blue channel
byte aby= (byte) (zg.byteValue());//alpha channel
//System.out.println("gba is: "+(int)gby +", "+(int)bby+", "+(int)aby);
all[toLoc(i,j,k,w,h,l)+1]=gby;//green
all[toLoc(i,j,k,w,h,l)+2]=bby;//blue
all[toLoc(i,j,k,w,h,l)+3]=aby;//alpha
}
}
}
}
ByteBuffer bb=ByteBuffer.wrap(all);
final GL gl = drawable.getGL();
final GL2 gl2 = gl.getGL2();
final int[] bindLocation = new int[1];
gl.glGenTextures(1, bindLocation, 0);
gl2.glBindTexture(GL2.GL_TEXTURE_3D, bindLocation[0]);
gl2.glPixelStorei(GL.GL_UNPACK_ALIGNMENT, 1);//-byte alignment
gl2.glTexParameteri(GL2.GL_TEXTURE_3D, GL.GL_TEXTURE_WRAP_S, GL2.GL_CLAMP);
gl2.glTexParameteri(GL2.GL_TEXTURE_3D, GL.GL_TEXTURE_WRAP_T, GL2.GL_CLAMP);
gl2.glTexParameteri(GL2.GL_TEXTURE_3D, GL2.GL_TEXTURE_WRAP_R, GL2.GL_CLAMP);
gl2.glTexParameteri(GL2.GL_TEXTURE_3D, GL.GL_TEXTURE_MAG_FILTER, GL.GL_LINEAR);
gl2.glTexParameteri(GL2.GL_TEXTURE_3D, GL.GL_TEXTURE_MIN_FILTER, GL.GL_LINEAR);
gl2.glTexEnvf(GL2.GL_TEXTURE_ENV, GL2.GL_TEXTURE_ENV_MODE, GL.GL_REPLACE);
gl2.glTexImage3D( GL2.GL_TEXTURE_3D, 0,GL.GL_RGBA,
w, h, l, 0,
GL.GL_RGBA, GL.GL_UNSIGNED_BYTE, bb );//GL_UNSIGNED_BYTE
Is there a better way to get a large array of signed data into the shader?
gl2.glTexImage3D( GL2.GL_TEXTURE_3D, 0,GL.GL_RGBA,
w, h, l, 0, GL.GL_RGBA, GL.GL_UNSIGNED_BYTE, bb );
Well, there are two ways to go about doing this, depending on how much work you want to do in the shader vs. what OpenGL version you want to limit things to.
The version that requires more shader work also requires a bit more out of your code. See, what you want to do is have your shader take unsigned bytes, then reinterpret them as signed bytes.
The way that this would typically be done is to pass unsigned normalized bytes (as you're doing), which produces floating-point values on the [0, 1] range, then simply expand that range by multiplying by 2 and subtracting 1, yielding numbers on the [-1, 1] range. This means that your uploading code needs to take it's [-128, 127] signed bytes and convert them into [0, 255] unsigned bytes by adding 128 to them.
I have no idea how to do this in Java, which does not appear to have an unsigned byte type at all. You can't just pass a 2's complement byte and expect it to work in the shader; that's not going to happen. The byte value -128 would map to the floating-point value 1, which isn't helpful.
If you can manage to convert the data properly as I described above, then your shader access would have to unpack from the [0, 1] range to the [-1, 1] range.
If you have access to GL 3.x, then you can do this quite easily, with no shader changes:
gl2.glTexImage3D( GL2.GL_TEXTURE_3D, 0,GL.GL_RGBA8_SNORM,
w, h, l, 0, GL.GL_RGBA, GL.GL_BYTE, bb );
The _SNORM in the image format means that it is a signed, normalized format. So your bytes on the range [-128, 127] will be mapped to floats on the range [-1, 1]. Exactly what you want.

Trying to read raw image data into Java through JNI

I'm using JNI to obtain raw image data in the following format:
The image data is returned in the format of a DATA32 (32 bits) per pixel in a linear array ordered from the top left of the image to the bottom right going from left to right each line. Each pixel has the upper 8 bits as the alpha channel and the lower 8 bits are the blue channel - so a pixel's bits are ARGB (from most to least significant, 8 bits per channel). You must put the data back at some point.
The DATA32 format is essentially an unsigned int in C.
So I obtain an int[] array and then try to create a Buffered Image out of it by
int w = 1920;
int h = 1200;
BufferedImage b = new BufferedImage(w, h, BufferedImage.TYPE_INT_ARGB);
int[] f = (new Capture()).capture();
for(int i = 0; i < f.length; i++){;
b.setRGB(x, y, f[i]);
}
f is the array with the pixel data.
According to the Java documentation this should work since BufferedImage.TYPE_INT_ARGB is:
Represents an image with 8-bit RGBA color components packed into integer pixels. The image has a DirectColorModel with alpha. The color data in this image is considered not to be premultiplied with alpha. When this type is used as the imageType argument to a BufferedImage constructor, the created image is consistent with images created in the JDK1.1 and earlier releases.
Unless by 8-bit RGBA, them mean that all components added together are encoded in 8bits? But this is impossible.
This code does work, but the image that is produced is not at all like the image that it should produce. There are tonnes of artifacts. Can anyone see something obviously wrong in here?
Note I obtain my pixel data with
imlib_context_set_image(im);
data = imlib_image_get_data();
in my C code, using the library imlib2 with api http://docs.enlightenment.org/api/imlib2/html/imlib2_8c.html#17817446139a645cc017e9f79124e5a2
i'm an idiot.
This is merely a bug.
I forgot to include how I calculate x,y above.
Basically I was using
int x = i%w;
int y = i/h;
in the for loop, which is wrong. SHould be
int x = i%w;
int y = i/w;
Can't believe I made this stupid mistake.