Try to build and get Open Image Denoise working with Bazel - c++

The bounty expires in 7 days. Answers to this question are eligible for a +500 reputation bounty.
Vertexwahn wants to draw more attention to this question.
Currently, I try to get Open Image Denoise to work with Bazel.
Therefore, I implemented rules_oidn.
To try it out, you can do a
git clone https://github.com/Vertexwahn/rules_oidn.git
cd rules_oidn
cd tests
Run example with Ubuntu 22.04:
bazel run --config=gcc11 //:example
Run example with Visual Studio 2022:
bazel run --config=vs2022 //:example
The example takes a noisy image and denoises it.
...
int main() {
cout << "Simple denoising example" << endl;
Image3f color = load_image_openexr("data/cornel_box.naive_diffuse.box_filter.spp128.embree.exr");
//Image3f color = load_image_openexr("data/noisy_10spp.exr");
Image3f normal = load_image_openexr("data/normal_10spp.exr");
Image3f albedo = load_image_openexr("data/albedo_10spp.exr");
Image3f out{color.width(), color.height()};
// for debug reasons the color image can be initialized with a const color
if(false) {
for (int x = 0; x < color.width(); ++x) {
for (int y = 0; y < color.height(); ++y) {
color.set_pixel(x,y,.5f, .5f, .5f);
}
}
}
float* colorPtr = color.data();
float* albedoPtr = albedo.data();
float* normalPtr = normal.data();
float* outputPtr = out.data();
int width = out.width();
int height = out.height();
oidn::DeviceRef device = oidn::newDevice();
device.set("verbose", 1);
device.commit();
// Create a filter for denoising a beauty (color) image using optional auxiliary images too
oidn::FilterRef filter = device.newFilter("RT"); // generic ray tracing filter
filter.setImage("color", colorPtr, oidn::Format::Float3, width, height); // beauty
//filter.setImage("albedo", albedoPtr, oidn::Format::Float3, width, height); // auxiliary
//filter.setImage("normal", normalPtr, oidn::Format::Float3, width, height); // auxiliary
filter.setImage("output", outputPtr, oidn::Format::Float3, width, height); // denoised beauty
filter.set("hdr", true); // beauty image is HDR
filter.commit();
// Filter the image
filter.execute();
// Check for errors
const char* errorMessage;
if (device.getError(errorMessage) != oidn::Error::None) {
std::cout << "Error: " << errorMessage << std::endl;
}
store_open_exr("denoised.exr", out);
return 0;
}
Unfortunately, the denoised image contains black stripes:
I tested the same input with https://github.com/DeclanRussell/IntelOIDenoiser and got the expected result (without black stripes).
If I choose a constant color image, e.g.
// for debug reasons the color image can be initialized with a const color
if(true) {
for (int x = 0; x < color.width(); ++x) {
for (int y = 0; y < color.height(); ++y) {
color.set_pixel(x,y,.5f, .5f, .5f);
}
}
}
I also get black stripes.
Currently, I am missing a good strategy to find the issue.
Any hints or solutions to fix the issue are welcome.

Related

Loading a BMP image at a specific index in OpenGL

I have to load a 24 bit BMP file at a certain (x,y) index of glut window from a file using OpenGL. I have found a function that uses glaux library to do so. Here the color mentioned in ignoreColor is ignored during rendering.
void iShowBMP(int x, int y, char filename[], int ignoreColor)
{
AUX_RGBImageRec *TextureImage;
TextureImage = auxDIBImageLoad(filename);
int i,j,k;
int width = TextureImage->sizeX;
int height = TextureImage->sizeY;
int nPixels = width * height;
int *rgPixels = new int[nPixels];
for (i = 0, j=0; i < nPixels; i++, j += 3)
{
int rgb = 0;
for(int k = 2; k >= 0; k--)
{
rgb = ((rgb << 8) | TextureImage->data[j+k]);
}
rgPixels[i] = (rgb == ignoreColor) ? 0 : 255;
rgPixels[i] = ((rgPixels[i] << 24) | rgb);
}
glRasterPos2f(x, y);
glDrawPixels(width, height, GL_RGBA, GL_UNSIGNED_BYTE, rgPixels);
delete []rgPixels;
free(TextureImage->data);
free(TextureImage);
}
But the problem is that glaux is now obsolete. If I call this function, the image is rendered and shown for a minute, then an error pops up (without any error message) and the glut window disappears. From the returned value shown in the console, it seems like a runtime error.
Is there any alternative to this function that doesn't use glaux? I have seen cimg, devil etc but none of them seems to work like this iShowBMP function. I am doing my project in Codeblocks.
I have to load every frame to keep the implementation consistent with other parts of the program. Also, the bmp file whose name has been passed as a parameter to the function has both width and height in powers of 2.
The last two free() statements were not getting executed for some unknown reasons, so the memory consumption was increasing. That's why the program was crashing after a moment. Later I solved it using stb_image.h.

ScalableTSDFVolume Integrate from TUM-RGBD Dataset

I am using Open3D 0.15 and C++11 on Ubuntu 18.04.
The main function I'm interested in is the ScalabeTSDFVolume Integrate() function, using the TUM RGBD dataset (the xyz set to be exact), based off of the IntegrateRGBD example from the Open3D repo.
Since the TUM-RGBD dataset does not provide an association file that matches the RGBD images and the trajectory info, I've created my own small code that matches the timestamp on the TUM dataset's image data and the trajectory information, and converting the 7-dimension [x y z rx ry rz rw] trajectory information into Eigen::Matrix4d, using the same equation that Open3D's FileTUM.cpp uses:
do
{
// Read the timestamp first
gt >> p_gt.timestamp;
double poseArr[7];
// push the remaining 7 numbers to the poseArr
for (int i = 0; i < 7; i++)
gt >> poseArr[i];
// copy paste of the tum trajectory reader
Eigen::Matrix4d transform;
transform.setIdentity();
transform.topLeftCorner<3, 3>() =
Eigen::Quaterniond(poseArr[6], poseArr[3], poseArr[4], poseArr[5]).toRotationMatrix();
transform.topRightCorner<3, 1>() = Eigen::Vector3d(poseArr[0], poseArr[1], poseArr[2]);
p_gt.pose = transform.inverse();
gtF.push_back(p_gt);
} while (std::getline(gt, line));
The code runs fine, but the issue is when I try to integrate multiple frames into the same volume and extract its pointcloud or mesh.
I can tell that the RGBD information is being fed into the program correctly, by extracting the mesh at the very first frame:
first frame mesh extraction
But there is a significant artifact when I try to extract the mesh when more frames are integrated, like this:
30 frames mesh extraction
From my previous experience, this probably has to do with the fact that the transformation matrices are not in the correct axis. If anyone has tried to use the TUM dataset with Open3D and encountered the same problem, I would greatly appreciate any info on this.
Edit:
For reference, this is the modified code I'm using for the reconstruction.
int main(int argc, char *argv[]) {
using namespace open3d;
std::string filebase("/home/geometry/Documents/rgbd_dataset_freiburg1_xyz");
VirtualSensor::CameraParameters kinect{ 525.0,525.0,319.5,239.5,5000};
VirtualSensor::CameraParameters camPar = kinect;
VirtualSensor v1(filebase,camPar);
bool save_pointcloud = true;
bool save_mesh = true;
bool save_voxel = false;
int every_k_frames = 50;
double length = 4.0;
double uLength = 6.0;
int resolution = 512;
double sdf_trunc_percentage = 0.01;
int verbose = 2;
utility::SetVerbosityLevel((utility::VerbosityLevel)verbose);
auto camera_intrinsic = camera::PinholeCameraIntrinsic(640, 480, 525.0, 525.0, 319.5, 239.5);
int index = 0;
int save_index = 0;
int pairSize = 30;
// initialise TSDF
pipelines::integration::ScalableTSDFVolume volume(
length / (double)resolution, length * sdf_trunc_percentage,
pipelines::integration::TSDFVolumeColorType::RGB8);
//pipelines::integration::UniformTSDFVolume uVolume(uLength, resolution, uLength*sdf_trunc_percentage, pipelines::integration::TSDFVolumeColorType::RGB8);
utility::FPSTimer timer("Process RGBD stream",
pairSize);
geometry::Image depth, color;
// start loop
for(int i = 0; i < pairSize; i++){
utility::LogInfo("Processing frame {:d} ...", index);
io::ReadImage(v1.GetDepthPath(i), depth);
io::ReadImage(v1.GetColorPath(i), color);
auto rgbd = geometry::RGBDImage::CreateFromColorAndDepth(
color, depth, 5000.0, 6.0, false);
if (index == 0 ||
(every_k_frames > 0 && index % every_k_frames == 0))
volume.Reset();
}
volume.Integrate(*rgbd,
camera_intrinsic, // intrinsic never changes
v1.GetCounterGT(i)); // get the groundtruth pose from my class
index++;
// saving mesh/pc logic
if (index == pairSize ||
(every_k_frames > 0 && index % every_k_frames == 0)) {
utility::LogInfo("Saving fragment {:d} ...", save_index);
std::string save_index_str = std::to_string(save_index);
if (save_pointcloud) {
utility::LogInfo("Saving pointcloud {:d} ...", save_index);
auto pcd = volume.ExtractPointCloud();
io::WritePointCloud("pointcloud_" + save_index_str + ".ply",
*pcd);
}
if (save_mesh) {
utility::LogInfo("Saving mesh {:d} ...", save_index);
auto mesh = volume.ExtractTriangleMesh();
io::WriteTriangleMesh("mesh_" + save_index_str + ".ply",
*mesh);
}
if (save_voxel) {
utility::LogInfo("Saving voxel {:d} ...", save_index);
auto voxel = volume.ExtractVoxelPointCloud();
io::WritePointCloud("voxel_" + save_index_str + ".ply",
*voxel);
}
save_index++;
}
timer.Signal();
}
return 0;
}

How to read data from a UTexture2D in C++

I am trying to read the pixel data from a populated UTexture2D in an Unreal Engine C++ project. Before I post the question here, I tried to use the method described in this link: https://answers.unrealengine.com/questions/25594/accessing-pixel-values-of-texture2d.html. However, it doesn't work for me. All pixel values I got from the texture are some garbage data.
I just want to get the depth values from the SceneCapture2D and a post-processing material that contains SceneTexture: Depth node. I need the depth values available in C++ so that I can do further processing with OpenCV. In Directx11, staging texture can be used for CPU read, but in the unreal engine, I don't know how to create a 'staging texture' like Dx11 has. I can't get the correct pixel values from the current method which makes me think I may try to access a no-CPU-readable texture.
Here is my experimental code for reading data back from an RGB UTexture2D.
Initialize the RGB Texture:
VideoTextureColor= UTexture2D::CreateTransient(640, 480, PF_B8G8R8A8);
VideoTextureColor->UpdateResource();
VideoUpdateTextureRegionColor = new FUpdateTextureRegion2D(0, 0, 0, 0, 640, 480);
ColorRegionData = new FUpdateTextureRegionsData;
PixelDepthData.Init(FColor(0, 0, 0, 255), 640 * 480);
// Populate the texture with blue color
for (int i = 0; i < 640; i++) {
for (int j = 0; j < 480; j++) {
int idx = j * 640 + i;
PixelDepthData[idx].B = 255;
PixelDepthData[idx].G = 0;
PixelDepthData[idx].R = 0;
PixelDepthData[idx].A = 255;
}
}
UpdateTextureRegions(
VideoTextureColor,
(int32)0,
(uint32)1,
VideoUpdateTextureRegionColor,
(uint32)(4 * 640),
(uint32)4,
(uint8*)PixelDepthData.GetData(),
false,
ColorRegionData
);
Then, update read its value back to the PixelDepthData (TArray type) array and update this texture with values storing in the PixelDepthData, which is its old value.
UpdateTextureRegions(
VideoTextureColor,
(int32)0,
(uint32)1,
VideoUpdateTextureRegionColor,
(uint32)(4 * 640),
(uint32)4,
(uint8*)PixelDepthData.GetData(),
false,
ColorRegionData
);
ENQUEUE_UNIQUE_RENDER_COMMAND_ONEPARAMETER(
FRealSenseDelegator,
ARealSenseDelegator*, RealSenseDelegator, this,
{
FColor* tmpImageDataPtr = static_cast<FColor*>((RealSenseDelegator->VideoTextureColor)->PlatformData->Mips[0].BulkData.Lock(LOCK_READ_ONLY));
for (uint32 j = 0; j < 480; j++) {
for (uint32 i = 0; i < 640; i++) {
uint32 idx = j * 640 + i;
RealSenseDelegator->PixelDepthData[idx] = tmpImageDataPtr[idx];
RealSenseDelegator->PixelDepthData[idx].A = 255;
}
}
(RealSenseDelegator->VideoTextureColor)->PlatformData->Mips[0].BulkData.Unlock();
}
);
All I got is a white color texture instead of a blue color texture in the visualization scene.
Does anyone know how to read the data of the UTexture2D Object?
I figured that out. You have to get the UTexture2D's RHI texture reference first, and then use RHILockTexture2D to read it's data, and you have to do it in the RenderThread. The following code just an example:
FTexture2DResource* uTex2DRes = (FTexture2DResource*)(RealSenseDelegator->VideoTexturePixelDepth)->Resource;
float* cpuDataPtr = (float*)RHILockTexture2D(
uTex2DRes->GetTexture2DRHI(),
0,
RLM_ReadOnly,
destStride,
false);
for (uint32 j = 0; j < 480; j++) {
for (uint32 i = 0; i < 640; i++) {
uint32 idx = j * 640 + i;
// TODO Read the pixel data right here
}
}
RHIUnlockTexture2D(uTex2DRes->GetTexture2DRHI(), 0, false);
To do this in the Render Thread, you have to use the Macro such as ENQUEUE_UNIQUE_RENDER_COMMAND_ONEPARAMETER // If you just one to pass one parameter to the render thread, use this one.+

Convert cv::Mat to openni::VideoFrameRef

I have a kinect streaming data into a cv::Mat. I am trying to get some example code running that uses OpenNI.
Can I convert my Mat into an OpenNI format image somehow?
I just need the depth image, and after fighting with OpenNI for a long time, have given up on installing it.
I am using OpenCV 3, Visual Studio 2013, Kinect v2 for Windows.
The relevant code is:
void CDifodoCamera::loadFrame()
{
//Read the newest frame
openni::VideoFrameRef framed; //I assume I need to replace this with my Mat...
depth_ch.readFrame(&framed);
const int height = framed.getHeight();
const int width = framed.getWidth();
//Store the depth values
const openni::DepthPixel* pDepthRow = (const openni::DepthPixel*)framed.getData();
int rowSize = framed.getStrideInBytes() / sizeof(openni::DepthPixel);
for (int yc = height-1; yc >= 0; --yc)
{
const openni::DepthPixel* pDepth = pDepthRow;
for (int xc = width-1; xc >= 0; --xc, ++pDepth)
{
if (*pDepth < 4500.f)
depth_wf(yc,xc) = 0.001f*(*pDepth);
else
depth_wf(yc,xc) = 0.f;
}
pDepthRow += rowSize;
}
}
First you need to understand how your data is coming... If it is already in cv::Mat you should be receiving two images, one for the RGB information that usually is a 3 channel uchar cv::Mat and another image for the depth information that usually it is saved in a 16 bit representation in milimeters (you can not save float mat as images, but you can as yml/xml files using opencv).
Assuming you want to read and process the image that contains the depth information, you can change your code to:
void CDifodoCamera::loadFrame()
{
//Read the newest frame
//the depth image should be png since it is the one which supports 16 bits and it must have the ANYDEPTH flag
cv::Mat depth_im = cv::imread("img_name.png",CV_LOAD_IMAGE_ANYDEPTH);
const int height = depth_im.rows;
const int width = depth_im.cols;
for (int y = 0; y < height; y++)
{
for (int x = 0; x < width; x++)
{
if (depth_im<unsigned short>(y,x) < 4500)
depth_wf(y,x) = 0.001f * (float)depth_im<unsigned short>(y,x);
else
depth_wf(y,x) = 0.f;
}
}
}
I hope this helps you. If you have any question just ask :)

Windows .exe ridiculously slower than Linux Executable

So, I'm writing (or have written) a contour mapping application for mapping power frequency in North America, and it works really well... On linux. I run it on a loop to update a bmp map file, which I'm eventually going to output on a website. It can run and update itself in about 3 seconds, which is great for me. The problem came when I tried to port the application to windows. I moved the code into Visual Studios 2012. Linked the libraries, compiled and such, I had to make it ignore a few warnings about float to integer conversion, but I got it working.
Then, I had it run, and it didn't seem to do anything. But, after adding a few output commands, I realized it was doing something, it was just doing it probably 100x slower than the Linux executable! I mean, admittedly, the code is pretty intensive (around 400,000 iterations), but this is just ridiculous.
I saw a lot of other topics about VS running slow in debug mode, but even after I compile and run an executable, it's still just as slow.
Here's relevant code, let me know if you have any ideas. Some of the functions you won't recognize because either they aren't relevant to performance (I only call them once, and I know they aren't the source of the speed problem) or they come from the Easy BMP library I'm using for image manipulation. Right now, I have it set to output a 100X100 image, but originally I was outputting a 800x500 pixel image:
float get_value(int x, int y, int dataNum, vector<vector<float> > data)
{
vector<float> distance;
float value=0; float distanceTotal = 0;
for(int i=0; i<dataNum; i++)
{
distance.push_back(sqrt(pow(data[0][i]-x,2) + pow(data[1][i]-y,2)));
if(distance[i] < 2)
return 0;
distance[i] = 1/pow(distance[i],3);
distanceTotal+=distance[i];
}
for(int i=0; i<dataNum; i++)
{
value+=distance[i]/distanceTotal*data[2][i];
}
return value;
}
int _tmain(int argc, _TCHAR* argv[])
{
//set image attributes
int height=100; int width = 100; int colorScale = 5*255; string dataFile = "data2.txt";
//get data and colormap
vector<vector<float> > data = getData(dataFile, width, height);
vector<RGBApixel> colorMap = makeColorMap(colorScale);
int dataNum = data[0].size();
pair<float,float> range = make_pair(*min_element(data[2].begin(),data[2].end()),*max_element(data[2].begin(),data[2].end()));
//make image
BMP newMap;
newMap.SetBitDepth(16);
newMap.SetSize(width,height);
for(int x=0; x<width; x++)
{
for(int y=0; y<height; y++)
{
//for debug purposes
cout << x << " " << y << endl;
float value = get_value(x,y,dataNum,data);
//get color value based on data value
int colorValue = floor((value-range.first)/(range.second-range.first)*colorScale);
//handle border cases
if(colorValue < 0 )
colorValue=0;
else if(colorValue > colorScale-1)
colorValue=colorScale-1;
newMap.SetPixel(x,y,colorMap[colorValue]);
}
}
newMap.WriteToFile("map.bmp");
return 0;
}
Any thoughts?
Thanks!