Can a KTX image file be a cubemap arrays? - c++

Is it valid for a KTX image to be a cubemaps arrays, or is that not a thing?
I have some code that I'm currently using for uploading the data from a KTX file to the GPU. Currently, the code works for a regular 2d image, a cubemap, and a texture array. However, it would not support a KTX image that is a cubemap array, if that is a thing.
If it is possible, what is the code below missing to accomplish that?
uint32_t offset = 0;
for (uint32_t layer = 0; layer < layers; layer++) {
for (uint32_t face = 0; face < faces; face++) {
for (uint32_t level = 0; level < mipLevels; level++) {
offset = tex->GetImageOffset(layer, face, level);
vk::BufferImageCopy bufferCopyRegion = {};
bufferCopyRegion.imageSubresource.aspectMask = vk::ImageAspectFlagBits::eColor;
bufferCopyRegion.imageSubresource.mipLevel = level;
bufferCopyRegion.imageSubresource.baseArrayLayer = (faces == 6 ? face : layer); // TexArray or Cubemap, not both.
bufferCopyRegion.imageSubresource.layerCount = 1;
bufferCopyRegion.imageExtent.width = width >> level;;
bufferCopyRegion.imageExtent.height = height >> level;
bufferCopyRegion.imageExtent.depth = 1;
bufferCopyRegion.bufferOffset = offset;
bufferCopyRegions.push_back(bufferCopyRegion);
}
}
}
Vulkan command to transfer the image.
// std::vector<vk::BufferImageCopy> regions;
cmdBuf->copyBufferToImage(srcBufferHandle, destImageHandle,
vk::ImageLayout::eTransferDstOptimal, uint32_t(regions.size()), regions.data());

Yes, KTX also supports cube map arrays (see the KTX specification). Those are stored using layers.
The Vulkan spec states the following on how cube maps are stored in a cube map array:
For cube arrays, each set of six sequential
layers is a single cube, so the number of cube maps in a cube map array view is layerCount / 6, and
image array layer (baseArrayLayer + i) is face index (i mod 6) of cube i / 6.
So you need to change the baseArrayLayer of your buffer copy region accordingly.
Sample code:
// Setup buffer copy regions to get the data from the ktx file to your own image
for (uint32_t layer = 0; layer < ktxTexture->numLayers; layer++) {
for (uint32_t face = 0; face < 6; face++) {
for (uint32_t level = 0; level < ktxTexture->numLevels; level++) {
ktx_size_t offset;
KTX_error_code ret = ktxTexture_GetImageOffset(ktxTexture, level, layer, face, &offset);
VkBufferImageCopy bufferCopyRegion = {};
bufferCopyRegion.imageSubresource.aspectMask = VK_IMAGE_ASPECT_COLOR_BIT;
bufferCopyRegion.imageSubresource.mipLevel = level;
bufferCopyRegion.imageSubresource.baseArrayLayer = layer * 6 + face;
bufferCopyRegion.imageSubresource.layerCount = 1;
bufferCopyRegion.imageExtent.width = ktxTexture->baseWidth >> level;
bufferCopyRegion.imageExtent.height = ktxTexture->baseHeight >> level;
bufferCopyRegion.imageExtent.depth = 1;
bufferCopyRegion.bufferOffset = offset;
bufferCopyRegions.push_back(bufferCopyRegion);
}
}
}
// Create the image view for a cube map array
VkImageViewCreateInfo view = vks::initializers::imageViewCreateInfo();
view.viewType = VK_IMAGE_VIEW_TYPE_CUBE_ARRAY;
view.format = format;
view.components = { VK_COMPONENT_SWIZZLE_R, VK_COMPONENT_SWIZZLE_G, VK_COMPONENT_SWIZZLE_B, VK_COMPONENT_SWIZZLE_A };
view.subresourceRange = { VK_IMAGE_ASPECT_COLOR_BIT, 0, 1, 0, 1 };
view.subresourceRange.layerCount = 6 * cubeMap.layerCount;
view.subresourceRange.levelCount = cubeMap.mipLevels;
view.image = cubeMap.image;
vkCreateImageView(device, &view, nullptr, &cubeMap.view);

Related

In FBX, how do you know which vert indexes correspond to which control point indexes?

I am currently trying to load an FBX mesh for use with DirectX, but my FBX file has it's UVs stored by vert index and the normals stored by control point index. How do I know which vertexes have which control point's values?
My code for loading positions, uvs and normals is ripped straight from the fbx example code, but I can post it if needed.
edit: as requested, here are the parts of my code I am talking about.
The UV code will go into the if statement for mapping mode by vert index, while the normal code is set to mapping mode by ctrl point
//load uvs
if (lUVElement->GetMappingMode() == FbxGeometryElement::eByControlPoint)
{
for (int lPolyIndex = 0; lPolyIndex < lPolyCount; ++lPolyIndex)
{
// build the max index array that we need to pass into MakePoly
const int lPolySize = mesh->GetPolygonSize(lPolyIndex);
for (int lVertIndex = 0; lVertIndex < lPolySize; ++lVertIndex)
{
//get the index of the current vertex in control points array
int lPolyVertIndex = mesh->GetPolygonVertex(lPolyIndex, lVertIndex);
//the UV index depends on the reference mode
int lUVIndex = lUseIndex ? lUVElement->GetIndexArray().GetAt(lPolyVertIndex) : lPolyVertIndex;
lUVValue = lUVElement->GetDirectArray().GetAt(lUVIndex);
_floatVec->push_back((float)lUVValue.mData[0]);
_floatVec->push_back((float)lUVValue.mData[1]);
}
}
}
else if (lUVElement->GetMappingMode() == FbxGeometryElement::eByPolygonVertex)
{
int lPolyIndexCounter = 0;
for (int lPolyIndex = 0; lPolyIndex < lPolyCount; ++lPolyIndex)
{
// build the max index array that we need to pass into MakePoly
const int lPolySize = mesh->GetPolygonSize(lPolyIndex);
for (int lVertIndex = 0; lVertIndex < lPolySize; ++lVertIndex)
{
if (lPolyIndexCounter < lIndexCount)
{
//the UV index depends on the reference mode
int lUVIndex = lUseIndex ? lUVElement->GetIndexArray().GetAt(lPolyIndexCounter) : lPolyIndexCounter;
lUVValue = lUVElement->GetDirectArray().GetAt(lUVIndex);
_floatVec->push_back((float)lUVValue.mData[0]);
_floatVec->push_back((float)lUVValue.mData[1]);
lPolyIndexCounter++;
}
}
}
}
//and now normals
if (lNormalElement->GetMappingMode() == FbxGeometryElement::eByControlPoint)
{
//Let's get normals of each vertex, since the mapping mode of normal element is by control point
for (int lVertexIndex = 0; lVertexIndex < mesh->GetControlPointsCount(); lVertexIndex++)
{
int test = mesh->GetControlPointsCount();
int lNormalIndex = 0;
//reference mode is direct, the normal index is same as vertex index.
//get normals by the index of control vertex
if (lNormalElement->GetReferenceMode() == FbxGeometryElement::eDirect)
lNormalIndex = lVertexIndex;
//reference mode is index-to-direct, get normals by the index-to-direct
if (lNormalElement->GetReferenceMode() == FbxGeometryElement::eIndexToDirect)
lNormalIndex = lNormalElement->GetIndexArray().GetAt(lVertexIndex);
//Got normals of each vertex.
FbxVector4 lNormal = lNormalElement->GetDirectArray().GetAt(lNormalIndex);
_floatVec->push_back((float)lNormal[0]);
_floatVec->push_back((float)lNormal[1]);
_floatVec->push_back((float)lNormal[2]);
}
}
else if (lNormalElement->GetMappingMode() == FbxGeometryElement::eByPolygonVertex)
{
//etc... code wont go here
}
}
}
}
So how can I know which vertexes will have which normals?

Generate image from an unorganized Point Cloud in PCL

I have an unorganized point cloud of the scene. Below is a screenshot of the point cloud-
I want to compose an image from this point cloud. Below is the code snippet-
#include <iostream>
#include <pcl/io/pcd_io.h>
#include <pcl/point_types.h>
#include <opencv2/opencv.hpp>
int main(int argc, char** argv)
{
pcl::PointCloud<pcl::PointXYZRGBA>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZRGBA>);
pcl::io::loadPCDFile("file.pcd", *cloud);
cv::Mat image = cv::Mat(cloud->height, cloud->width, CV_8UC3);
for (int i = 0; i < image.rows; i++)
{
for (int j = 0; j < image.cols; j++)
{
pcl::PointXYZRGBA point = cloud->at(j, i);
image.at<cv::Vec3b>(i, j)[0] = point.b;
image.at<cv::Vec3b>(i, j)[1] = point.g;
image.at<cv::Vec3b>(i, j)[2] = point.r;
}
}
cv::imwrite("image.png", image);
return (0);
}
The PCD file can be found here. The above code throws following error at runtime-
terminate called after throwing an instance of 'pcl::IsNotDenseException'
what(): : Can't use 2D indexing with a unorganized point cloud
Since the cloud is unorganized, the HEIGHT field is 1. This makes me confused while defining the dimensions of the image.
Questions
How to compose an image from an unorganized point cloud?
How to convert pixels located in composed image back to point cloud (3D space)?
PS: I am using PCL 1.7 in Ubuntu 14.04 LTS OS.
What Unorganized point cloud means is that the points are NOT assigned to a fixed (organized) grid, therefore ->at(j, i) can't be used (height is always 1, and the width is just the size of the cloud.
If you want to generate an image from your cloud, I suggest the following process:
Project the point cloud to a plane.
Generate a grid (organized point cloud) on that plane.
Interpolate the colors from the unorganized cloud to the grid (organized cloud).
Generate image from your organized grid (your initial attempt).
To be able to convert back to 3D:
When projecting to the plane save the "projection vectors" (vector from original point to the projected point).
Interpolate that as well to the grid.
methods for creating the grid:
Project the point cloud to a plane (unorganized cloud), and optionally save the reconstruction information in the normals:
pcl::PointCloud<pcl::PointXYZINormal>::Ptr ProjectToPlane(pcl::PointCloud<pcl::PointXYZINormal>::Ptr cloud, Eigen::Vector3f origin, Eigen::Vector3f axis_x, Eigen::Vector3f axis_y)
{
PointCloud<PointXYZINormal>::Ptr aux_cloud(new PointCloud<PointXYZINormal>);
copyPointCloud(*cloud, *aux_cloud);
auto normal = axis_x.cross(axis_y);
Eigen::Hyperplane<float, 3> plane(normal, origin);
for (auto itPoint = aux_cloud->begin(); itPoint != aux_cloud->end(); itPoint++)
{
// project point to plane
auto proj = plane.projection(itPoint->getVector3fMap());
itPoint->getVector3fMap() = proj;
// optional: save the reconstruction information as normals in the projected cloud
itPoint->getNormalVector3fMap() = itPoint->getVector3fMap() - proj;
}
return aux_cloud;
}
Generate a grid based on an origin point and two axis vectors (length and image_size can either be predetermined as calculated from your cloud):
pcl::PointCloud<pcl::PointXYZINormal>::Ptr GenerateGrid(Eigen::Vector3f origin, Eigen::Vector3f axis_x , Eigen::Vector3f axis_y, float length, int image_size)
{
auto step = length / image_size;
pcl::PointCloud<pcl::PointXYZINormal>::Ptr image_cloud(new pcl::PointCloud<pcl::PointXYZINormal>(image_size, image_size));
for (auto i = 0; i < image_size; i++)
for (auto j = 0; j < image_size; j++)
{
int x = i - int(image_size / 2);
int y = j - int(image_size / 2);
image_cloud->at(i, j).getVector3fMap() = center + (x * step * axisx) + (y * step * axisy);
}
return image_cloud;
}
Interpolate to an organized grid (where the normals store reconstruction information and the curvature is used as a flag to indicate empty pixel (no corresponding point):
void InterpolateToGrid(pcl::PointCloud<pcl::PointXYZINormal>::Ptr cloud, pcl::PointCloud<pcl::PointXYZINormal>::Ptr grid, float max_resolution, int max_nn_to_consider)
{
pcl::search::KdTree<pcl::PointXYZINormal>::Ptr tree(new pcl::search::KdTree<pcl::PointXYZINormal>);
tree->setInputCloud(cloud);
for (auto idx = 0; idx < grid->points.size(); idx++)
{
std::vector<int> indices;
std::vector<float> distances;
if (tree->radiusSearch(grid->points[idx], max_resolution, indices, distances, max_nn_to_consider) > 0)
{
// Linear Interpolation of:
// Intensity
// Normals- residual vector to inflate(recondtruct) the surface
float intensity(0);
Eigen::Vector3f n(0, 0, 0);
float weight_factor = 1.0F / accumulate(distances.begin(), distances.end(), 0.0F);
for (auto i = 0; i < indices.size(); i++)
{
float w = weight_factor * distances[i];
intensity += w * cloud->points[indices[i]].intensity;
auto res = cloud->points[indices[i]].getVector3fMap() - grid->points[idx].getVector3fMap();
n += w * res;
}
grid->points[idx].intensity = intensity;
grid->points[idx].getNormalVector3fMap() = n;
grid->points[idx].curvature = 1;
}
else
{
grid->points[idx].intensity = 0;
grid->points[idx].curvature = 0;
grid->points[idx].getNormalVector3fMap() = Eigen::Vector3f(0, 0, 0);
}
}
}
Now you have a grid (an organized cloud), which you can easily map to an image. Any changes you make to the images, you can map back to the grid, and use the normals to project back to your original point cloud.
usage example for creating the grid:
pcl::PointCloud<pcl::PointXYZINormal>::Ptr original_cloud = ...;
// reference frame for the projection
// e.g. take XZ plane around 0,0,0 of length 100 and map to 128*128 image
Eigen::Vector3f origin = Eigen::Vector3f(0,0,0);
Eigen::Vector3f axis_x = Eigen::Vector3f(1,0,0);
Eigen::Vector3f axis_y = Eigen::Vector3f(0,0,1);
float length = 100
int image_size = 128
auto aux_cloud = ProjectToPlane(original_cloud, origin, axis_x, axis_y);
// aux_cloud now contains the points of original_cloud, with:
// xyz coordinates projected to XZ plane
// color (intensity) of the original_cloud (remains unchanged)
// normals - we lose the normal information, as we use this field to save the projection information. if you wish to keep the normal data, you should define a custom PointType.
// note: for the sake of projection, the origin is only used to define the plane, so any arbitrary point on the plane can be used
auto grid = GenerateGrid(origin, axis_x , axis_y, length, image_size)
// organized cloud that can be trivially mapped to an image
float max_resolution = 2 * length / image_size;
int max_nn_to_consider = 16;
InterpolateToGrid(aux_cloud, grid, max_resolution, max_nn_to_consider);
// Now you have a grid (an organized cloud), which you can easily map to an image. Any changes you make to the images, you can map back to the grid, and use the normals to project back to your original point cloud.
additional helper methods for how I use the grid:
// Convert an Organized cloud to cv::Mat (an image and a mask)
// point Intensity is used for the image
// if as_float is true => take the raw intensity (image is CV_32F)
// if as_float is false => assume intensity is in range [0, 255] and round it (image is CV_8U)
// point Curvature is used for the mask (assume 1 or 0)
std::pair<cv::Mat, cv::Mat> ConvertGridToImage(pcl::PointCloud<pcl::PointXYZINormal>::Ptr grid, bool as_float)
{
int rows = grid->height;
int cols = grid->width;
if ((rows <= 0) || (cols <= 0))
return pair<Mat, Mat>(Mat(), Mat());
// Initialize
Mat image = Mat(rows, cols, as_float? CV_32F : CV_8U);
Mat mask = Mat(rows, cols, CV_8U);
if (as_float)
{
for (int y = 0; y < image.rows; y++)
{
for (int x = 0; x < image.cols; x++)
{
image.at<float>(y, x) = grid->at(x, image.rows - y - 1).intensity;
mask.at<uchar>(y, x) = 255 * grid->at(x, image.rows - y - 1).curvature;
}
}
}
else
{
for (int y = 0; y < image.rows; y++)
{
for (int x = 0; x < image.cols; x++)
{
image.at<uchar>(y, x) = (int)round(grid->at(x, image.rows - y - 1).intensity);
mask.at<uchar>(y, x) = 255 * grid->at(x, image.rows - y - 1).curvature;
}
}
}
return pair<Mat, Mat>(image, mask);
}
// project image to cloud (using the grid data)
// organized - whether the resulting cloud should be an organized cloud
pcl::PointCloud<pcl::PointXYZI>::Ptr BackProjectImage(cv::Mat image, pcl::PointCloud<pcl::PointXYZINormal>::Ptr grid, bool organized)
{
if ((image.size().height != grid->height) || (image.size().width != grid->width))
{
assert(false);
throw;
}
PointCloud<PointXYZI>::Ptr cloud(new PointCloud<PointXYZI>);
cloud->reserve(grid->height * grid->width);
// order of iteration is critical for organized target cloud
for (auto r = image.size().height - 1; r >= 0; r--)
{
for (auto c = 0; c < image.size().width; c++)
{
PointXYZI point;
auto mask_value = mask.at<uchar>(image.rows - r - 1, c);
if (mask_value > 0) // valid pixel
{
point.intensity = mask_value;
point.getVector3fMap() = grid->at(c, r).getVector3fMap() + grid->at(c, r).getNormalVector3fMap();
}
else // invalid pixel
{
if (organized)
{
point.intensity = 0;
point.x = numeric_limits<float>::quiet_NaN();
point.y = numeric_limits<float>::quiet_NaN();
point.z = numeric_limits<float>::quiet_NaN();
}
else
{
continue;
}
}
cloud->push_back(point);
}
}
if (organized)
{
cloud->width = grid->width;
cloud->height = grid->height;
}
return cloud;
}
usage example for working with the grid:
// image_mask is std::pair<cv::Mat, cv::Mat>
auto image_mask = ConvertGridToImage(grid, false);
...
do some work with the image/mask
...
auto new_cloud = BackProjectImage(image_mask.first, grid, false);
For an unorganized point cloud, height and width have different meanings as you may have noticed. http://pointclouds.org/documentation/tutorials/basic_structures.php
It is not as simple to convert an unorganized point cloud to an image, as the points are represented as floats and there is no defined perspective. However, you can work around that by determining a perspective and creating discrete bins for the points. A similar question and answer can be found here: Converting a pointcloud to a depth/multi channel image

Ray Tracing using Nvidia Optix with Open Asset Import Library (assimp) - rendering multiple meshes

I'm trying to combine the versatility of Open Asset Import Library (reading in a variety of 3D model filetypes) with NVidia Optix ray tracing to render the models.
So far, it is working whenever the model I'm rendering is made up of a single mesh. When I try to render a file with more than one mesh, I get only partial results. I can't narrow down where the issue is, looking for some insight. Relevant code here:
Loading a file using assimp importer and creating the optix buffers:
int loadAsset(const char* path)
{
Assimp::Importer importer;
scene = importer.ReadFile(
path,
aiProcess_Triangulate
//| aiProcess_JoinIdenticalVertices
| aiProcess_SortByPType
| aiProcess_ValidateDataStructure
| aiProcess_SplitLargeMeshes
| aiProcess_FixInfacingNormals
);
if (scene) {
getBoundingBox(&scene_min, &scene_max);
scene_center.x = (scene_min.x + scene_max.x) / 2.0f;
scene_center.y = (scene_min.y + scene_max.y) / 2.0f;
scene_center.z = (scene_min.z + scene_max.z) / 2.0f;
float3 optixMin = { scene_min.x, scene_min.y, scene_min.z };
float3 optixMax = { scene_max.x, scene_max.y, scene_max.z };
aabb.set(optixMin, optixMax);
unsigned int numVerts = 0;
unsigned int numFaces = 0;
if (scene->mNumMeshes > 0) {
printf("Number of meshes: %d\n", scene->mNumMeshes);
// get the running total number of vertices & faces for all meshes
for (unsigned int i = 0; i < scene->mNumMeshes; i++) {
numVerts += scene->mMeshes[i]->mNumVertices;
numFaces += scene->mMeshes[i]->mNumFaces;
}
printf("Found %d Vertices and %d Faces\n", numVerts, numFaces);
// set up buffers
optix::Buffer vertices = context->createBuffer(RT_BUFFER_INPUT, RT_FORMAT_FLOAT3, numVerts);
optix::Buffer normals = context->createBuffer(RT_BUFFER_INPUT, RT_FORMAT_FLOAT3, numVerts);
optix::Buffer faces = context->createBuffer(RT_BUFFER_INPUT, RT_FORMAT_UNSIGNED_INT3, numFaces);
optix::Buffer materials = context->createBuffer(RT_BUFFER_INPUT, RT_FORMAT_UNSIGNED_INT, numVerts);
// unused buffer
Buffer tbuffer = context->createBuffer(RT_BUFFER_INPUT, RT_FORMAT_FLOAT2, 0);
// create material
std::string defaultPtxPath = "C:\\ProgramData\\NVIDIA Corporation\\OptiX SDK 4.1.0\\SDK\\build\\lib\\ptx\\";
Program phong_ch = context->createProgramFromPTXFile(defaultPtxPath + "optixPrimitiveIndexOffsets_generated_phong.cu.ptx", "closest_hit_radiance");
Program phong_ah = context->createProgramFromPTXFile(defaultPtxPath + "optixPrimitiveIndexOffsets_generated_phong.cu.ptx", "any_hit_shadow");
Material matl = context->createMaterial();
matl->setClosestHitProgram(0, phong_ch);
matl->setAnyHitProgram(1, phong_ah);
matl["Kd"]->setFloat(0.7f, 0.7f, 0.7f);
matl["Ka"]->setFloat(1.0f, 1.0f, 1.0f);
matl["Kr"]->setFloat(0.0f, 0.0f, 0.0f);
matl["phong_exp"]->setFloat(1.0f);
std::string triangle_mesh_ptx_path(ptxPath("triangle_mesh.cu"));
Program meshIntersectProgram = context->createProgramFromPTXFile(triangle_mesh_ptx_path, "mesh_intersect");
Program meshBboxProgram = context->createProgramFromPTXFile(triangle_mesh_ptx_path, "mesh_bounds");
optix::float3 *vertexMap = reinterpret_cast<optix::float3*>(vertices->map());
optix::float3 *normalMap = reinterpret_cast<optix::float3*>(normals->map());
optix::uint3 *faceMap = reinterpret_cast<optix::uint3*>(faces->map());
unsigned int *materialsMap = static_cast<unsigned int*>(materials->map());
context["vertex_buffer"]->setBuffer(vertices);
context["normal_buffer"]->setBuffer(normals);
context["index_buffer"]->setBuffer(faces);
context["texcoord_buffer"]->setBuffer(tbuffer);
context["material_buffer"]->setBuffer(materials);
Group group = createSingleGeometryGroup(meshIntersectProgram, meshBboxProgram, vertexMap,
normalMap, faceMap, materialsMap, matl);
context["top_object"]->set(group);
context["top_shadower"]->set(group);
vertices->unmap();
normals->unmap();
faces->unmap();
materials->unmap();
}
return 0;
}
return 1;
}
And the relevant function for creating the geometries and filling the buffers:
Group createSingleGeometryGroup(Program meshIntersectProgram, Program meshBboxProgram, optix::float3 *vertexMap,
optix::float3 *normalMap, optix::uint3 *faceMap, unsigned int *materialsMap, Material matl) {
Group group = context->createGroup();
optix::Acceleration accel = context->createAcceleration("Trbvh");
group->setAcceleration(accel);
std::vector<GeometryInstance> gis;
unsigned int vertexOffset = 0u;
unsigned int faceOffset = 0u;
for (unsigned int m = 0; m < scene->mNumMeshes; m++) {
aiMesh *mesh = scene->mMeshes[m];
if (!mesh->HasPositions()) {
throw std::runtime_error("Mesh contains zero vertex positions");
}
if (!mesh->HasNormals()) {
throw std::runtime_error("Mesh contains zero vertex normals");
}
printf("Mesh #%d\n\tNumVertices: %d\n\tNumFaces: %d\n", m, mesh->mNumVertices, mesh->mNumFaces);
// add points
for (unsigned int i = 0u; i < mesh->mNumVertices; i++) {
aiVector3D pos = mesh->mVertices[i];
aiVector3D norm = mesh->mNormals[i];
vertexMap[i + vertexOffset] = optix::make_float3(pos.x, pos.y, pos.z) + aabb.center();
normalMap[i + vertexOffset] = optix::normalize(optix::make_float3(norm.x, norm.y, norm.z));
materialsMap[i + vertexOffset] = 0u;
}
// add faces
for (unsigned int i = 0u; i < mesh->mNumFaces; i++) {
aiFace face = mesh->mFaces[i];
// add triangles
if (face.mNumIndices == 3) {
faceMap[i + faceOffset] = optix::make_uint3(face.mIndices[0], face.mIndices[1], face.mIndices[2]);
}
else {
printf("face indices != 3\n");
faceMap[i + faceOffset] = optix::make_uint3(-1);
}
}
// create geometry
optix::Geometry geometry = context->createGeometry();
geometry->setPrimitiveCount(mesh->mNumFaces);
geometry->setIntersectionProgram(meshIntersectProgram);
geometry->setBoundingBoxProgram(meshBboxProgram);
geometry->setPrimitiveIndexOffset(faceOffset);
optix::GeometryInstance gi = context->createGeometryInstance(geometry, &matl, &matl + 1);
gis.push_back(gi);
vertexOffset += mesh->mNumVertices;
faceOffset += mesh->mNumFaces;
}
printf("VertexOffset: %d\nFaceOffset: %d\n", vertexOffset, faceOffset);
// add all geometry instances to a geometry group
GeometryGroup gg = context->createGeometryGroup();
gg->setChildCount(static_cast<unsigned int>(gis.size()));
for (unsigned i = 0u; i < gis.size(); i++) {
gg->setChild(i, gis[i]);
}
Acceleration a = context->createAcceleration("Trbvh");
gg->setAcceleration(a);
group->setChildCount(1);
group->setChild(0, gg);
return group;
}
Running the above code on a sample file from assimp (using the dwarf.x, file contains 2 meshes) yields this result:
You can see only part of the second mesh (the dwarf's body) is rendered. I tried rendering each mesh separately, one at a time, and they render in full. But when putting them together I get this.
I'm thinking the issue is either with creating the geometry, perhaps I have these lines wrong:
geometry->setPrimitiveCount(mesh->mNumFaces);
geometry->setPrimitiveIndexOffset(faceOffset);
or the assimp postprocessing flags
scene = importer.ReadFile(
path,
aiProcess_Triangulate
//| aiProcess_JoinIdenticalVertices
| aiProcess_SortByPType
| aiProcess_ValidateDataStructure
| aiProcess_SplitLargeMeshes
| aiProcess_FixInfacingNormals
);
(note above, I had to comment out JoinIdenticalVertices because it gave me a horribly wrong result shown below):
Has anyone been able to successfully combine nvidia optix with open asset import library for rendering files with multiple meshes?
I found a solution, although not sure how optimal.
Each mesh still gets its own geometry, however instead of creating single vertex, index and normal buffers which are shared among all geometries, I create separate buffers for each geometry.
Then, instead of
context["vertex_buffer"]->setBuffer(vertices);
context["normal_buffer"]->setBuffer(normals);
context["index_buffer"]->setBuffer(faces);
context["texcoord_buffer"]->setBuffer(tbuffer);
context["material_buffer"]->setBuffer(materials);
I use
geometry["vertex_buffer"]->setBuffer(vertices);
geometry["normal_buffer"]->setBuffer(normals);
geometry["index_buffer"]->setBuffer(faces);
geometry["texcoord_buffer"]->setBuffer(tbuffer);
geometry["material_buffer"]->setBuffer(materials);
The result:

How to read data from a UTexture2D in C++

I am trying to read the pixel data from a populated UTexture2D in an Unreal Engine C++ project. Before I post the question here, I tried to use the method described in this link: https://answers.unrealengine.com/questions/25594/accessing-pixel-values-of-texture2d.html. However, it doesn't work for me. All pixel values I got from the texture are some garbage data.
I just want to get the depth values from the SceneCapture2D and a post-processing material that contains SceneTexture: Depth node. I need the depth values available in C++ so that I can do further processing with OpenCV. In Directx11, staging texture can be used for CPU read, but in the unreal engine, I don't know how to create a 'staging texture' like Dx11 has. I can't get the correct pixel values from the current method which makes me think I may try to access a no-CPU-readable texture.
Here is my experimental code for reading data back from an RGB UTexture2D.
Initialize the RGB Texture:
VideoTextureColor= UTexture2D::CreateTransient(640, 480, PF_B8G8R8A8);
VideoTextureColor->UpdateResource();
VideoUpdateTextureRegionColor = new FUpdateTextureRegion2D(0, 0, 0, 0, 640, 480);
ColorRegionData = new FUpdateTextureRegionsData;
PixelDepthData.Init(FColor(0, 0, 0, 255), 640 * 480);
// Populate the texture with blue color
for (int i = 0; i < 640; i++) {
for (int j = 0; j < 480; j++) {
int idx = j * 640 + i;
PixelDepthData[idx].B = 255;
PixelDepthData[idx].G = 0;
PixelDepthData[idx].R = 0;
PixelDepthData[idx].A = 255;
}
}
UpdateTextureRegions(
VideoTextureColor,
(int32)0,
(uint32)1,
VideoUpdateTextureRegionColor,
(uint32)(4 * 640),
(uint32)4,
(uint8*)PixelDepthData.GetData(),
false,
ColorRegionData
);
Then, update read its value back to the PixelDepthData (TArray type) array and update this texture with values storing in the PixelDepthData, which is its old value.
UpdateTextureRegions(
VideoTextureColor,
(int32)0,
(uint32)1,
VideoUpdateTextureRegionColor,
(uint32)(4 * 640),
(uint32)4,
(uint8*)PixelDepthData.GetData(),
false,
ColorRegionData
);
ENQUEUE_UNIQUE_RENDER_COMMAND_ONEPARAMETER(
FRealSenseDelegator,
ARealSenseDelegator*, RealSenseDelegator, this,
{
FColor* tmpImageDataPtr = static_cast<FColor*>((RealSenseDelegator->VideoTextureColor)->PlatformData->Mips[0].BulkData.Lock(LOCK_READ_ONLY));
for (uint32 j = 0; j < 480; j++) {
for (uint32 i = 0; i < 640; i++) {
uint32 idx = j * 640 + i;
RealSenseDelegator->PixelDepthData[idx] = tmpImageDataPtr[idx];
RealSenseDelegator->PixelDepthData[idx].A = 255;
}
}
(RealSenseDelegator->VideoTextureColor)->PlatformData->Mips[0].BulkData.Unlock();
}
);
All I got is a white color texture instead of a blue color texture in the visualization scene.
Does anyone know how to read the data of the UTexture2D Object?
I figured that out. You have to get the UTexture2D's RHI texture reference first, and then use RHILockTexture2D to read it's data, and you have to do it in the RenderThread. The following code just an example:
FTexture2DResource* uTex2DRes = (FTexture2DResource*)(RealSenseDelegator->VideoTexturePixelDepth)->Resource;
float* cpuDataPtr = (float*)RHILockTexture2D(
uTex2DRes->GetTexture2DRHI(),
0,
RLM_ReadOnly,
destStride,
false);
for (uint32 j = 0; j < 480; j++) {
for (uint32 i = 0; i < 640; i++) {
uint32 idx = j * 640 + i;
// TODO Read the pixel data right here
}
}
RHIUnlockTexture2D(uTex2DRes->GetTexture2DRHI(), 0, false);
To do this in the Render Thread, you have to use the Macro such as ENQUEUE_UNIQUE_RENDER_COMMAND_ONEPARAMETER // If you just one to pass one parameter to the render thread, use this one.+

Extracting skin data from an FBX file

I need to convert animation data from Autodesk's FBX file format to one that is compatible with DirectX; specifically, I need to calculate the offset matrices for my skinned mesh. I have written a converter( which in this case converts .fbx to my own 'scene' format ) in which I would like to calculate an offset matrix for my mesh. Here is code:
//
// Skin
//
if(bHasDeformer)
{
// iterate deformers( TODO: ACCOUNT FOR MULTIPLE DEFORMERS )
for(int i = 0; i < ncDeformers && i < 1; ++i)
{
// skin
FbxSkin *pSkin = (FbxSkin*)pMesh->GetDeformer(i, FbxDeformer::eSkin);
if(pSkin == NULL)
continue;
// bone count
int ncBones = pSkin->GetClusterCount();
// iterate bones
for (int boneIndex = 0; boneIndex < ncBones; ++boneIndex)
{
// cluster
FbxCluster* cluster = pSkin->GetCluster(boneIndex);
// bone ref
FbxNode* pBone = cluster->GetLink();
// Get the bind pose
FbxAMatrix bindPoseMatrix, transformMatrix;
cluster->GetTransformMatrix(transformMatrix);
cluster->GetTransformLinkMatrix(bindPoseMatrix);
// decomposed transform components
vS = bindPoseMatrix.GetS();
vR = bindPoseMatrix.GetR();
vT = bindPoseMatrix.GetT();
int *pVertexIndices = cluster->GetControlPointIndices();
double *pVertexWeights = cluster->GetControlPointWeights();
// Iterate through all the vertices, which are affected by the bone
int ncVertexIndices = cluster->GetControlPointIndicesCount();
for (int iBoneVertexIndex = 0; iBoneVertexIndex < ncVertexIndices; iBoneVertexIndex++)
{
// vertex
int niVertex = pVertexIndices[iBoneVertexIndex];
// weight
float fWeight = (float)pVertexWeights[iBoneVertexIndex];
}
}
}
How do I convert the fbx transforms to a bone offset matrix?