normal estimation of mesh open3d c++ - c++

I have a problem when re-computing my surface normals of a mesh in open3d. The problem is that the estimation is not good enough and I don't know how to make it better. From the image below it cannot be seen that the mesh has a hole in the middle of the belly.
However, if the same mesh is seen from the side it clearly has a hole.
If I then use MeshLab to do the normal estimation the hole can suddenly clearly be seen. MeshLab support 4 different functions for the normal estimation of a mesh but the result are more or less the same no matter which function I use.
Here is the same mesh after estimating normals with MeshLab.
I find it very strange that open3d does no even come close to the accuracy of MeshLabs normal estimation, and I believe that it is most likely because I miss some important calculation before using open3d normal estimation function.
Here is the code which is use for the normal estimation in open3d:
void ReconstructionSystem::constructMeshDeformation(glm::vec3& intersectionPosition, glm::vec3& robotPosition) {
double depthOfProbe = glm::distance(intersectionPosition.z, robotPosition.z);
double affectedArea = 0.015 * (depthOfProbe*100.0); // 0.015 is a random value the function simply works good with it
if (affectedArea > 0.08) {
affectedArea = 0.08;
}
deformatedMesh = std::make_shared<open3d::geometry::TriangleMesh>(*final_mesh);
int i = 0;
for each (Eigen::Vector3d vertex in deformatedMesh->vertices_) {
glm::vec3 vex = glm::vec3(vertex.x(), vertex.y(), vertex.z());
double dist = glm::distance(vex, intersectionPosition);
if (dist < affectedArea) {
double ratio = dist / affectedArea;
double deformationAmount = glm::cos((2.0 * M_PI * ratio) / 4.0);
deformatedMesh->vertices_.at(i).z() -= depthOfProbe * deformationAmount;
}
i++;
}
*deformatedMesh = deformatedMesh->ComputeVertexNormals();
}

Related

Determine minimum parallax for correct triangulation of 3D points in OpenCV

I am triangulating 3D points using OpenCV triangulation function for monocular sequence that sometimes works fine but I have noticed when two camera poses are close to each other then the triangulated points are far away. I can understand the issue that is since the camera poses are close then the ray intersection from two cameras is being take place far away from the camera. That is why it creates the 3D points far away. I have also noticed that the distance requirement between two cameras for correct triangulation varies in different cases.Currently I am trying to find parallax between two pose and if that is above a certain threshold(I have chosen 27) then proceed to triangulate but I does not look correct for all the cases.
My code for calculating parallax as following-
float checkAvgParallex(SE3& prevPose, SE3& currPose, std::vector<Point2f>& prevPoints, std::vector<Point2f>& currPoints, Mat& K) {
Eigen::Matrix3d relRot = Eigen::Matrix3d::Identity();
Eigen::Matrix3d prevRot = prevPose.rotationMatrix();
Eigen::Matrix3d currRot = currPose.rotationMatrix();
relRot = prevRot * currRot;
float avg_parallax = 0.;
int nbparallax = 0;
std::set<float> set_parallax;
bearingVectors_t prevBVs;
bearingVectors_t currBVs;
points2bearings(prevPoints, K, prevBVs);
points2bearings(currPoints, K, currBVs);
for (int i = 0; i < prevPoints.size(); i++) {
Point2f unpx = projectCamToImage(relRot * currBVs[i], K);
float parallax = cv::norm(unpx - prevPoints[i]);
avg_parallax += parallax;
nbparallax++;
set_parallax.insert(parallax);
}
if (nbparallax == 0)
return 0.0;
avg_parallax /= nbparallax;
auto it = set_parallax.begin();
std::advance(it, set_parallax.size() / 2);
avg_parallax = *it;
return avg_parallax;
}
And sometime when parallax between camera does not exceed 27 so, triangulation won't work, due to this my further pose calculation in SLAM system stops due to lack of 3D points.
So can anyone suggest me alternative strategy using which I can estimate correct 3D points and my SLAM system wont suffer due to lack of 3D points, please?

How to properly handle refraction in raytracing

I am currently working on a raytracer just for fun and I have trouble with the refraction handling.
The code source of the whole raytracer can be found on Github EDIT: The code migrated to Gitlab.
Here is an image of the render:
The right sphere is set to have a refraction indice of 1.5 (glass).
On top of the refraction, I want to handle a "transparency" coefficient which is defined as such :
0 --> Object is 100% opaque
1 --> Object is 100% transparent (no trace of the original object's color)
This sphere has a transparency of 1.
Here is the code handling the refraction part. It can be found on github here.
Color handleTransparency(const Scene& scene,
const Ray& ray,
const IntersectionData& data,
uint8 depth)
{
Ray refracted(RayType::Transparency, data.point, ray.getDirection());
Float_t eta = data.material->getRefraction();
if (eta != 1 && eta > Globals::Epsilon)
refracted.setDirection(Tools::Refract(ray.getDirection(), data.normal, eta));
refracted.setOrigin(data.point + Globals::Epsilon * refracted.getDirection());
return inter(scene, refracted, depth + 1);
}
// http://graphics.stanford.edu/courses/cs148-10-summer/docs/2006--degreve--reflection_refraction.pdf
Float_t getFresnelReflectance(const IntersectionData& data, const Ray& ray)
{
Float_t n = data.material->getRefraction();
Float_t cosI = -Tools::DotProduct(ray.getDirection(), data.normal);
Float_t sin2T = n * n * (Float_t(1.0) - cosI * cosI);
if (sin2T > 1.0)
return 1.0;
using std::sqrt;
Float_t cosT = sqrt(1.0 - sin2T);
Float_t rPer = (n * cosI - cosT) / (n * cosI + cosT);
Float_t rPar = (cosI - n * cosT) / (cosI + n * cosT);
return (rPer * rPer + rPar * rPar) / Float_t(2.0);
}
Color handleReflectionAndRefraction(const Scene& scene,
const Ray& ray,
const IntersectionData& data,
uint8 depth)
{
bool hasReflexion = data.material->getReflexion() > Globals::Epsilon;
bool hasTransparency = data.material->getTransparency() > Globals::Epsilon;
if (!(hasReflexion || hasTransparency) || depth >= MAX_DEPTH)
return 0;
Float_t reflectance = data.material->getReflexion();
Float_t transmittance = data.material->getTransparency();
Color reflexion;
Color transparency;
if (hasReflexion && hasTransparency)
{
reflectance = getFresnelReflectance(data, ray);
transmittance = 1.0 - reflectance;
}
if (hasReflexion)
reflexion = handleReflection(scene, ray, data, depth) * reflectance;
if (hasTransparency)
transparency = handleTransparency(scene, ray, data, depth) * transmittance;
return reflexion + transparency;
}
Tools::Refract is simply calling glm::refract internally. (So that I can change easily if I want)
I don't handle notions of n1 and n2: n2 is considered to always be 1 for air.
Am I mising something obvious ?
EDIT
After adding a way to know if a ray is inside an object (and negating the normal if so) I have this :
While looking around to find help, I stumbled upon this post but I don't think the answer answers anything. By reading it, I don't understand what I'm supposed to do at all.
EDIT 2
I've tried a lot of things and I am currently at this point :
It's better but I'm still not sure if it's right. I'm using this image as an inspiration :
But this one is using two indexes of refraction (To be closer to reality) while I want to simplify and always consider air as the second (in or out) material.
What I essentially changed in my code is here :
inline Vec_t Refract(Vec_t v, const IntersectionData& data, Float_t eta)
{
Float_t n = eta;
if (data.isInside)
n = 1.0 / n;
double cosI = Tools::DotProduct(v, data.normal);
return v * n - data.normal * (-cosI + n * cosI);
}
Here is another view of the same spheres :
EDIT: I've figured that the previous version of this was not entirely correct so I edit the answer.
After reading all the comments, the new versions of the question and doing some experimentation myself I produced the following version of refract routine:
float3 refract(float3 i, float3 n, float eta)
{
eta = 2.0f - eta;
float cosi = dot(n, i);
float3 o = (i * eta - n * (-cosi + eta * cosi));
return o;
}
This time calling it does not require any additional operations:
float3 refr = refract(rayDirection, normal, refrIdx);
The only thing I am still not sure is the inverting of the refractive index when doing the inside ray intersection. In my test the produced image haven't differ much no matter I inverted the index or not.
Below some images with different indices:
For more images see the link, because the site do not allow me to put more of them here.
I am answering this as a physicist rather than a programmer as haven't had time to read all the code so won't be giving the code to do the fix just the general idea.
From what you have said above the black ring is for when n_object is less than n_air. This is only usually true if you are inside an object say if you were inside water or the like but materials have been constructed with weird properties like that and it should be supported.
In this type of situation there are rays of light that can't be diffracted as the diffraction formula put the refracted ray on the SAME side of the interface between the materials, which obviously doesn't make sense as diffraction. In this situation the surface will instead act like it's a reflective surface. This is the situation that is often referred to as total internal reflection.
If being fully exact then almost ever refractive object will also partially reflective too and the fraction of light that is reflected or transmitted (and therefore refracted) is given by the Fresnel equations. For this case though it would still be a good approximation to just treat is as reflective if the angle is too far and transmitting (and therefore refractive) otherwise.
Also there are situations where this black ring effect can be seen if reflection is not possible (due to it being dark in those directions) but light that is transmitted being possible. This could be done by say taking a tube of card that fits tightly to the edge of the object and is pointed directly away and only shining light inside the tube not outside.

cocos2dx detect intersection with polygon sprite

I am using cocos2d-x 3.8.
I try to create two polygon sprites with the following code.
I know we can detect intersect with BoundingBox but is too rough.
Also, I know we can use Cocos2d-x C++ Physics engine to detect collisions but doesn't it waste a lot of resource of the mobile device? The game I am developing does not need physics engine.
is there a way to detect the intersect of polygon sprites?
Thank you.
auto pinfoTree = AutoPolygon::generatePolygon("Tree.png");
auto treeSprite= Sprite::create(pinfoTree);
treeSprite-> setPosition(width / 4 * 3 - 30 , height / 2 - 200);
this->addChild(treeSprite);
auto pinfoBird = AutoPolygon::generatePolygon("Bird.png");
auto Bird= Sprite::create(pinfoTree);
Bird->setPosition(width / 4 * 3, height / 2);
this->addChild(Bird)
This is a bit more complicated: AutoPolygon gives you a bunch of triangles - the PhysicsBody::createPolygon requires a convex polygon with clockwise winding… so these are 2 different things. The vertex count might even be limited. I think Box2d’s maximum count for 1 polygon is 8.
If you want to try this you’ll have to merge the triangles to form polygons. An option would be to start with one triangle and add more as long as the whole thing stays convex. If you can’t add any more triangles start a new polygon. Add all the polygons as PhysicsShapes to your physics body to form a compound object.
I would propose that you don’t follow this path because
Autopolygon is optimized for rendering - not for best fitting
physics - that is a difference. A polygon traced with Autopolygon will always be bigger than the original sprite - Otherwise you would see rendering artifacts.
You have close to no control over the generated polygons
Tracing the shape in the app will increase your startup time
Triangle meshes and physics outlines are 2 different things
I would try some different approach: Generate the collision shapes offline. This gives you a bunch of advantages:
You can generate and tweak the polygons in a visual editor e.g. by
using PhysicsEditor
Loading the prepares polygons is way faster
You can set additional parameters like mass etc
The solution is battle proven and works out of the box
But if you want to know how polygon intersect work. You can look at this code.
// Calculate the projection of a polygon on an axis
// and returns it as a [min, max] interval
public void ProjectPolygon(Vector axis, Polygon polygon, ref float min, ref float max) {
// To project a point on an axis use the dot product
float dotProduct = axis.DotProduct(polygon.Points[0]);
min = dotProduct;
max = dotProduct;
for (int i = 0; i < polygon.Points.Count; i++) {
flaot d = polygon.Points[i].DotProduct(axis);
if (d < min) {
min = dotProduct;
} else {
if (dotProduct> max) {
max = dotProduct;
}
}
}
}
// Calculate the distance between [minA, maxA] and [minB, maxB]
// The distance will be negative if the intervals overlap
public float IntervalDistance(float minA, float maxA, float minB, float maxB) {
if (minA < minB) {
return minB - maxA;
} else {
return minA - maxB;
}
}
// Check if polygon A is going to collide with polygon B.
public boolean PolygonCollision(Polygon polygonA, Polygon polygonB) {
boolean result = true;
int edgeCountA = polygonA.Edges.Count;
int edgeCountB = polygonB.Edges.Count;
float minIntervalDistance = float.PositiveInfinity;
Vector edge;
// Loop through all the edges of both polygons
for (int edgeIndex = 0; edgeIndex < edgeCountA + edgeCountB; edgeIndex++) {
if (edgeIndex < edgeCountA) {
edge = polygonA.Edges[edgeIndex];
} else {
edge = polygonB.Edges[edgeIndex - edgeCountA];
}
// ===== Find if the polygons are currently intersecting =====
// Find the axis perpendicular to the current edge
Vector axis = new Vector(-edge.Y, edge.X);
axis.Normalize();
// Find the projection of the polygon on the current axis
float minA = 0; float minB = 0; float maxA = 0; float maxB = 0;
ProjectPolygon(axis, polygonA, ref minA, ref maxA);
ProjectPolygon(axis, polygonB, ref minB, ref maxB);
// Check if the polygon projections are currentlty intersecting
if (IntervalDistance(minA, maxA, minB, maxB) > 0)
result = false;
return result;
}
}
The function can be used this way
boolean result = PolygonCollision(polygonA, polygonB);
I once had to program a collision detection algorithm where a ball was to collide with a rotating polygon obstacle. In my case the obstacles where arcs with certain thickness. and where moving around an origin. Basically it was rotating in an orbit. The ball was also rotating around an orbit about the same origin. It can move between orbits. To check the collision I had to just check if the balls angle with respect to the origin was between the lower and upper bound angles of the arc obstacle and check if the ball and the obstacle where in the same orbit.
In other words I used the various constrains and properties of the objects involved in the collision to make it more efficient. So use properties of your objects to cause the collision. Try using a similar approach depending on your objects

Implementing soft shadows in a ray tracer

what I am trying to do is implementing soft shadows in my simple ray tracer, developed in C++. The idea behind this, if I understood correctly, is to shoot multiple rays towards the light, instead of a single ray towards the center of the light, and average the results. The rays are therefore shot in different positions of the light. So far I am using random points, which I don't know if it is correct or if I should use points regularly distributed on the light surface. Assuming that I am doing right, I choose a random point on the light, which in my framework is implemented as a sphere. This is given by:
Vec3<T> randomPoint() const
{
T x;
T y;
T z;
// random vector in unit sphere
std::random_device rd; //used for the new <random> library
std::mt19937 gen(rd());
std::uniform_real_distribution<> dis(-1, 1);
do
{
x = dis(gen);
y = dis(gen);
z = dis(gen);
} while (pow(x, 2) + pow(y, 2) + pow(z, 2) > 1); // simple rejection sampling
return center + Vec3<T>(x, y, z) * radius;
}
After this, I don't know how exactly I should move since my rendering equation (in my simple ray tracer) is defined as follows:
Vec3<float> surfaceColor = 0
for(int i < 0; i < lightsInTheScene.size(); i++){
surfaceColor += obj->surfaceColor * transmission *
std::max(float(0), nHit.dot(lightDirection)) * g_lights[i]->emissionColor;
}
return surfaceColor + obj->emissionColor;
where transmission is a simple float which is set to 0 in case the ray that goes from my hitPoint to the lightCenter used to find an object in the middle.
So, what I tried to do was:
creating multiple rays towards random points on the light
counting how many of them hit an object on their path and memorize this number
for simplicity: Let's imagine in my case that I shoot 3 shadow rays from my point towards random points on the light. Only 2 of 3 rays reach the light. Therefore the final color of my pixel will be = color * shadowFactor where shadowFactor = 2/3. In my equation then I delete the transmission factor (which is now wrong) and I use the shadowFactor instead. The problem is that in my equation I have:
std::max(float(0), nHit.dot(lightDirection))
Which I don't know how to change since I don't have anymore a lightDirection which points towards the center of the light. Can you please help me understanding what should I do it and what's wrong so far? Thanks in advance!
You should evaluate the entire BRDF for the picked light samples. Then, you will also have the light direction (vector from object position to picked light sample). And you can average these results. Note that most area lights have a non-isotropic light emission characteristic (i.e. the amount of light emitted from a point varies by the outgoing direction).
Averaging the visibility does not produce correct results (although they are usually visually plausible).

Linear sampled Gaussian blur quality issue

I recently implemented a linear sampled gaussian blur based on this article: Linear Sampled Gaussian Blur
It generally came out well, however it appears there is slight aliasing on text and thinner collections of pixels. I'm pretty stumped as to what is causing this, is it an issue with my shader or weight calculations or is it an inherit draw back of using this method?
I'd like to add that I don't run into this issue when I sample each pixel regularly instead of using bilinear filtering.
Any insights are much appreciated. Here's a code sample of how I work out my weights:
int support = int(sigma * 3.0f);
float total = 0.0f;
weights.push_back(exp(-(0*0)/(2*sigma*sigma))/(sqrt(2*constants::pi)*sigma));
total += weights.back();
offsets.push_back(0);
for (int i = 1; i <= support; i++)
{
float w1 = exp(-(i*i)/(2*sigma*sigma))/(sqrt(2*constants::pi)*sigma);
float w2 = exp(-((i+1)*(i+1))/(2*sigma*sigma))/(sqrt(2*constants::pi)*sigma);
weights.push_back(w1 + w2);
total += 2.0f * weights[i];
offsets.push_back((i * w1 + (i + 1) * w2) / weights[i]);
}
for (int i = 0; i < support; i++)
{
weights[i] /= total;
}
And here is the fragment shader (there is another vertical version of this shader too):
void main()
{
vec3 acc = texture2D(tex_object, v_tex_coord.st).rgb*weights[0];
for (int i = 1; i < NUM_SAMPLES; i++)
{
acc += texture2D(tex_object, (v_tex_coord.st+(vec2(offsets[i], 0.0)/tex_size))).rgb*weights[i];
acc += texture2D(tex_object, (v_tex_coord.st-(vec2(offsets[i], 0.0)/tex_size))).rgb*weights[i];
}
gl_FragColor = vec4(acc, 1.0);
Here is a screenshot depicting the issue:
This looks like a correct gaussian blur to me. The extent to which text is disrupted depends on your sigma. What value are you using?
Also I would check the scaling matrix for the projection you are using.
If you want to blur but without affecting text and thin pixel lines, you might think of
compositing the result with the output of a mild high-pass filter
use a smaller sigma
change the shape of the kernel so it's not gaussian: rather than exp(-i*i/s*s), you might try a function with higher excess kurtosis. You could try a linear up/down function, or one of the functions listed on this page instead: http://en.wikipedia.org/wiki/Kurtosis . They will all lead to blurs with varying degrees of disrupting fine detail.
This is an inherent issue with the bilinear filtering. It's unavoidable.