I am trying to clean some of the noise in the point clouds that I obtain using a Intel RealSense D435 camera. In particular, I have noticed that the edges in these point clouds tend to be smoothed out to curves which is making segementation based on normals next to impossible.
I have done some research and found that the Intel Realsense SDK does include some post processing algorithms that aim to preserve edges and clean noise (see here). The examples here, however, all seem to be for modifying the RealSense SDK using c++ and I don't see how I would implement them into ROS.
My question is: is it possible to implement these post processing algorithms using ROS? Currently, I start a camera stream using the rs_camera.launch file, but I don't see any parameters I can set here to enable these post-processing methods. Is there a separate node that I can run for this post processing?
It seems that you have to create your own filtering node.
You can have this launch file running a node that publishes a sensor_msgs::PointCloud2 and then create another one that will convert this message type to rs2::pointcloud and use this in your filters as in the rs-post-processing tutorial. After the filtering, you should convert the rs2::pointcloud back to sensor_msgs::PointCloud2.
Later on, it might be interesting to take a look at nodelets to improve your filtering pipeline, so you can separate the filters in different nodelets and use them in other cases as well.
In that sense, ROS is just a way for you to send/receive messages over different nodes.
Related
I want to write a program that takes a point cloud data and extract the set of points that belong to a vertical plane. The process mustn’t take more than 100 milliseconds. What is the best way to do this?
I tried using RANSAC filter but, it's slow and also the result is not good.
pcl::SACSegmentation<pcl::PointXYZ> seg;
seg.setOptimizeCoefficients(true);
seg.setModelType(pcl::SACMODEL_PLANE);
seg.setMethodType(pcl::SAC_RANSAC);
seg.setMaxIterations(1000);
seg.setDistanceThreshold(0.01);
First of all, I would recommend to use the latest PCL release (or even the master branch), compiled from source. Compared to PCL 1.9.1, this includes several speed improvements in the sample consensus module. Additionally, by compiling from source, you make sure that you can use everything your computer is capable of (e.g. SIMD instructions).
With the latest PCL release (or master branch) you can also use the parallel RANSAC implementation by calling seg.setNumberOfThreads(0).
If this is still too slow, you can try to downsample the cloud before passing it to SACSegmentation, e.g. with https://pointclouds.org/documentation/classpcl_1_1_random_sample.html
If you only want vertical planes (that is, planes that are parallel to a specified axis), you should use SACMODEL_PARALLEL_PLANE instead of SACMODEL_PLANE and call https://pointclouds.org/documentation/classpcl_1_1_s_a_c_segmentation.html#a23abc3e522ccb2b2846a6c9b0cf7b7d3 and https://pointclouds.org/documentation/classpcl_1_1_s_a_c_segmentation.html#a7a2dc31039a1717f83ca281f6970eb18
It seems that dlib needs a loss layer that dictates how the layers most distant to our input layer are treated. I cannot find any documentation towards the loss layers but it seems that there is no way to have just some summation layer.
Summing up all the values of the last layer would be exactly what I need for the regression, though (see also: https://deeplearning4j.org/linear-regression)
I was thinking along the lines of writing a custom loss layer but could not find information about this, either.
So, have I overseen some corresponding layer here or is there a possibility to have what I need?
The loss layers in dlib are listed in the menu on dlib's machine learning page. Look for the words "loss layers". There is lots of documentation.
The current released version of dlib doesn't include a regression loss. However, if you get the current code from github you can use the new loss_mean_squared layer to do regression. See: https://github.com/davisking/dlib/blob/master/dlib/dnn/loss_abstract.h
I am writing C++ class for managing multiple cameras and reading frames from them. Let's say it is wrapper for OpenCV. Currently I am finding cameras by trying to create devices from 0-10 range and If there is output I know that I've found working camera. I can always save internal IDs of those cameras to distinguish them but what If another camera is plugged in? It may break the order of IDs. So is there any way to distinguish OpenCV cameras for example by getting their hardware IDs?
I know this doesn't help you much, but the short answer is "No, OpenCV doesn't currently provide that capability."
According to the doc, any hardware ids are not properties you can retrieve using the get method or any other.
Having said that, if you're very intent on using OpenCV, I would still test the behavior of OpenCV 2.4.10 on various platforms and using various middleware and see how it behaves. If you get a consistent behavior, then you can run with it, but be somewhat ready for it to break in the future. What would work for you is that OpenCV is using various middleware in the backend, such as V4L, Qt, etc., and these are well-maintained and more-or-less consistent.
In retrospect, I would stay away from OpenCV's video interface altogether right now for commercial software, unless you're okay with the situation I described. Beware that OpenCV 3.0 videoio library is unstable at this point and has open bug reports.
Kinect Fusion requires the client code to specify what is effectively a bounding-box and voxel resolution within the box before initialisation. The details of the reconstruction within the box are held in GPU memory, and therefore one runs into limits rather quickly. Certainly for a space the size of, say, a standard residential house at high resolution the amount of GPU memory required is (way too) high.
The Fusion SDK allows one to copy the data to/from GPU memory, and to reset the bounding volume, resolution etc at any time, so in theory one could synthesise a large volume by stitching together a number of small volumes, each of which a normal GPU can handle. It seems to me to be a technique with quite a few subtle and difficult problems associated with it though.
Nevertheless this seems to have been done with kintinuous. But kintinuous does not seem to have any source code (or object code for that matter) publicly available. This is also mentioned in this SO post.
I was wondering if this has been implemented in any form with public source code. I've not been able to find anything on this except the above mentioned kintinuous.
Kintinuous is opensource now since it's first commit on Oct 22, 2015
Here is another blog on the tag kintinuous: https://hackaday.com/tag/kintinuous/
You can have a experimental open source large-scale kinect fusion source code in PCL. they do volume stitch when camera pose crosses some threshold. check this
and there is actually new version of scalable kinect fusion in MSR, but they haven't yet put it into the SDK, so you can't use it right a way.
They use a hierarchical data structure to store the unbouded reconstruction volume. you can go check this then download their paper, and implement it by yourself.
============EDIT=============
you can have another code from technology university of Munich, named 'fastfusion'. they use a multi-resolution octree to store the voxels, and extract mesh every second in other thread. It uses OpenNI.
It doesn't contain camera tracking, but you can use their dvo-slam for visual odometry.
and a recently released project named InfiniTAM use Hash table to store the voxels. it can runs on Kinect v2.
I found this code that supports moving volume:
http://www.ccs.neu.edu/research/gpc/rxkinfu/index.html
It is based on KinFu from PCL.
What kind of debugging is available for image processing/computer vision/computer graphics applications in C++? What do you use to track errors/partial results of your method?
What I have found so far is just one tool for online and one for offline debugging:
bmd: attaches to a running process and enables you to view a block of memory as an image
imdebug: enables printf-style of debugging
Both are quite outdated and not really what I would expect.
What would seem useful for offline debugging would be some style of image logging, lets say a set of commands which enable you to write images together with text (probably in the form of HTML, maybe hierarchical), easy to switch off at both compile and run time, and the least obtrusive it can get.
The output could look like this (output from our simple tool):
http://tsh.plankton.tk/htmldebug/d8egf100-RF-SVM-RBF_AC-LINEAR_DB.html
Are you aware of some code that goes in this direction?
I would be grateful for any hints.
Coming from a ray tracing perspective, maybe some of those visual methods are also useful to you (it is one of my plans to write a short paper about such techniques):
Surface Normal Visualization. Helps to find surface discontinuities. (no image handy, the look is very much reminiscent of normal maps)
color <- rgb (normal.x+0.5, normal.y+0.5, normal.z+0.5)
Distance Visualization. Helps to find surface discontinuities and errors in finding a nearest point. (image taken from an abandoned ray tracer of mine)
color <- (intersection.z-min)/range, ...
Bounding Volume Traversal Visualization. Helps visualizing a bounding volume hierarchy or other hierarchical structures, and helps to see the traversal hotspots, like a code profiler (e.g. Kd-trees). (tbp of http://ompf.org/forum coined the term Kd-vision).
color <- number_of_traversal_steps/f
Bounding Box Visualization (image from picogen or so, some years ago). Helps to verify the partitioning.
color <- const
Stereo. Maybe useful in your case as for the real stereographic appearance. I must admit I never used this for debugging, but when I think about it, it could prove really useful when implementing new types of 3d-primitives and -trees (image from gladius, which was an attempt to unify realtime and non-realtime ray tracing)
You just render two images with slightly shifted position, focusing on some point
Hit-or-not visualization. May help to find epsilon errors. (image taken from metatrace)
if (hit) color = const_a;
else color = const_b
Some hybrid of several techniques.
Linear interpolation: lerp(debug_a, debug_b)
Interlacing: if(y%2==0) debug_a else debug_b
Any combination of ideas, for example the color-tone from Bounding Box Visualization, but with actual scene-intersection and lighting applied
You may find some more glitches and debugging imagery on http://phresnel.org , http://phresnel.deviantart.com , http://picogen.deviantart.com , and maybe http://greenhybrid.deviantart.com (an old account).
Generally, I prefer to dump bytearray of currently processed image as raw data triplets and run Imagemagick to create png from it with number e.g img01.png. In this way i can trace the algorithms very easy. Imagemagick is run from the function in the program using system call. This make possible do debug without using any external libs for image formats.
Another option, if you are using Qt is to work with QImage and use img.save("img01.png") from time to time like a printf is used for debugging.
it's a bit primitive compared to what you are looking for, but i have done what you suggested in your OP using standard logging and by writing image files. typically, the logging and signal export processes and staging exist in unit tests.
signals are given identifiers (often input filename), which may be augmented (often process name or stage).
for development of processors, it's quite handy.
adding html for messages would be simple. in that context, you could produce viewable html output easily - you would not need to generate any html, just use html template files and then insert the messages.
i would just do it myself (as i've done multiple times already for multiple signal types) if you get no good referrals.
In Qt Creator you can watch image modification while stepping through the code in the normal C++ debugger, see e.g. http://labs.qt.nokia.com/2010/04/22/peek-and-poke-vol-3/