Trouble when using Efficient_Ransac in CGAL - c++

I want to use the Efficient Ransac implementation of CGAL, but whenever I try to set my own parameters, the algorithm doesn't detect any shape anymore.
This work is related to the Polyfit implementation in CGAL. I want to fine tune the plane detection to see the influence it has on the algorithm. When I use the standard call to ransac.detect(), it works perfectly. However, when I want to set my own parameters it just doesn't find any plane, even if I set them manually to the default values.
Here is my code, strongly related to this example
#include <CGAL/Exact_predicates_inexact_constructions_kernel.h>
#include <CGAL/IO/read_xyz_points.h>
#include <CGAL/IO/Writer_OFF.h>
#include <CGAL/property_map.h>
#include <CGAL/Surface_mesh.h>
#include <CGAL/Shape_detection/Efficient_RANSAC.h>
#include <CGAL/Polygonal_surface_reconstruction.h>
#ifdef CGAL_USE_SCIP
#include <CGAL/SCIP_mixed_integer_program_traits.h>
typedef CGAL::SCIP_mixed_integer_program_traits<double> MIP_Solver;
#elif defined(CGAL_USE_GLPK)
#include <CGAL/GLPK_mixed_integer_program_traits.h>
typedef CGAL::GLPK_mixed_integer_program_traits<double> MIP_Solver;
#endif
#if defined(CGAL_USE_GLPK) || defined(CGAL_USE_SCIP)
#include <CGAL/Timer.h>
#include <fstream>
typedef CGAL::Exact_predicates_inexact_constructions_kernel Kernel;
typedef Kernel::Point_3 Point;
typedef Kernel::Vector_3 Vector;
// Point with normal, and plane index
typedef boost::tuple<Point, Vector, int> PNI;
typedef std::vector<PNI> Point_vector;
typedef CGAL::Nth_of_tuple_property_map<0, PNI> Point_map;
typedef CGAL::Nth_of_tuple_property_map<1, PNI> Normal_map;
typedef CGAL::Nth_of_tuple_property_map<2, PNI> Plane_index_map;
typedef CGAL::Shape_detection::Efficient_RANSAC_traits<Kernel, Point_vector, Point_map, Normal_map> Traits;
typedef CGAL::Shape_detection::Efficient_RANSAC<Traits> Efficient_ransac;
typedef CGAL::Shape_detection::Plane<Traits> Plane;
typedef CGAL::Shape_detection::Point_to_shape_index_map<Traits> Point_to_shape_index_map;
typedef CGAL::Polygonal_surface_reconstruction<Kernel> Polygonal_surface_reconstruction;
typedef CGAL::Surface_mesh<Point> Surface_mesh;
int main(int argc, char ** argv)
{
Point_vector points;
// Loads point set from a file.
const std::string &input_file = argv[1];
//const std::string input_file(input);
std::ifstream input_stream(input_file.c_str());
if (input_stream.fail()) {
std::cerr << "failed open file \'" <<input_file << "\'" << std::endl;
return EXIT_FAILURE;
}
std::cout << "Loading point cloud: " << input_file << "...";
CGAL::Timer t;
t.start();
if (!input_stream ||
!CGAL::read_xyz_points(input_stream,
std::back_inserter(points),
CGAL::parameters::point_map(Point_map()).normal_map(Normal_map())))
{
std::cerr << "Error: cannot read file " << input_file << std::endl;
return EXIT_FAILURE;
}
else
std::cout << " Done. " << points.size() << " points. Time: " << t.time() << " sec." << std::endl;
// Shape detection
Efficient_ransac ransac;
ransac.set_input(points);
ransac.add_shape_factory<Plane>();
std::cout << "Extracting planes...";
t.reset();
// Set parameters for shape detection.
Efficient_ransac::Parameters parameters;
// Set probability to miss the largest primitive at each iteration.
parameters.probability = 0.05;
// Detect shapes with at least 500 points.
parameters.min_points = 100;
// Set maximum Euclidean distance between a point and a shape.
parameters.epsilon = 0.01;
// Set maximum Euclidean distance between points to be clustered.
parameters.cluster_epsilon = 0.01;
// Set maximum normal deviation.
// 0.9 < dot(surface_normal, point_normal);
parameters.normal_threshold = 0.9;
// Detect shapes.
ransac.detect(parameters);
//ransac.detect();
Efficient_ransac::Plane_range planes = ransac.planes();
std::size_t num_planes = planes.size();
std::cout << " Done. " << num_planes << " planes extracted. Time: " << t.time() << " sec." << std::endl;
// Stores the plane index of each point as the third element of the tuple.
Point_to_shape_index_map shape_index_map(points, planes);
for (std::size_t i = 0; i < points.size(); ++i) {
// Uses the get function from the property map that accesses the 3rd element of the tuple.
int plane_index = get(shape_index_map, i);
points[i].get<2>() = plane_index;
}
//////////////////////////////////////////////////////////////////////////
std::cout << "Generating candidate faces...";
t.reset();
Polygonal_surface_reconstruction algo(
points,
Point_map(),
Normal_map(),
Plane_index_map()
);
std::cout << " Done. Time: " << t.time() << " sec." << std::endl;
//////////////////////////////////////////////////////////////////////////
Surface_mesh model;
std::cout << "Reconstructing...";
t.reset();
if (!algo.reconstruct<MIP_Solver>(model)) {
std::cerr << " Failed: " << algo.error_message() << std::endl;
return EXIT_FAILURE;
}
const std::string& output_file(input_file+"_result.off");
std::ofstream output_stream(output_file.c_str());
if (output_stream && CGAL::write_off(output_stream, model))
std::cout << " Done. Saved to " << output_file << ". Time: " << t.time() << " sec." << std::endl;
else {
std::cerr << " Failed saving file." << std::endl;
return EXIT_FAILURE;
}
//////////////////////////////////////////////////////////////////////////
// Also stores the candidate faces as a surface mesh to a file
Surface_mesh candidate_faces;
algo.output_candidate_faces(candidate_faces);
const std::string& candidate_faces_file(input_file+"_candidate_faces.off");
std::ofstream candidate_stream(candidate_faces_file.c_str());
if (candidate_stream && CGAL::write_off(candidate_stream, candidate_faces))
std::cout << "Candidate faces saved to " << candidate_faces_file << "." << std::endl;
return EXIT_SUCCESS;
}
#else
int main(int, char**)
{
std::cerr << "This test requires either GLPK or SCIP.\n";
return EXIT_SUCCESS;
}
#endif // defined(CGAL_USE_GLPK) || defined(CGAL_USE_SCIP)
When launched, I have the following message:
Loading point cloud: Scene1/test.xyz... Done. 169064 points. Time: 0.428 sec.
Extracting planes... Done. 0 planes extracted. Time: 8.328 sec.
Generating candidate faces... Done. Time: 0.028 sec.
Reconstructing... Failed: at least 4 planes required to reconstruct a closed surface mesh (only 1 provided)
While I have this when launching the code the ransac detection function without parameters:
Loading point cloud: Scene1/test.xyz... Done. 169064 points. Time: 0.448 sec.
Extracting planes... Done. 18 planes extracted. Time: 3.088 sec.
Generating candidate faces... Done. Time: 94.536 sec.
Reconstructing... Done. Saved to Scene1/test.xyz_result.off. Time: 30.28 sec.
Can someone help me setting my own parameters for the ransac shape detection?

However, when I want to set my own parameters it just doesn't find any
plane, even if I set them manually to the default values.
Just to be sure: "setting them manually to the default values" is not what you are doing in the code you shared.
Default values are documented as:
1% of the total number of points for min_points, which should be around 1700 points in your case, not 100
1% of the bounding box diagonal for epsilon and cluster_epsilon. For that obviously I don't know if that is what you used (0.01) as I don't have access to your point set, but if you want to reproduce default values, you should use the CGAL::Bbox_3 object at some point
If you use these values, there's no reason why it should behave differently than with no parameters given (if it does not work, then please let me know because there may be a bug).

Related

How do you access the individual vertices inside of a pcl::PolygonMesh?

I'm currently using visual studio 2022
I'm using vcpkg pcl:x64 library install.
pcl version: 1.9.1-12
I'm expecting to be able to access 3 vertices per polygon.
Unfortunately, I can't seem to access the vertices associated with each triangle.
#include <Eigen/Dense>
#include <pcl/common/io.h>
#include <pcl/filters/extract_indices.h>
#include <pcl/filters/passthrough.h>
#include <pcl/point_cloud.h>
#include <pcl/point_types.h>
#include <pcl/PolygonMesh.h>
#include <pcl/TextureMesh.h>
int main
{
pcl::PolygonMesh mesh;
pcl::io::loadPolygonFileOBJ("pathtomesh.meshfile.obj", mesh);
pcl::PointXYZ v = mesh.polygons[0].vertices[0];
}
the error i recieve is:
no suitable constructor exists to convert from "boost::random::seed_seq::result_type" to "pcl::PointXYZ"
It seems as though vertices is an unsigned int and not a pcl::PointXYZ. This is kinda weird to me because I was expecting a double or a floating point to store the vertices coordinates. It turns out that mesh.polygons[0].vertices[0] returns the indices of each point in the mesh that are stored in the point cloud. So i was able to find the points that the function mesh.polygons[0].vertices[0] were pointing at by using converting the mesh to a pcl::PointCloudpcl::PointXYZ and putting the indexes into that function.
pcl::PolygonMesh mesh;
pcl::io::loadPolygonFileOBJ("D:\\testOBJs\\cube.obj", mesh);
pcl::PointCloud<pcl::PointXYZ>::Ptr allVertices(new pcl::PointCloud<pcl::PointXYZ>);
pcl::fromPCLPointCloud2(mesh.cloud, *allVertices);
std::cout << "All Vertices" << std::endl;
for (int i = 0; i < allVertices->size(); i++)
{
std::cout << std::to_string(i) + " " << allVertices->points[i] << std::endl;
}
std::cout << "All Polygons" << std::endl;
for (int i = 0; i < mesh.polygons.size(); i++)
{
std::cout << std::endl;
std::cout << mesh.polygons[i].vertices[0] << std::endl;
std::cout << mesh.polygons[i].vertices[1] << std::endl;
std::cout << mesh.polygons[i].vertices[2] << std::endl;
std::cout << std::endl;
}

Boost convex hull with longitude and latitude

I am trying to use boost's convex_hull algorithm with longitude/latitude coordinates.
From here: https://archive.fosdem.org/2017/schedule/event/geo_boost_geography/attachments/slides/1748/export/events/attachments/geo_boost_geography/slides/1748/FOSDEM17_vissarion.pdf
I can see that we can calculate the distance between two points and even find the area using longitude/latitude coordinates (see page 19 and 22 of the PDF document).
Combining that with https://www.boost.org/doc/libs/1_75_0/libs/geometry/doc/html/geometry/reference/algorithms/convex_hull.html
I came up with this: https://wandbox.org/permlink/2AGPUtHPWrlGFMTf, but it does not compile, code here for convenience:
#include <iostream>
#include <boost/geometry.hpp>
#include <boost/geometry/geometries/polygon.hpp>
#include <boost/geometry/geometries/adapted/boost_tuple.hpp>
namespace bg = boost::geometry;
int main()
{
typedef bg::model::point<double, 2, bg::cs::geographic<bg::degree>> point;
typedef boost::geometry::model::polygon<point> polygon;
polygon poly;
bg::read_wkt(" POLYGON ((4.346693 50.858306, 4.367945 50.852455, 4.366227 50.840809, 4.344961 50.833264, 4.338074 50.848677,4.346693 50.858306))",
poly );
polygon hull;
boost::geometry::convex_hull(poly, hull);
using boost::geometry::dsv;
std::cout << "polygon: " << dsv(poly) << std::endl << "hull: " << dsv(hull) << std::endl;
}
Any help is much appreciated.
Yeah, though you may be right that a strategy /can/ be made, that is not implemented.
A little side-by-side tester clearly shows that the strategy is not implemented for the geographical coordinate system:
template <typename cs> void test() {
using point = bg::model::point<double, 2, cs>;
using polygon = bg::model::polygon<point>;
polygon poly;
bg::read_wkt("POLYGON((4.346693 50.858306, 4.367945 50.852455, 4.366227 "
"50.840809, 4.344961 50.833264, 4.338074 50.848677,4.346693 "
"50.858306))",
poly);
std::cout << std::fixed;
std::cout << "Polygon: " << bg::dsv(poly) << std::endl;
std::cout << "Perimeter: " << bg::perimeter(poly) << std::endl;
std::cout << "Area: " << bg::area(poly) << std::endl;
using Strategy = typename bg::strategy_convex_hull<polygon, point>::type;
std::cout << "Strategy " << boost::core::demangle(typeid(Strategy).name()) << "\n";
if constexpr (not std::is_same_v<Strategy, bg::strategy::not_implemented>) {
polygon hull;
bg::convex_hull(poly, hull);
std::cout << "Hull: " << bg::dsv(hull) << std::endl;
}
}
It also demonstrates that some other strategies are implemented (e.g. distance).
See it Live On Coliru
int main() {
std::cout << "Cartesian:\n";
std::cout << "----------\n";
test<bg::cs::cartesian>();
std::cout << "\nGeographic:\n";
std::cout << "-----------\n";
test<bg::cs::geographic<bg::degree>>();
}
Simplifying the the typenames in the output:
Cartesian:
----------
Polygon: (((4.346693, 50.858306), (4.367945, 50.852455), (4.366227, 50.840809), (4.344961, 50.833264), (4.338074, 50.848677), (4.346693, 50.858306)))
Perimeter: 0.086184
Area: 0.000488
Strategy bg::strategy::convex_hull::graham_andrew<polygon, point>
Hull: (((4.338074, 50.848677), (4.346693, 50.858306), (4.367945, 50.852455), (4.366227, 50.840809), (4.344961, 50.833264), (4.338074, 50.848677)))
Geographic:
-----------
Polygon: (((4.346693, 50.858306), (4.367945, 50.852455), (4.366227, 50.840809), (4.344961, 50.833264), (4.338074, 50.848677), (4.346693, 50.858306)))
Perimeter: 7663.398262
Area: 3848183.734567
Strategy bg::strategy::not_implemented
A look at the documented strategies suggests that graham_andrew is in fact the only one available.
You should probably find out what tweaks are required to get things to work. it is technically possible to force convex_hull to use the Graham/Andrew strategy, but that seems ill-advised as the trait implies that the strategy is specifically deselected based on the coordinate system:
/*!
\brief Traits class binding a convex hull calculation strategy to a coordinate system
\ingroup convex_hull
\tparam Tag tag of coordinate system
\tparam Geometry the geometry type (hull operates internally per hull over geometry)
\tparam Point point-type of output points
*/
template
<
typename Geometry1,
typename Point,
typename CsTag = typename cs_tag<Point>::type
>
struct strategy_convex_hull
{
typedef strategy::not_implemented type;
};
Digging into the implementation of the strategy here's a hopeful hint:
// TODO: User-defiend CS-specific side strategy
typename strategy::side::services::default_strategy<cs_tag>::type side;
Perhaps we could be "done" with "just" specializing the Side Strategy for your coordinate systems? And more interestingly: a strategy::side::geographic exists. I'm out of my depth understanding the parameters (e.g. what the geodetic solution policy means?), but maybe yourself can take it from there?
I'm convinced that if you know what needs to be done, the helpful devs over at the mailing list will be very willing to guide the technical questions on how to fit it into the library.

Reading out Pixel Values, png++, C++

I am trying to load the pixel rgb/ga information of a png image into a matrix, using the library png++, to do some computations with them.
My Code (which does not work at the moment):
#include <iostream>
#include <png++/image.hpp>
#include <png++/rgb_pixel.hpp>
int main(int argc, const char * argv[]) {
const std::string path="img_03.png";
png::image< png::basic_rgb_pixel <unsigned char> > pic(path);
pixel=pic.get_pixel(0, 0);
pixelp = &pixel;
std::cout << "value=" << pic[10][10].red << std::endl; //output: '?'
std::cout << "value=" << pixel.red << std::endl; //nothing
std::cout << "pointer=" << pixelp << std::endl; //delivers adress
pic.read(path);
std::cout << "value=" << pic[10][10].red << std::endl; //nothing
pic.write("picOutput.png"); //same picture
return 0;
}
However, none of those methods work to get the rgb values of each pixel.
If there is another way to get rgb/ga information of each pixel, please mention it.
The line pic.write("picOutput.png"); delivers the same png i loaded in the line pic.read(path). This is a personal exercise for me to get more used to C++, criticise my code as much as you can.
Thanks!
Here comes the solution:
change line:
std::cout << "value=" << pic[10][10].red << std::endl; //nothing
with:
std::cout << "value=" << (int) pic[10][10].red << std::endl; //nothing
because std::cout can't output types of unsigned char.
Thanks to Alex!
For in-depth explanation, look here:
cout not printing unsigned char

This predefined function slowing down my program's performance

I am working on a PCL (Point Cloud Library) project. One part of it requires me to clip point clouds, for which I need to know the minimum and maximum coordinates of given point cloud.
PCL provides a predefined function called getminmax3d(). I tried and It works well, The only problem is, It takes a lot of time when I input a large point cloud file. I made my own definition of getminmax3d() and it takes lesser time. I am not understanding why these two behave like this.
I tried with 5 point cloud data files. In all cases, program that uses predefined function takes long time as compare to the program for which I defined the definition.
Here is the code:
First implementation - It uses predefined function getminmax3d()
#include <iostream>
#include <pcl/io/pcd_io.h>
#include <pcl/point_types.h>
#include <pcl/common/common.h>
int main (int, char**)
{
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud;
cloud = pcl::PointCloud<pcl::PointXYZ>::Ptr (new pcl::PointCloud<pcl::PointXYZ>);
pcl::io::loadPCDFile<pcl::PointXYZ> ("your_pcd_file.pcd", *cloud);
pcl::PointXYZ minPt, maxPt;
pcl::getMinMax3D (*cloud, minPt, maxPt);
std::cout << "Max x: " << maxPt.x << std::endl;
std::cout << "Max y: " << maxPt.y << std::endl;
std::cout << "Max z: " << maxPt.z << std::endl;
std::cout << "Min x: " << minPt.x << std::endl;
std::cout << "Min y: " << minPt.y << std::endl;
std::cout << "Min z: " << minPt.z << std::endl;
return (0);
}
Second implementation - This source code uses a user-defined function definition to replace functionality of getminmax3d()
#include <iostream>
#include <pcl/io/pcd_io.h>
#include <pcl/point_types.h>
#include <pcl/common/time.h>
int main (int argc, char** argv)
{
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud (new pcl::PointCloud<pcl::PointXYZ>);
if (pcl::io::loadPCDFile<pcl::PointXYZ> ("rhino.pcd", *cloud) == -1) //* load the file
{
PCL_ERROR ("Couldn't read file rhino.pcd \n");
return (-1);
}
float min_x = cloud->points[0].x, min_y = cloud->points[0].y, min_z = cloud->points[0].z, max_x = cloud->points[0].x, max_y = cloud->points[0].y, max_z = cloud->points[0].z;
pcl::StopWatch watch;
for (size_t i = 1; i < cloud->points.size (); ++i){
if(cloud->points[i].x <= min_x )
min_x = cloud->points[i].x;
else if(cloud->points[i].y <= min_y )
min_y = cloud->points[i].y;
else if(cloud->points[i].z <= min_z )
min_z = cloud->points[i].z;
else if(cloud->points[i].x >= max_x )
max_x = cloud->points[i].x;
else if(cloud->points[i].y >= max_y )
max_y = cloud->points[i].y;
else if(cloud->points[i].z >= max_z )
max_z = cloud->points[i].z;
}
pcl::console::print_highlight ("Time taken: %f\n", watch.getTimeSeconds());
std::cout << "Min x: " << min_x <<"\t";
std::cout << "Max x: " << max_x << std::endl;
std::cout << "Min y: " << min_y <<"\t";
std::cout << "Max y: " << max_y << std::endl;
std::cout << "Min z: " << min_z <<"\t";
std::cout << "Max z: " << max_z << std::endl;
return (0);
}
I tried both programs on following 5 point cloud files.
Result obtained:
ttf : Time taken factor
ttf = 15 means user definition is about 15 times faster than predefined functions. ttf value is measured by taking average of 10 trials for both implementations.
PCD file Filetype File size ttf
Rhino.pcd XYZ 2.57 MB 15.260
Bun_zipper XYZCI 1.75 MB 17.422
Armadillo XYZ 5.26 MB 15.847
Dragon_vrip XYZ 14.7 MB 17.013
Happy_vrip XYZ 18.0 MB 14.981
I am wondering why predefined function is taking more time? I want to reduce my program source code lines. I've always believed that using standard header files and their function gives you best performance, But in this case it seems to fail.
This is where you can find standard definition.
Would anyone please help me to find out why second implementation takes less times(approx 15 times), even the standard definition of getminmax3d() is similar to mine.
pcl::getMinMax3D has a very inefficient implementation. To search for the minimum and max point it does the following:
Eigen::Array4f min_p, max_p;
min_p.setConstant (FLT_MAX);
max_p.setConstant (-FLT_MAX);
for (size_t i = 0; i < cloud.points.size (); ++i)
{
// ... (check the validity of the point if it is not a dense cloud)
pcl::Array4fMapConst pt = cloud.points[i].getArray4fMap ();
min_p = min_p.min (pt);
max_p = max_p.max (pt);
}
And if you check for the getArray4fMap() function:
typedef Eigen::Map<Eigen::Array4f, Eigen::Aligned> Array4fMap;
inline pcl::Array4fMap getArray4fMap() const {
return (pcl::Array4fMap(data));
}
For each point in the cloud it is constructing an Eigen::Map and then comparing it against the current minimum and maximum points. This is VERY inefficient.
The predefined function pcl::getMinMax3D is able to be faster with optimization flags set and in Release. Since if SSE intrinsics are used by Eigen, then the operations happen on 4 aligned bytes.
More information at
https://gitter.im/PointCloudLibrary/pcl?at=5e3899d06f9d3d34981c0687

Declaring an object variable makes all the previous instructions slower

I ran into a fastidious problem and I have no idea of what is causing it. I hope you can help me find a solution.
Framework: I implemented a sparse_matrix class using the CSR representation and I used this object as the basis for a recommendation system. The class is defined as follows:
class sparse_matrix_csr
{
public:
sparse_matrix_csr();
sparse_matrix_csr(const std::vector<int> &row_indices);
sparse_matrix_csr(const std::vector<int> &row_indices, const size_t nb_columns);
// other member functions omitted
private:
std::vector<int> _row_indices;
std::vector<int> _row_start_indices;
std::vector<int> _column_indices;
std::vector<double> _values;
bool _column_sorted_by_index;
};
The _row_indices vector contains the indices of the rows of the matrix. _row_start_indices contains the index of the first element of a given row in the _column_indices, which contains the column indices, and _values vector, which contains the elements of the matrix. In particular, the constructor sparse_matrix_csr(const std::vector<int> &row_indices, const size_t nb_columns) is implemented as follows:
sparse_matrix_csr::sparse_matrix_csr(const std::vector<int> &row_indices,
const size_t nb_columns):
_row_indices(row_indices),
_row_start_indices(row_indices.size()),
_column_indices(row_indices.size() * nb_columns, 0),
_values(row_indices.size() * nb_columns, 0.0),
_column_sorted_by_index(true)
{
for (size_t i = 0; i < _row_start_indices.size(); ++i)
_row_start_indices[i] = i * nb_columns;
}
This constructor takes in the indices of the rows of the sparse matrix and the number of elements that will be contained in each row. In fact, in the application I am considering, I have same matrices that are sparse only wrt the rows.
Problem: The algorithm is structured as follows
\\ Instruction block 1
{
\\ Do something
}
sparse_matrix_csr mat(list_indices, nb_columns);
\\ Instruction block 2
{
\\ Do something
}
If I run the first block of instructions alone (commenting out all that follows), my algorithm runs smoothly. However, if I uncomment the second part of the algorithm, the first part of the algorithm slows down a lot. I have been able to identify the critical line for this slow-down in the declaration of mat. However, I cannot explain this retro-action on the first part of the algorithm. The full algorithm is reported at the end of my question.
My considerations: I have never observed such a retro-action and therefore I am a little confused. One possibility that I have considered is that I have a problem in memory management which causes the slow-down of the previous part of the algorithm. I am currently working on some matrices that have around 2e6 elements, i.e. around 4e6 values stored in 2 vectors. I read on another post on stack overflow that the data contained in a std::vector are allocated in the heap. Is that always true? even if I initialize a vector with a given size as I do in the sparse matrix constructors above?
If you need some clarification, do not hesitate!
Thanks in advance,
Pierpaolo
Full algorithm:
void collaborative_filtering_mpi (std::string ratings_file, std::string targets_file,
std::string output_file, int k_neighbors, double shrinkage_factor,
bool output_debug_data)
{
std::cout << "COLLABORATIVE FILTERING ALGORITHM - k = " << k_neighbors
<< " , d = " << shrinkage_factor << std::endl;
stopwatch sw_total;
sw_total.start();
//-------------------------------------
// STEP 1: Read data from input files
//-------------------------------------
std::cout << "1) Read input files: ";
stopwatch sw;
sw.start();
//-------------------------------
// 1.1) Read user rating matrix
//-------------------------------
// Initialization
sparse_matrix_csr user_item_rating_matrix(ratings_file, true, true, true); // file is sorted, skip header
//-------------------------------------------------
// 1.2) Read targets (user, item) to be predicted
//-------------------------------------------------
// Initialization
sparse_matrix_csr targets(targets_file, true, false, false); // do not read ratings column
sw.stop();
double time_input = sw.get_duration();
std::cout << time_input << std::endl;
//-----------------------------------------
// STEP 2: Pre-computations
//-----------------------------------------
std::cout << "2) Pre-computations: ";
sw.start();
//-----------------------------------------------------------------------
// 2.1) Sort user_item_rating_matrixand compute relevant sizes of the problem
//-----------------------------------------------------------------------
// Sort: if file is sorted, sort should do nothing.
user_item_rating_matrix.sort_columns_by_index();
// Compute list of users and list of items
std::vector<int> list_users;
user_item_rating_matrix.rows(list_users);
std::vector<int> list_items;
user_item_rating_matrix.columns(list_items);
// [DEBUG]: print user_rating_matrix
if (output_debug_data)
{
std::ofstream ofs_debug("data_debug/debug_user_rating_matrix.txt");
ofs_debug << user_item_rating_matrix;
ofs_debug.close();
}
//-------------------------------------------------------------
// 2.2) Compute inverse user rating matrix (indexed by column)
//-------------------------------------------------------------
// Initialize item_user_rating_matrix: it is a sparse matrix with
// items on the rows, users on the columns and rating as values.
// This variable will be helpful when computing similarities.
sparse_matrix_csr item_user_rating_matrix;
// Compute item_user_rating<_matrix by transposing the user_rating_matrix
transpose (user_item_rating_matrix, item_user_rating_matrix);
// [DEBUG]: print item_user_matrix_on_file
if (output_debug_data)
{
std::ofstream ofs_debug("data_debug/debug_item_user_matrix.txt");
ofs_debug << item_user_rating_matrix;
ofs_debug.close();
}
//-----------------------------------------------
// 2.3) sort targets and compute relevant sizes
//-----------------------------------------------
// Compute list of target items
std::vector<int> list_target_items;
targets.columns(list_target_items);
// [DEBUG]: print targets
if (output_debug_data)
{
std::ofstream ofs_debug("data_debug/debug_targets.txt");
ofs_debug << targets;
ofs_debug.close();
}
//--------------------------------------------------------------
// 2.4) Compute difference between list_items and list_targets
//--------------------------------------------------------------
std::vector<int> list_non_target_items;
compute_difference_vector (list_items, list_target_items, list_non_target_items);
// [DEBUG]
if (output_debug_data)
{
std::ofstream ofs_debug("data_debug/debug_difference_vector.txt");
ofs_debug << "list_items - size: " << list_items.size() << std::endl;
for (std::vector<int>::const_iterator iter = list_items.begin(); iter != list_items.end(); ++iter)
ofs_debug << (*iter) << ",";
ofs_debug << std::endl << std::endl;
ofs_debug << "list_target_items - size: " << list_target_items.size() << std::endl;
for (std::vector<int>::const_iterator iter = list_target_items.begin(); iter != list_target_items.end(); ++iter)
ofs_debug << (*iter) << ",";
ofs_debug << std::endl << std::endl;
ofs_debug << "list_non_target_items - size: " << list_non_target_items.size() << std::endl;
for (std::vector<int>::const_iterator iter = list_non_target_items.begin(); iter != list_non_target_items.end(); ++iter)
ofs_debug << (*iter) << ",";
ofs_debug << std::endl << std::endl;
ofs_debug.close();
}
//--------------------------------------------
// 2.5) Compute average rating for each user
//--------------------------------------------
dictionary<int, double> average_rating_vector;
compute_average_rating(user_item_rating_matrix, average_rating_vector);
if (output_debug_data)
{
std::ofstream ofs_debug("data_debug/debug_average_rating_vector.txt");
for (dictionary<int, double>::const_iterator iter = average_rating_vector.begin();
iter != average_rating_vector.end(); ++iter)
ofs_debug << (*iter).get_key() << ": " << (*iter).get_value() << std::endl;
ofs_debug.close();
}
sw.stop();
std::cout << sw.get_duration() << std::endl;
//-------------------------------------
// STEP 3: Similarity matrix
//-------------------------------------
std::cout << "3) Compute similarity matrix: ";
sw.start();
// Initialize similarity_matrix with target items on the rows.
sparse_matrix_csr similarity_matrix(list_target_items,
list_target_items.size() + list_non_target_items.size());
// compute similarity matrix
compute_similarity_matrix_mpi(similarity_matrix,
item_user_rating_matrix,
average_rating_vector,
list_target_items,
list_non_target_items,
shrinkage_factor);
// [DEBUG]: print similarity matrix sorted by similarity
if (output_debug_data)
{
std::ofstream ofs("data_debug/debug_similarity_matrix.txt");
ofs << similarity_matrix;
ofs.close();
}
sw.stop();
std::cout << sw.get_duration() << std::endl;
//---------------------------------------------------------------
// STEP 4: Find top-K similar elements with positive similarity
//---------------------------------------------------------------
std::cout << "4) Find top-K similar elements:" << std::endl;
sw.start();
if (k_neighbors > 0)
{
//---------------------------------------------------
// 4.1) Sort similarity matrix by rating (row-wise)
//---------------------------------------------------
std::cout << " .... Sort similarity matrix by rating: ";
sw.start();
// Sort all the rows of the similarity_matrix by similarity.
// If two items have the same rating, sort them in descending order of item.
//similarity_matrix.sort_columns_by_value();
similarity_matrix.sort_columns_by_value();
// [DEBUG]: print similarity matrix sorted by similarity
if (output_debug_data)
{
std::ofstream ofs ("data_debug/debug_similarity_matrix_sorted_by_rating.txt");
ofs << similarity_matrix;
ofs.close();
}
sw.stop();
std::cout << sw.get_duration() << std::endl;
//--------------------------------------------------------
// 4.2) Cut the useless columns of the similarity matrix
//--------------------------------------------------------
std::cout << " .... Reduce similarity matrix: ";
sw.start();
sparse_matrix_csr similarity_matrix_reduced(list_target_items,
k_neighbors);
reduce_similarity_matrix_mpi (similarity_matrix,
similarity_matrix_reduced,
k_neighbors);
// [DEBUG]: print similarity matrix sorted by similarity
if (output_debug_data)
{
std::ofstream ofs ("data_debug/debug_similarity_matrix_reduced.txt");
ofs << similarity_matrix_reduced;
ofs.close();
}
sw.stop();
std::cout << sw.get_duration() << std::endl;
//---------------------------------------------------
// 4.3) Sort the reduced similarity matrix by items
//---------------------------------------------------
std::cout << " .... Sort similarity matrix by index: ";
sw.start();
// Sort all the rows of the similarity_matrix by index.
similarity_matrix_reduced.sort_columns_by_index();
// [DEBUG]: print similarity matrix sorted by similarity
if (output_debug_data)
{
std::ofstream ofs ("data_debug/debug_similarity_matrix_sorted.txt");
ofs << similarity_matrix_reduced;
ofs.close();
}
sw.stop();
std::cout << sw.get_duration() << std::endl;
//-----------------------------------------
// STEP 5: Compute predictions for targets
//-----------------------------------------
std::cout << "5) Compute predictions: ";
sw.start();
compute_predicted_ratings_mpi (targets,
user_item_rating_matrix,
similarity_matrix_reduced);
sw.stop();
std::cout << sw.get_duration() << std::endl;
}
else
{
//---------------------------------------------------
// 4.3) Sort the reduced similarity matrix by items
//---------------------------------------------------
std::cout << " .... Sort similarity matrix by index: ";
sw.start();
// Sort all the rows of the similarity_matrix by index.
// similarity_matrix.sort_columns_by_index();
similarity_matrix.sort_columns_by_index();
// [DEBUG]: print similarity matrix sorted by similarity
if (output_debug_data)
{
std::ofstream ofs ("data_debug/debug_similarity_matrix_sorted.txt");
ofs << similarity_matrix;
ofs.close();
}
sw.stop();
std::cout << sw.get_duration() << std::endl;
//-----------------------------------------
// STEP 5: Compute predictions for targets
//-----------------------------------------
std::cout << "5) Compute predictions: ";
sw.start();
compute_predicted_ratings_mpi (targets,
user_item_rating_matrix,
similarity_matrix);
sw.stop();
std::cout << sw.get_duration() << std::endl;
}
//-----------------------------------------------------
// STEP 6: Print the prediction matrix in output file
//-----------------------------------------------------
std::cout << "6) Print predictions on file: ";
sw.start();
std::ofstream ofs_output(output_file);
targets.print(ofs_output);
sw.stop();
double time_output = sw.get_duration();
std::cout << time_output << std::endl;
sw_total.stop();
double time_total = sw_total.get_duration();
std::cout << ">> Total computation time: " << time_total << std::endl;
std::cout << ">> Total computation time - no input/output: " << (time_total - time_input - time_output) << std::flush;
}
UPDATE - 05 February 2015: I don't know what happened, but the code is now running fine on my 64-bit machine. However, it is still very slow on a 32-bit VM that I need to use to run my code. I measured the size of a sparse_matrix object using sizeof and it occupies 52 bit. I think this might be causing a cache problem (credit goes to #Dark Falcon). Do you have any ideas on how I could solve this problem and make my algorithm run efficiently on the 32-bit VM?
UPDATE - 06 February 2015: Well, the problem comes and go. I tried to change the implementation of the sparse_matrix_csr class wrapping the data in a shared_ptr in the following way:
class sparse_matrix_csr
{
public:
\\ public methods omitted
private:
class sparse_matrix_csr_data
{
public:
sparse_matrix_csr_data() {}
sparse_matrix_csr_data(const std::vector<int> &row_indices): _row_indices(row_indices) {}
sparse_matrix_csr_data(const std::vector<int> &row_indices, const size_t nb_columns);
sparse_matrix_csr_data(const std::string file, bool file_is_sorted, bool skip_header, bool read_values);
// data
std::vector<int> _row_indices;
std::vector<int> _row_start_indices;
std::vector<int> _column_indices;
std::vector<double> _values;
bool _column_sorted_by_index;
};
std::shared_ptr<sparse_matrix_csr_data> _data;
};
This modification did not improve things. I am currently having problems both on the 32-bit VM and the 64-bit laptop. I have no idea about what is causing the program to be so slow.