I'm working on implementing a divide-and-conquer algorithm that finds the two points closest to eachother and the distance between them. My final solution finds the correct answer (compared to using brute force), but about 1/3 of the time would return a segmentation fault error. I've been working on fixing this issue for a couple of days now, adding print statements here and there, but couldn't find the issue. I would appreciate it if someone took a look at my code.
Your "divide" loop assumes that X and Y have the same number of elements. If Y has fewer than X you'll run into undefined behavior, which could be funny results or a crash.
Try changing your access method for the vectors. Vector subscript-operator does seem to have an eratic behaviour in some compilers/platforms (at least from my perspective). I think the problem comes from it being a linked list, resulting in being non-contiguous in memory, but that's just a guess. (EDIT: std::vector are contiguous in memory, the problem is something else) Try using iterators instead (easy-peasy with auto keyword), such as:
for (auto iter = Yprime.begin(); iter != Yprime.end(); iter++) {
// Your code here
}
Hope this works for your case!
Related
Given a vector of (random) numbers, I'm looking for the magnitude difference between the smallest element and the largest element. The STL has the minmax_element function for this purpose. Depending on the result, I would like to perform some action in my code, i.e., if the difference is large enough, I would like to do stuff.
Right now, my code looks as follow:
auto res = std::minmax_element(vec.begin(), vec.end());
if (std::abs(*res.second - *res.first) > threshold)
// do stuff
This algorithm works in principle. However, in most of my cases the if condition will be fulfilled. I would even bet in most of my cases I can simply compare the first and the second element in my vector in order to check the if condition and do stuff.
Having said this, it seems a bit odd that I run through the whole vector all the time, although I mostly don't have to. Is there a STL-algorithm, which takes the early exit into, or has someone an appropriate STL algorithm solution in mind. I could easily write a hand-crafted for-loop with an early exit in order to do what I want, but maybe there are better options.
I have a list of points with x,y coordinates:
List_coord=[(462, 435), (491, 953), (617, 285),(657, 378)]
This list lenght (4 element here) can be very large from few hundred up to 35000 elements.
I want to remove too close points by threshold in this list.
note:Points are never at the exact same position.
My current code for that:
while iteration<5:
for pt in List_coord:
for PT in List_coord:
if (abs(pt[0]-PT[0])+abs(pt[1]-PT[1]))!=0 and abs(pt[0]-PT[0])<threshold and abs(pt[1]-PT[1])<threshold:
List_coord.remove(PT)
iteration=iteration+1
Explication of my terrible code :) :
I check if the very distance is 0 then it means that i am comparing
the same point
then i check the distance in x and in y..
Iteration:
I need few iterations to avoid missing one remove because the list change inside the loop itself...
This code is working but it is a very low process!
I am sure there is another method much easier but i wasn't able to find even if some allready answered questions are close to mine..
note:I would like to avoid using extra library for that code if it is possible
Python will be a bit slow at this ;-)
The solution you will probably want is called quad-trees, but I'll mention a simpler approach first, in case it's preferable.
The usual approach is to group the points so that you can easily reject points that are clearly far away from each other.
One approach might be to sort the list twice, once by x once by y. You can prove that if two points are too-close, they must be close in one dimension or the other. Thus your inner loop can break out early. If it sees a point that is too far away from the outer point in the sorted direction, it can know for a fact that all future points in that list are also too far away. Thus it doesn't have to look any further. Do this in X and Y and you're set!
This approach is going to tend to be dominated by the O(n log n) sort times. However, if all of your points share a single x value, you'll end up doing the same slow O(n^2) iteration that you're doing right now because you never terminate the inner loop early.
The more robust solution is to use quadtrees. Quadtrees are designed to solve the kind of problem you are looking at. The idea is to build a tree such that you can rapidly exclude large numbers of points. I'd recommend this.
If your number of points gets too large, I'd recommend getting a clustering library. Efficient clustering is a very difficult task, and often done in C++ or another fast language.
I have a bit of an issue, I was recently told that for an un-ordered value for input, a bunch of random values, lets say 1 Million of them, that using a set would be more efficient than using a vector, and then sorting said vector with the basic sort algorithm function, but when I used them, and checked them through the time function, in the terminal, and valgrind, it showed that both time complexity, and space usage were faster for the vector, even with the addition of the sort function being called. The person who gave me the advice to use the set is a lot more experienced than me in the C++ language, but I always have to test things out myself prior to taking peoples advice. The test codes follow.
For Set
std::set<int> testSet;
for(int i(0); i<= 1000000; ++i)
testSet.insert(-i);
For Vector
std::vector<int> testVector;
for(int i(0); i<= 1000000; ++i)
testVector.push_back(i * -1);
std::sort(testVector.begin(), testVector.end());
I know that these are not random variables, it wouldn't be fair since set does not allow duplicates, and vector does sothey would be different sizes for this basic function point. Can anyone clarify why the set should be used, sans the point of the no duplicates one.
I did not do any tests with the unordered set either. Not too sure of the differences between the two given points.
This is too vague and ignores/misses out several crucial factors. If your friend said precisely this, then your friend (regardless of his or her experience) was wrong. More likely you are somewhat misinterpreting their words and reading into them a simplified version of matters.
When you want a sorted final product, the sorting is "amortized" when you insert into a set, because you get little bits of sorting action each time. If you will be inserting periodically and many times, then that spreading-out of the workload may be what you want. The total, when added up, may still be more than for a vector (consider the occasional rebalancing and so forth; your vector just needs to be moved to a larger block of memory once in a while), but you've spread it out so as not to noticeably slow down some individual other part of your program.
But if you're just dumping all the elements into a vector and sorting straight away, not only is there less work for the container & algorithm to do but you probably don't mind it taking a noticeable amount of time.
You haven't really stated your use case in any detail so I won't pretend to give specifics here, but the only possible answer to your question as posed is both "it depends" and "the question is fundamentally somewhat meaningless"; you cannot just take two data structures and sorting methodologies, and ask "which is more efficient?" without a use case. You have, however, correctly measured the time and space requirements and if you've done that against your real-world use case then, well, you have your answer don't you?
It is follow-up question for this MIC question. When adding items to the vector of reference wrappers I spend about 80% of time inside ++ operator whatever iterating approach I choose.
The query works as following
VersionView getVersionData(int subdeliveryGroupId, int retargetingId,
const std::wstring &flightName) const {
VersionView versions;
for (auto i = 0; i < 3; ++i) {
for (auto j = 0; j < 3; ++j) {
versions.insert(m_data.get<mvKey>().equal_range(boost::make_tuple(subdeliveryGroupId + i, retargetingId + j,
flightName)));
}
}
return versions;
}
I've tried following ways to fill the reference wrapper
template <typename InputRange> void insert(const InputRange &rng) {
// 1) base::insert(end(), rng.first, rng.second); // 12ms
// 2) std::copy(rng.first, rng.second, std::back_inserter(*this)); // 6ms
/* 3) size_t start = size(); // 12ms
auto tmp = std::reference_wrapper<const
VersionData>(VersionData(0,0,L""));
resize(start + boost::size(rng), tmp);
auto beg = rng.first;
for (;beg != rng.second; ++beg, ++start)
{
this->operator[](start) = std::reference_wrapper<const VersionData>(*beg);
}
*/
std::copy(rng.first, rng.second, std::back_inserter(*this));
}
Whatever I do I pay for operator ++ or the size method which just increments the iterator - meaning I'm still stuck in ++. So the question is if there is a way to iterate result ranges faster. If there is no such a way is it worth to try and go down the implementation of equal_range adding new argument which holds reference to the container of reference_wrapper which will be filled with results instead of creating range?
EDIT 1: sample code
http://coliru.stacked-crooked.com/a/8b82857d302e4a06/
Due to this bug it will not compile on Coliru
EDIT 2: Call tree, with time spent in operator ++
EDIT 3: Some concrete stuff. First of all I didnt started this thread just because the operator++ takes too much time in overall execution time and I dont like it just "because" but at this very moment it is the major bottleneck in our performance tests. Each request usually processed in hundreds of microseconds, request similar to this one (they are somewhat more complex) are processed ~1000-1500 micro and it is still acceptable. The original problem was that once the number of items in datastructure grows to hundreds of thousands the performance deteriorates to something like 20 milliseconds. Now after switching to MIC (which drastically improved the code readability, maintainability and overall elegance) I can reach something like 13 milliseconds per request of which 80%-90% spent in operator++. Now the question if this could be improved somehow or should I look for some tar and feathers for me? :)
The fact that 80% of getVersionData´s execution time is spent in operator++ is not indicative of any performance problem per se --at most, it tells you that equal_range and std::reference_wrapper insertion are faster in comparison. Put another way, when you profile some piece of code you will typically find locations where the most time is spent, but whether this is a problem or not depends on the required overall performance.
#kreuzerkrieg, your sample code does not exercise any kind of insertion into a vector of std::reference_wrappers! Instead, you're projecting the result of equal_range into a boost::any_range, which is expected to be fairly slow at iteration --basically, increment ops resolve to virtual calls.
So, unless I'm seriously missing something here, the sample code performance or lack thereof does not have anything to do with whatever your problem is in real code (assuming VersionView, of which you don't show the code, is not using boost::any_range).
That said, if you can afford replacing your ordered indices with equivalent hashed indices, iteration will probably be faster, but this is is an utter shot in the dark given you're not showing the real stuff.
I think that you're measuring the wrong things entirely. When I scale up from 3x3x11111 to 10x10x111111 (so 111x as many items in the index), it still runs in 290ms.
And populating the stuff takes orders of magnitude more time. Even deallocating the container appears to take more time.
What Doesn't Matter?
I've contributed a version with some trade offs, which mainly show that there's no sense in tweaking things: View On Coliru
there's a switch to avoid the any_range (it doesn't make sense using that if you care for performance)
there's a switch to tweak the flyweight:
#define USE_FLYWEIGHT 0 // 0: none 1: full 2: no tracking 3: no tracking no locking
again, it merely shows you could easily do without, and should consider doing so unless you need the memory optimization for the string (?). If so, consider using the OPTIMIZE_ATOMS approach:
the OPTIMIZE_ATOMS basically does fly weight for wstring there. Since all the strings are repeated here it will be mighty storage efficient (although the implementation is quick and dirty and should be improved). The idea is much better applied here: How to improve performance of boost interval_map lookups
Here's some rudimentary timings:
As you can see, basically nothing actually matters for query/iteration performance
Any Iterators: Doe They Matter?
It might be the culprit on your compiler. On my compile (gcc 4.8.2) it wasn't anything big, but see the disassembly of the accumulate loop without the any iterator:
As you can see from the sections I've highlighted, there doesn't seem to be much fat from the algorithm, the lambda nor from the iterator traversal. Now with the any_iterator the situation is much less clear, and if your compile optimizes less well, I can imagine it failing to inline elementary operations making iteration slow. (Just guessing a little now)
Ok, so the solution I applied is as following:
in addition to the odered_non_unique index (the 'byKey') I've added random_access index. When the data is loaded I rearrange the random index with m_data.get.begin(). Then when the MIC is queried for the data I just do boost::equal_range on the random index with custom predicate which emulates the same logic that was applied in ordering of 'byKey' index. That's it, it gave me fast 'size()' (O(1), as I understand) and fast traversal.
Now I'm ready for your rotten tomatoes :)
EDIT 1:
of course I've changed the any_range from bidirectional traversal tag to the random access one
I want to insert a pair< string, vector<float> > into a map, first it works, but after several loops, it cannot insert any more and throw me a segmentation fault. Can anybody give a possible reason?
Btw: I first read a file and generate the map (about 200,000 elements) and I read another file and update the old map. the error occurs while the updating step.
Can anybody help me with the info I gave above? Thanks a lot
The code is pretty long.....I just erase the previous key and then insert a new one, it seems not complicated.....but drives me crazy....could you guess what happened here?
Thanks A lot for all your answers! And I found it is really a good place for solving problems. Thanks again, I'll try to simplify my codes and add it here today or tomorrow.
Update: I used the code from MSN and it works, thanks a lot that you solved my problem without seeing my code......also many thanks to other kind-hearted people here! However, i can only choose one as the answer.
Are you inserting using the iterator you called erase() on? Or using that iterator in any way? After erase(p) is called, p is invalidated.
Without more information, it's not easy to say, but what are you inserting? Could it simply be that you run out of memory? Although, I do think normal C++ would throw an exception in that case, are you using any custom allocators, malloc, or arrays on the stack which are overrun perhaps?
Perhaps a snippet of code describing what you do could be helpful in determining the cause of your problem.
could you guess what happened here?
You're abusing memory in some way.
There are a lot of ways to do that, in C++!
Rather than guess, and without reading your code, I suggest run the kind of platform-specific debugger which will detect this problem, for example valgrind.
Your alternative is to make the problem smaller: reproduce the problem in only a few lines of code, which you can then post for people to look at.
The type in question is pair<string, vector<float> >. You will be copying that pair on every insert. If either the string or the vector are big then you could be running out of memory.
Edit: to fix running out of memory, you can change how you insert key-value pairs to:
pair<map::iterator, bool> insert_result= map.insert(make_pair(name, vector<float>());
if (insert.second) { insert_result.first->second.swap(vector_read_in); }
That will ensure that you do not copy memory, only move it.
It easily happens if you either modify the keys of the elements already in the data structure or if you have a bad compare function, which misleads the search algorithm.
If you can detect which concrete insert operation causes the seg.fault, then try debugging/logging with what values the compare function is called.
Alternatively, you should print the contents of the map before the erroneous insert, the keys will probably not be in order.
please post some code, you cant expect us to debug your problem on guess work alone.
...but ill give it a stab anyway :) Also what compiler, and system are you doing this on?
If you are reading the data in a loop you may run out of stack space which would cause a seg fault.
Ill edit my answer if you post some code.
Remember that if you are looping through the map, finding stuff to delete, save off the key for later deletion. If you delete while iterating, you run the risk of invalidating the iteration loop.
std::map<string, vector<float> >::iterator iter = my_map.begin();
while (iter != my_map.end()) {
if (somethingBadAboutItem(iter)) {
my_map.erase(iter); // this can mess up my_map iteration above
// and cause bad data access later
}
++iter;
}
instead, try
std::map<string, vector<float> >::iterator iter = my_map.begin();
std::vector<string> keys_to_delete;
while (iter != my_map.end()) {
if (somethingBadAboutItem(iter)) {
keys_to_delete.push_back(iter->first);
}
++iter;
}
for (std::size_t i = 0; i < keys_to_delete.size(); ++i) {
iter = my_map.find(keys_to_delete[i]);
my_map.erase(iter);
}
I am interested if other people have found something more elegant than this, but this is my preferred technique.
Bill: Have you tried something like this:
for(std::map<string, vector<float> >::iterator iter = my_map.begin(); iter != my_map.end();) {
if(somethingBadAboutItem(iter)) {
my_map.erase(iter++);
} else {
++iter;
}
}
The key is to postincrement the iterator when deleting, so it's still valid when incremented but you erase (and hence invalidate) a copy pointing to the previous item.
I'm not sure this technique necessarily works for all STL containers; I can't remember all the invalidation rules offhand (seems unlikely to work for vector for example, but you're normally better off using remove_if for those for non-trivial sizes) but it should be fine for map.