I am trying to find shortest path between two nodes in a dataset. I
implemented dijkstra algorithm and am using it to prove
given two nodes (like: Andrew_Card
and Dick_Cheney) there does not
exist a path between the source and destination. However, I am finding
that my program is getting killed by the operating system.
After debugging I found that the problem could be related to resource
allocation in RAM. As for dijkstra algorithm, if the number of nodes,
n=16,375,503, then the space requirement is
n*n = 16,375,503 * 16,375,503 > 10^{14}.
To run this algorithm in memory we need at least
(10^{14} * 4) / (1024 * 1024 * 1024) = 10^5 GB (approximately equal)
of RAM.
So, it is not possible to find the shortest path using
dijkstra if we intend to keep a large connected graph in-memory.
Please correct me if I am wrong as I am stuck on this since a long time? Or if there could be some other possible reason which I should check, then please point me to it too.
I implemented the program in C++
No. of edges=25,908,132
If the number of edges is relatively low(so that all edges can fit into main memory), you can just store the graph using adjacency list. It requires O(V + E) memory, instead of O(V^2). Moreover, you can use Dijkstra's algorithm with a priority queue. It works well for sparse graphs(it has O(E log V) time complexity). This approach should work fine for a graph with about 2 * 10^7 vertices and edges(a good implementation can easily fit into main memory and run for no more than several minutes).
If you need JUST the distance between two nodes, use something like A*.
But if you're doing all points shortest paths, then you're definitely stuck with O(n^2) space. You're finding O(n^2) answers - so you can't really do any better than having to store all of them.
In terms of making sure that your program is indeed running out of memory, wrap your callsite in a try-catch block and see if you are getting a std::bad_alloc exception. Until you see the exception you are catching, don't make assumptions about which part of your program is failing
In terms of finding the shortest route between two nodes, you probably should up more literature to find what is the most suitable algorithm for your use case.
A*: http://en.wikipedia.org/wiki/A*_search_algorithm
Contraction hierarchy: http://algo2.iti.kit.edu/schultes/hwy/contract.pdf
You should find a way to reduce number of nodes. Your number of nodes are high. You could use Voronoi Diagram to reduce number of nodes.
In improvement could be to store vertexes in the priority queue just once. And update the priority queue, instead of adding the same vertex to the priority queue again and again.
Related
I'm working on implementing a ModelClass for any 3D model in my DirectX 11/12 pipeline.
My specific problem lies within calculating the min and max for the BoundingBox structure I wish to use as a member of the ModelClass.
I have two approaches to calculating them.
Approach 1.
When each vertex is being read from file, store a current minx,y,z and maxx,y,z and check each vertex as it is loaded in against the current min/max x,y,z.
Approach 2.
After all the vertices have been loaded, sort them by x, then y, then z, finding the lowest and highest value at each point.
Which Approach would you recommend and why?
Approach 1
Time complexity is in O(n) and memory complexity is O(1).
It is simple to implement.
Approach 2
Time complexity is O(nLogn) memory complexity is potentially at least linear (if you make a copy of the arrays or if you use merge sort) or O(1) if you use an in place sorting algorithm like quicksort.
This has to be done 3 times one for each dimension.
All in all Approach 1 is best in all scenarios I can think of.
Sorting generally is not a cheap operation especially as your models are getting larger. Therefore it to me like Approach 1 is more efficient but if unsure I suggest measuring it see which one takes longer.
If you are using a library like Asspimp I believe the library takes care of bounding boxes but this might not be an option if you create the pipeline as a learning opportunity.
I am using STL priority_queue as an data structure in my graph application. You can safely assume it like a advance version of Prim's spanning tree algorithm.
With in the Algorithm I want to find a node in the priority queue (not just a minimum node) efficiently.[ this is needed because cost of node might get changed and need to be fixed in priority_queue]
All i have to do is augment the priority_queue and index it based on my node key's also. I don't find any way this can be done in STL. Can anyone have better idea how to do it in STL?
The std::priority_queue<T> doesn't support efficient look-up of nodes: it uses a d-ary heap, typically with d == 2. This representation doesn't keep nodes put. If you really want to use a std::priority_queue<T> with Prim's algorithm, the only way is to just add nodes with their current shortest distance and possibly add each node multiple times. This turns the size of the into O(E) instead of O(N), though, i.e., for graphs with many edges it will result in a much higher complexity.
You can use something like std::map<...> but that really suffers from pretty much the same problem: you can either locate the next node to extract efficiently or you can locate the nodes to update efficiently.
The "proper" approach is to use a node-based priority queue, e.g., a Fibanocci-heap: Since the nodes stay put, you can get a handle from the heap when inserting a node and efficiently update the distance of a node through the handle. Access to the closest node is efficient using the few top nodes in the heap's set of trees. The overall performance of basic heap operations (push(), top(), and pop()) are slower for Fibonacci heaps than for d-ary heaps but the efficient update of individual nodes makes their use worthwhile. I seem to recall that Prim's algorithm actually required Fibonacci-heaps anyway to achieve the tight complexity bound.
I know that there is an implementation of Fibonacci-heaps at Boost. An efficient implementation of Fibonacci heaps isn't entirely trivial but they are more efficient than just being of theoretical interest.
For the past few days I've tried to implement this algorithm. This far I've managed to make a dynamic 2d array and insert the distances between nodes, a function to remove a path between nodes and a function that tells me if there is a path between two nodes.
Now I would like to implement a function that returns the shortest path from node A to node B. I know how dijkstras algorithm works and I've read the pseudo code on wiki without being able to write any code my self. I'm really stuck here.
I've been thinking about how the code should look like and what should happen thats why I've made that function that tells me if theres a path between two nodes. Do I need any more help functions which would make implementing of dijkstras easier?
For now I have only 3 nodes but the code I would like to write needs to work in general for n nodes.
Any kind of help is appreciated.
You are probably thinking to much.
You need 2 things. A clean graph structure you understand. A good description of the algorithm you understand.
If you have both. Just start writing some code. Helpers needed will become obvious on the way.
-- edit --
You will probably need some of the following datastructures
std::vector
std::list
std::priority_queue
I found several codes for this algorithm, but maybe it is better the simplest one in order to undertand it better, so you can check the differences between yours and this one, and complete yours. It is always better to program your way.
Have a look at this one and see if it helps.
http://vinodcse.wordpress.com/2006/05/19/code-for-dijkstras-algorithm-in-c-2/
Good luck.
Edit: Code deleted, and I'm going to give hints:
Store graph as list of adjacency lists of each vertex. (something like this vector < vector < pair<int,int> > > g (n);)
Use some data-structure to keep track what is the vertex with minimal distance in current state. (maybe set, or priority_queue to have O(m log(n)) complexity)
Each time take high_priority vertex (vertex with minimal current distance), delete it from your data_structure and update distances of adjacent to deleted one vertexes.
Note: If you want to get minimal path as well, then keep some vector<int> previous and each time when updating distance of vertex (say v) set previous[v] = index of vertex from where you came here. Your path is last, prev[last], prev[prev[last]],...,first in reversed order.
In an optimization problem I keep in a queue a lot of candidate solutions which I examine according to their priority.
Each time I handle one candidate, it is removed form the queue but it produces several new candidates making the number of cadidates to grow exponentially. To handle this I assign a relevancy to each candidate, whenever a candidate is added to the queue, if there is no more space avaliable, I replace (if appropiate) the least relevant candidate currently in the queue with the new one.
In order to do this efficiently I keep a large (fixed size) array with the candidates and two linked indirect binary heaps: one handles the candidates in decreasing priority order, and the other in ascending relevancy.
This is efficient enough for my purposes and the supplementary space needed is about 4 ints/candidate which is also reasonable. However it is complicated to code, and it doesn't seem optimal.
My question is if you know of a more adequate data structure or of a more natural way to perform this task without losing efficiency.
Here's an efficient solution that doesn't change the time or space complexity over a normal heap:
In a min-heap, every node is less than both its children. In a max-heap, every node is greater than its children. Let's alternate between a min and max property for each level making it: every odd row is less than its children and its grandchildren, and the inverse for even rows. Then finding the smallest node is the same as usual, and finding the largest node requires that we look at the children of the root and take the largest. Bubbling nodes (for insertion) becomes a bit tricker, but it's still the same O(logN) complexity.
Keeping track of capacity and popping the smallest (least relevant) node is the easy part.
EDIT: This appears to be a standard min-max heap! See here for a description. There's a C implementation: header, source and example. Here's an example graph:
(source: chonbuk.ac.kr)
"Optimal" is hard to judge (near impossible) without profiling.
Sometimes a 'dumb' algorithm can be the fastest because intel CPUs are incredibly fast at dumb array scans on contiguous blocks of memory especially if the loop and the data can fit on-chip. By contrast, jumping around following pointers in a larger block of memory that doesn't fit on-chip can be tens or hundreds or times slower.
You may also have the issues when you try to parallelize your code if the 'clever' data structure introduces locking thus preventing multiple threads from progressing simultaneously.
I'd recommend profiling both your current, the min-max approach and a simple array scan (no linked lists = less memory) to see which performs best. Odd as it may seem, I have seen 'clever' algorithms with linked lists beaten by simple array scans in practice often because the simpler approach uses less memory, has a tighter loop and benefits more from CPU optimizations. You also potentially avoid memory allocations and garbage collection issues with a fixed size array holding the candidates.
One option you might want to consider whatever the solution is to prune less frequently and remove more elements each time. For example, removing 100 elements on each prune operation means you only need to prune 100th of the time. That may allow a more asymmetric approach to adding and removing elements.
But overall, just bear in mind that the computer-science approach to optimization isn't always the practical approach to the highest performance on today and tomorrow's hardware.
If you use skip-lists instead of heaps you'll have O(1) time for dequeuing elements while still doing searches in O(logn).
On the other hand a skip list is harder to implement and uses more space than a binary heap.
I did recently attach the 3rd version of Dijkstra algorithm for shortest path of single source into my project.
I realize that there are many different implementations which vary strongly in performance and also do vary in the quality of result in large graphs. With my data set (> 100.000 vertices) the runtime varies from 20 minutes to a few seconds. Th shortest paths also vary by 1-2%.
Which is the best implementation you know?
EDIT:
My Data is a hydraulic network, with 1 to 5 vertices per node. Its comparable to a street map. I made some modifications to a already accelerated algorithm (using a sorted list for all remaining nodes) and now find to the same results in a fraction of time. I have searched for such a thing quite a while. I wonder if such a implementation already exists.
I can not explain the slight differences in results. I know that Dijkstra is not heuristic, but all the implementations seem to be correct. The faster solutions have the results with shorter paths. I use double precision math exclusively.
EDIT 2:
I found out that the differences in the found path are indeed my fault. I had inserted special handling for some vertices (only valid in one direction) and forgot about that in the other implementation.
BUT im still more than surprised that Dijkstra can be accelerated dramatically by the following change:
In general a Dijkstra algorithm contains a loop like:
MyListType toDoList; // List sorted by smallest distance
InsertAllNodes(toDoList);
while(! toDoList.empty())
{
MyNodeType *node = *toDoList.first();
toDoList.erase(toDoList.first());
...
}
If you change this a little bit, it works the same, but performs better:
MyListType toDoList; // List sorted by smallest distance
toDoList.insert(startNode);
while(! toDoList.empty())
{
MyNodeType *node = *toDoList.first();
toDoList.erase(toDoList.first());
for(MyNeigborType *x = node.Neigbors; x != NULL; x++)
{
...
toDoList.insert(x->Node);
}
}
It seems, that this modification reduces the runtime by a order not of magnitude, but a order of exponent. It reduced my runtime form 30 Seconds to less than 2. I can not find this modification in any literature. It's also very clear that the reason lies in the sorted list. insert/erase performs much worse with 100.000 elements that with a hand full of.
ANSWER:
After a lot of googling i found it myself. The answer is clearly:
boost graph lib. Amazing - i had not found this for quite a while. If you think, that there is no performance variation between Dijkstra implementations, see wikipedia.
The best implementations known for road networks (>1 million nodes) have query times expressed in microseconds. See for more details the 9th DIMACS Implementation Challenge(2006). Note that these are not simply Dijkstra, of course, as the whole point was to get results faster.
May be I am not answering your question. My point is why to use Dijkstra when there are pretty much more efficient algorithms for your problem. If your graph fullfills the triangular property (it is an euclidian graph)
|ab| +|bc| > |ac|
(the distance from node a to node b plus distance from node b to node c is bigger than the distance from node a to node c) then you can apply the A* algorithm.
This algorithm is pretty efficient. Otherwise consider using heuristics.
The implementation is not the major issue. The algorithm to be used does matter.
Two points I'd like to make:
1) Dijkstra vs A*
Dijkstra's algorithm is a dynamic programming algorithm, not an heuristic. A* is an heuristic because it also uses an heuristic function (lets say h(x) ) to "estimate" how close a point x is getting to the end point. This information is exploited in subsequent decisions of which nodes to explore next.
For cases such as an Euclidean graph, then A* works well because the heuristic function is easy to define (one can simply use the Euclidean distance, for example). However, for non Euclidean graphs it may be harder to define the heuristic function, and a wrong definition can lead to a non-optimal path.
Therefore, dijkstra has the advantage over A* which is that it works for any general graph (with the exception of A* being faster in some cases). It could well be that certain implementations use these algorithms interchangeably, resulting in different results.
2) The dijkstra algorithm (and others such as A*) use a priority queue to obtain the next node to explore. A good implementation may use a heap instead of a queue, and an even better one may use a fibonacci heap. This could explain the different run times.
The last time I checked, Dijkstra's Algorithm returns an optimal solution.
All "true" implementations of Dijkstra's should return the same result each time.
Similarly, asymptotic analysis shows us that minor optimisations to particular implementations are not going to affect performance significantly as the input size increases.
It's going to depend on a lot of things. How much do you know about your input data? Is it dense, or sparse? That will change which versions of the algorithm are the fastest.
If it's dense, just use a matrix. If its sparse, you might want to look at more efficient data structures for finding the next closest vertex. If you have more information about your data set than just the graph connectivity, then see if a different algorithm would work better like A*.
Problem is, there isn't "one fastest" version of the algorithm.