Wikipedia says this about dynamic programming :
In mathematics, computer science, economics, and bioinformatics, dynamic programming is a method for solving a complex problem by breaking it down into a collection of simpler subproblems. It is applicable to problems exhibiting the properties of overlapping subproblems and optimal substructure. When applicable, the method takes far less time than other methods that don't take advantage of the subproblem overlap (like depth-first search).
and also from Introduction to Algorithms (Cormen) , I have learnt that dynamic programming is a method applied to solve repeating computations that have already been computed once. In layman's terms ,
if you're going to compute something again and again , better store it somewhere.
Applying this on Fibonacci I could write its algorithm as follows :
arr[3] = {1,1,1} //first two index for computation , last to keep track
fibbDyn(n){
if(n>=1 || a[2]>n ) return n; // return on base conditions
else {
res = arr[0] + fibbDyn(n-1);
arr[0]=arr[1];
arr[1]=res;
arr[2]+=1; // increment value by 1
return res;
}
}
While I believe this algorithm follows the example of dynamic programming as it reduces the extra computations being done in original recursive fibbonaci version :
fibb(n){
if (n>=1)return n;
else return fibb(n-1) + fibb(n-2);
}
as here due to two separate calls at each recursive step else return fibb(n-1) + fibb(n-2) many computations are repeated.
an iterative solution will probably look like :
int FibonacciIterative(int n)
{
if (n == 0) return 0;
if (n == 1) return 1;
int prevPrev = 0;
int prev = 1;
int result = 0;
for (int i = 2; i <= n; i++)
{
result = prev + prevPrev;
prevPrev = prev;
prev = result;
}
return result;
}
So my question is , will an iterative solution to Fibonacci problem be classified as dynamic programming?
My reasoning for a disagreement is that , an iterative solution dosen't exhibits Overlapping subproblems such as an recursive solution is exhibiting. In an iterative solution , there are no redundant and repetitive computations being made , so it shouldn't be included in dynamic programming.
relevant articles : optimal substructure , overlapping subproblems ,
dynamic programming.
Yes. That's just a special case of Bottom Up dynamic programming. You're allowed to discard table entries that you know you will never use again, in the case of Fibonacci that means you only need to keep 2 entries, and then you can forget it was ever a table and just use two named variables. So, it ends up looking different, almost too simple. But the structure of that algorithm is still DP. The overlapping subproblems that you say aren't there are still there, because you use every result twice (once when it's in prev, again when it's in prevPrev), except in the end. Of course there are no redundant computations made, but then that's the idea of DP - remove redundant computation by reuse.
There is a general "plan of attack" for problems that allow dynamic programming, namely
state the problem recursively
(prove that DP can be applied)
identify an ordering of sub-problems such that they are topologically sorted (so computing a solution relies only on trivial solutions and previously-computed solutions, not on future ones)
fill a table iteratively in that order, if there was a nice order. Maybe keep the Top Down structure if the order is annoying.
In the case of Fibonacci, what happened is that the order is trivial (that's not even particularly uncommon, but it makes it look as if we're "not really doing anything special"), and the dependencies never go back more than 2 places, so the only part of the table that has to be remembered is the previous two cells. So applying all that, you get the well-known iterative algorithm. That doesn't mean it's not DP anymore, it means that the DP was extremely successful.
As for the properties (optimal substructure, overlapping subproblems), they're properties of the problem, they don't go away no matter how you decide to solve it. But you can still see them back in the iterative algorithm, as I pointed out in the first paragraph.
On the wikipedia page of Dynamic Programming read - Dynamic programming in computer programming. This explains about the two approaches, Top Down that falls out as recursive formulation of the problem and Bottom Up in which we iteratively generate solution to bigger problems and current solution are stored in table. In this case, a table is not required and the job can be done using two variables.
Thus, in case of Iterative Approach, only two variables are being used for holding the values of subproblems; ie, prev, (i-1 th) and prevPrev, (i-2 th). Here, prev and prevPrev are used to find the solution of the ith iteration (Bigger problem).
result = prev + prevPrev;
is nothing but representation of ith iteration result, which is equal to prev(i-1) + prevPrev(i-2). Thus, reuse of subproblems is taking place in the iterative approach too.
This is the bottom up approach of dynamic programming and the recursive approach is Top Down approach of dynamic programming.
Related
Hey guys i have a quiz tommorow i am facing some problems finding time complexity of recursive programm of selection sort can any one explain how is it n^2. And also one more question that some sorting algorithms loops have n/2 time complexity what is the meaning of /2 sorry for newbie question.
#include <iostream>
using namespace std;
// recursive function to perform selection sort on subarray arr[i..n-1]
void selectionSort(int arr[], int i, int n)
{
// find the minimum element in the unsorted subarray[i..n-1]
// and swap it with arr[i]
int min = i;
for (int j = i + 1; j < n; j++)
{
// if arr[j] element is less, then it is the new minimum
if (arr[j] < arr[min])
min = j; // update index of min element
}
// swap the minimum element in subarray[i..n-1] with arr[i]
swap(arr[min], arr[i]);
if (i + 1 < n)
selectionSort(arr, i + 1, n);
}
Finding time complexity is often described in a way that is not really very helpful. Here is how it works for Selection Sort.
passes
The very first time through the algorithm, you must scan all n elements of the data.
The very next time through (on recursion), you must scan all but one, which is (n-1).
And so on. We can write the number of times we look at/compare an element as:
n + (n-1) + (n-2) + ... + 2 + 1
(You can quibble about that last term, but for simplicity here we just won't care. You'll see why in just a second.)
math identities
This particular series (notice all the additions) is called an “arithmetic progression”. The difference between each term is 1. The first term is n and the last term is 1 (or 2, whatever).
And using some math most people don’t remember (or weren’t taught) in High School, we can rewrite that sum as:
n(n+1)
──────
2
(Yes, again, that last term is actually supposed to be a two, even though I’ve left it at one.)
as n grows arbitrarily large
Big O doesn’t care much about friendly values of n. It cares about what happens when n gets arbitrarily big. We say “n grows towards infinity”.
As it does, we can notice two things happening:
That division by 2 becomes insignificant: ∞/2 → ∞
That addition by 1 (or whatever) is also insignificant: ∞(∞+1) → ∞(∞)
So ultimately, we have infinity, er, n multiplied by itself. The equation simplifies to:
n(n+1)
────── → n²
2
complexity bounds
The worst case behavior is n², so we annotate that as “O(n²)”. But notice that the best case is also n². We annotate that as “Ω(n²)”. Finally, since the best and worst case behaviors are the same, we have a very nice tight bound on the behavior of the function. We annotate this as “Θ(n²)”.
Hence, selection sort has Θ(n²) complexity.
holy quiznak! I’m supposed to do this myself!!?
Yes, unfortunately, figuring out complexity is one of those things that people treat as if it were really easy — even when they don’t understand it well themselves. It takes some familiarity with math and some practice. The typical response is as in the links provided you above: ‘look at these common cases and choose the one that most closely matches’. I personally find that less satisfyingly instructive. It is also one of those things that university professors expect you to pick up right away.
My advice: don’t worry too hard. Do your best to understand the underlying principles. What we are doing is finding a function (like y=x²) that represents an algorithm’s behavior whenever its input is really large (n → ∞).
Being able to follow through the paths that the algorithm takes and recognizing when those paths are expensive is a more important ability than being able to come up with a mathematician’s answer. If you really need it you can always find a mathematician to help, either at the university or on the internet if you ask in the right places.
And, of course, feel free to take time to try to understand better and practice. Being able to do it right is a valuable skill, regardless of whether others see it.
:O)
http://ideone.com/QXyVzR
The above link contains a program I wrote to solve mazes using a BFS algorithm. The maze is represented as a 2D array, initially passed in as numbers, (0's represent an empty block which can be visited, any other number represent a "wall" block), and then converted into a record type which I defined, which keeps track of various data:
type mazeBlock = {
walkable : bool;
isFinish : bool;
visited : bool;
prevCoordinate : int * int
}
The output is a list of ordered pairs (coordinates/indices) which trace a shortest path through the maze from the start to the finish, the coordinates of which are both passed in as parameters.
It works fine for smaller mazes with low branching factor, but when I test it on larger mazes (say 16 x 16 or larger), especially on ones with no walls(high branching factor) it takes up a LOT of time and memory. I am wondering if this is inherent to the algorithm or related to the way I implemented it. Can any OCaml hackers out there offer me their expertise?
Also, I have very little experience with OCaml so any advice on how to improve the code stylistically would be greatly appreciated. Thanks!
EDIT:
http://ideone.com/W0leMv
Here is an cleaned-up, edited version of the program. I fixed some stylistic issues, but I didn't change the semantics. As usual, the second test still takes up a huge amount of resources and cannot seem to finish at all. Still seeking help on this issue...
EDIT2:
SOLVED. Thanks so much to both answerers. Here is the final code:
http://ideone.com/3qAWnx
In your critical section, that is mazeSolverLoop, you should only visited elements that have not been visited before. When you take the element from the queue, you should first check if the element has been visited, and in that case do nothing but recurse to get the next element. This is precisely what makes the good time complexity of the algorithm (you never visit a place twice).
Otherwise, yes, your OCaml style could be improved. Some remarks:
the convention in OCaml-land is rather to write_like_this instead of writeLikeThis. I recommend that you follow it, but admittedly that is a matter of taste and not an objective criterion.
there is no point in returning a datastructure if it is a mutable structure that was updated; why do you make a point to always return a (grid, pair) queue, when it is exactly the same as the input? You could just have those functions return unit and have code that is simpler and easier to read.
the abstraction level allowed by pairs is good and you should preserve it; you currently don't. There is no point in writing for example, let (foo, bar) = dimension grid in if in_bounds pos (foo, bar). Just name the dimension dim instead of (foo, bar), it makes no sense to split it in two components if you don't need them separately. Remark that for the neighbor, you do use neighborX and neighborY for array access for now, but that is a style mistake: you should have auxiliary functions to get and set values in an array, taking a pair as input, so that you don't have to destruct the pair in the main function. Try to keep all the code inside a single function at the same level of abstraction: all working on separate coordinates, or all working on pairs (named as such instead of being constructed/deconstructed all the time).
If I understand you right, for an N x N grid with no walls you have a graph with N^2 nodes and roughly 4*N^2 edges. These don't seem like big numbers for N = 16.
I'd say the only trick is to make sure you track visited nodes properly. I skimmed your code and don't see anything obviously wrong in the way you're doing it.
Here is a good OCaml idiom. Your code says:
let isFinish1 = mazeGrid.(currentX).(currentY).isFinish in
let prevCoordinate1 = mazeGrid.(currentX).(currentY).prevCoordinate in
mazeGrid.(currentX).(currentY) <-
{ walkable = true;
isFinish = isFinish1;
visited = true;
prevCoordinate = prevCoordinate1}
You can say this a little more economically as follows:
mazeGrid.(currentX).(currentY) <-
{ mazeGrid.(currentX).(currentY) with visited = true }
Apart from this code being horrible inefficient, is this way I´m writing recursive function here considered "good style". Like for example what I am doing creating a wrapper then passing it the int mid and a counter int count.
What this code does is getting values from the array then see if that combined with blockIndex is greater than the mid. So, appart from being inefficient would I get a job writing recursive functions like this?
int NumCriticalVotes :: CountCriticalVotesWrapper(Vector<int> & blocks, int blockIndex)
{
int indexValue = blocks.get(blockIndex);
blocks.remove(blockIndex);
int mid = 9;
return CountCriticalVotes(blocks, indexValue, mid, 0);
}
int NumCriticalVotes :: CountCriticalVotes(Vector<int> & blocks, int blockIndex, int mid, int counter)
{
if (blocks.isEmpty())
{
return counter;
}
if (blockIndex + blocks.get(0) >= mid)
{
counter += 1;
}
Vector<int> rest = blocks;
rest.remove(0);
return CountCriticalVotes(rest, blockIndex, mid, counter);
}
This is valid to the extent that it'll work for sufficiently small collections.
It is, however, quite inefficient -- for each recursive call you're creating a copy of the entire uncounted part of the Vector. So, if you count a vector containing, say, 1000 items, you'll first create a Vector of 999 items, then another of 998 items, then another of 997, and so on, all the way down to 0 items.
This would be pretty wasteful by itself, but seems to even get worse. You're then removing an item from your Vector -- but you're removing the first item. Assuming your Vector is something like std::vector, removing the last item takes constant time but removing the first item takes linear time -- i.e., to remove he first item, each item after that is shifted "forward" into the vacated spot.
This means that instead of taking constant space and linear time, your algorithm is quadratic in both space and time. Unless the collection involved is extremely small, it's going to be quite wasteful.
Instead of creating an entire new Vector for each call, I'd just pass around offsets into the existing Vector. This will avoid both copying and removing items, so it's pretty trivial to make it linear in both time and space (which is still well short of optimum, but at least not nearly as bad as quadratic).
To reduce the space used still further, treat the array as two halves. Count each half separately, then add together the results. This will reduce recursion depth to logarithmic instead of linear, which is generally quite substantial (e.g., for a 1000 items, it's a depth of about 10 instead of about a 1000. For a million items, the depth goes up to about 20 instead of a million).
Without know exactly what you are trying to accomplish, this is a very tough question to answer. The way I see recursion, or coding in general, is does it satisfy the following three requirements.
Does it accomplish all desired functionality?
Is it error resilient? Meaning it will not break when passed invalid inputs, or edge cases.
Does it accomplish its goal in sufficient time?
I think you are worried about number 3, and I can say that the time should fit the problem. For example, if you searching through 2 huge lists, O(n^2) is probably not acceptable. However, say you are searching through 2 small sets O(n^2) is probably sufficiently fast.
What I can say is to try timing different implementations of your algorithm on test cases. Just because your solution is recursive doesn't mean that it will always be faster than a "brute force" implementation. (This of course depends specifically on the case).
To answer your question, as far as recursion goes, this sample looks fine. However, will you get a job writing code like this? I don't know how well does this satisfy the other two coding requirements?
Very subjective question. The tail recursion is nice (in my book) but I'd balance that against creating a new vector on every call, which makes a linear algorithm quadratic. Independent of recursion, that's a big no-no, particularly as it is easily avoidable.
A few comments about what the code is intended to accomplish would also be helpful, although I suppose in context that would be less problematic.
The issues with your solution is your passing the count around. Stop pass the count and use the stack to keep track of it. The other issue is I'm not sure what your second condition is suppose to do.
int NumCriticalVotes :: CountCriticalVotesWrapper(Vector<int> & blocks, int blockIndex)
{
int indexValue = blocks.get(blockIndex);
blocks.remove(blockIndex);
int mid = 9;
return CountCriticalVotes(blocks, indexValue, mid);
}
int NumCriticalVotes :: CountCriticalVotes(Vector<int> & blocks, int blockIndex, int mid)
{
if (blocks.isEmpty())
{
return 0;
}
if (/*Not sure what the condition is*/)
{
return 1 + CountCriticalVotes(blocks, blockIndex, mid);
}
return CountCriticalVotes(blocks, blockIndex, mid);
}
In C++, traversing lists of arbitrary length using recursion is never a good practice. It's not just about performance. The standard doesn't mandate tail call optimization, so you have a risk of stack overflow if you can't guarantee that the list has some limited size.
Sure, the recursion depth has to be several hundred thousand with typical stack sizes, but it's hard to know when designing a program what kind of inputs it must be able to handle in the future. The problem might come back to haunt you much later.
This is an interview question I faced recently.
Given an array of 1 and 0, find a way to partition the bits in place so that 0's are grouped together, and 1's are grouped together. It does not matter whether 1's are ahead of 0's or 0's are ahead of 1's.
An example input is 101010101, and output is either 111110000 or 000011111.
Solve the problem in less than linear time.
Make the problem simpler. The input is an integer array, with each element either 1 or 0. Output is the same integer array with integers partitioned well.
To me, this is an easy question if it can be solved in O(N). My approach is to use two pointers, starting from both ends of the array. Increases and decreases each pointer; if it does not point to the correct integer, swap the two.
int * start = array;
int * end = array + length - 1;
while (start < end) {
// Assume 0 always at the end
if (*end == 0) {
--end;
continue;
}
// Assume 1 always at the beginning
if (*start == 1) {
++start;
continue;
}
swap(*start, *end);
}
However, the interview insists there is a sub-linear solution. This makes me thinking hard but still not get an answer.
Can anyone help on this interview question?
UPDATE: Seeing replies in SO stating that the problem cannot be solved in sub-linear time, I can confirm my original idea that there cannot be a solution of sub-linear.
Is it possible the interviewer plays a trick?
I don't see how there can be a solution faster than linear time.
Imagine a bit array that is all 1's. Any solution will require examining every bit in this array before declaring that it is already partitioned. Examining every bit takes linear time.
It's not possible. Doing it in less than linear time implies that you don't look at every array element (like a binary search). However since there is no way to know what any element of the array is without looking at it, you must look at each array element at least once.
You can use lookup tables to make it faster, but O(n/8) is still O(n), so either the interviewer was wrong or you misunderstood the question.
It is possible faster then in linear time given you have enough memory, it can be done in O(1)
Use the bitmask as index in a vector which maps to the partitioned bitmask.
using your example, at index 341 (101010101) the value 496 (111110000) is stored.
Perhaps the confusion comes from "less than linear time". For example, this solution counts the number of bits, that makes a masks containing that many bits. It only counts bits while there are uncounted on-bits:
// from http://graphics.stanford.edu/~seander/bithacks.html#CountBitsSetKernighan
unsigned count_bits(unsigned pX)
{
unsigned result;
for (result = 0; v; ++result)
{
pX &= pX - 1;
}
return result;
}
unsigned n = /* the number */;
// r contains 000...111, with number of 1's equal to number of 1's in v
unsigned r = 1 << count_bits(n);
Even though this minimizes the number of bits to count, it's still linear. So if this is what is meant by "sub-linear", there you go.
But if they really meant sub-linear as in logarithmic or constant, I don't see a way. You could conceivably make a look-up table for every value, but :/
Technically you could send each element of the array to a separate processor and then do it in less than linear time. If you have N processors, you could even do it in O(1) time!
As others said, I don't believe this can be done in less than linear time. For linear time solution, you can STL algorithms instead your own loop like this:
int a1[8] = {1,0,1,0,1,0,1,0};
std::fill(std::remove(a1, a1+8, 0), a1+8, 0);
Well.. It can be be done 'less than linear' time (cheeky method).
if(n % 2)
{
// Arrange all 1's to the right and DON'T check the right-most bit, because it's 1
}else{
// Arrange all 0's to the right and DON'T check the right-most bit, because it's 0.
}
So, technically you 'group' the bits in less than linear time :P
To me, the most likely interpretations are:
The bits are supposed to be in an int instead of an array, in which case you can use something like http://graphics.stanford.edu/~seander/bithacks.html#CountBitsSetKernighan or an 8-bit (or more) lookup table.
they used "sublinear" to mean "less than n operations" rather than less-than-O(n). But even that seems impossible for the same reasons listed below.
There is another miscommunication in the question
Otherwise the question is wrong, since all elements of the array must be examined to determine the answer, and that is at least 'n' operations.
Listing either 0s or 1s first, and the references to bits rather than bools make me think something like the first option was intended, even though, when dealing with only one word, it doesn't make very much difference. I'm curious to know what the interviewer actually had in mind.
Splitting this work among parallel processors costs N/M ( or O(N) ) only if you assume that parallelism increases more slowly than problem size does. For the last ten years or so, paralellism (via the GPU) has been increasing more rapidly than typical problem sizes, and this trend looks to continue for years to come. For a broad class of problems, it is instructive to assume "infinite parallelism" or more precisely, "parallelism greater than any expected problem size" because the march of progress in GPUs and cloud computing provides such a thing over time.
Assuming infinite parallelism, this problem can be solved in O(logN) time because the addition operator required to add up all the 0 and 1 bits is associative, and so it requires at least logN time steps to complete.
as we know that the trees are recursive data structures, We use recurrsion in writing the procedures of tree like delete method of BST etc.
the advantage of recurrsion is, our procedures becomes very small (for example the code of inorder traversal is of only 4 or 5 lines) rather than a non recurrsive procedure which would be lengthy but not as complex as recurssive procedure in understanding perspective. that is why i hate recurrsion and i prefer to write non recurrsive procedure and i have done that in binary serach trees and avl trees.
Now please elaborate that, prefering non recursive procedures over recurrsive procedures is bad or good thing."
Recursion is a tool like any other. You don't have to use every tool that's available but you should at least understand it.
Recursion makes a certain class of problems very easy and elegant to solve and your "hatred" of it is irrational at best. It's just a different way of doing things.
The "canonical" recursive function (factorial) is shown below in both recursive and iterative forms and, in my opinion, the recursive form more clearly reflects the mathematical definition of f(1) = 1, f(n) = n*f(n-1) for n>1.
Iterative: Recursive:
def fact(n): def fact(n):
r = n if n == 1:
while n > 1: return 1
r = r * n return n * fact(n-1)
n = n - 1
return r
Pretty much the only place I would prefer an iterative solution to a recursive one (for solutions that are really well suited for recursion) is when the growth in stack size may lead to problems (the above factorial function may well be one of those since stack growth depends on n but it may also be optimised to an iterative solution by the compiler). But this stack overflow rarely happens since:
Most stacks can be configured where necessary.
Recursion (especially tail-end recursion where the recursive call is the last thing that happens in the function) can usually be optimised to an iterative solution by an intelligent compiler.
Most algorithms I use in recursive situations (such as balanced trees and so on, as you mention) tend to be O(logN) and stack use doesn't grow that fast with increased data. For example, you can process a 16-way tree storing two billion entries with only seven levels of stack (167 =~ 2.6 billion).
You should read about Tail Recursion. In general, if a compiler manages to apply tail recursion to a procedure, it it quite effective, if not, then not so.
Also a important issue is the maximum recusion depth of your compiler -- usually it's limited by the stack size. The downside here is that there's no graceful way to handle a stack overflow.
Recursion is elegant, but prone to stack overflowing. Use tail-end recursion whenever possible to give the compiler the chance to convert it an iterative solution.
It's definitely you decision which tool you want to use - but keep in mind that most algorithms dealing with tree-like data structures are usually implemented recursively. As it's common practice, your code is easier to read and less surprising for others.
Recursion is a tool. Sometimes using the "tool of recursion" makes the code easier to read, although not necessarily easier to comprehend.
In general, recursive solutions tend to be good candidates where a "divide and conquer" approach to solving a specific problem is natural.
Typically, recursion is a good fit where you can look at a problem and say "aha, if I knew the answer for a simpler variant f this problem, I could use that solution to generate the answer I want" and "the simplest possible problem is P and its solution is S". Then, the code to solve the problem as a whole boils down to looking at the in-data, simplifying it, recursively generate a (simpler) answer and then go from the simpler answer to the answer as a whole.
If we consider the problem of counting the levels of a tree, the answer is that the height of the tree is 1 more than the height of the "tallest/deepest" of the children and the height of a leaf is 1. Something like the following code. The problem can be solved iteratively, but you'd, essentially, re-implement the call stack in your own data structures.
def tree_height (tree):
if tree.is_leaf():
return 1
childmax = 0;
for child in tree.children():
childmax=max(childmax, tree_height(child))
return childmax+1
It's also worth considering that Tail Call Optimization can make some recursive functions running in constant stack space.