I was thinking of a fast method to look for a submatrix m in a bigger mtrix M.
I also need to identify partial matches.
Couple of approaches I could think of are :
Optimize the normal bruteforce to process only incremental rows and columns.
May be extend Rabin-karp algorithm to 2-d but not sure how to handle partial matches with it.
I believe this is quite frequently encountered problem in image processing and would appreciate if someone could pour in their inputs or point me to resources/papers on this topic.
EDIT:
Smaller example:
Bigger matrix:
1 2 3 4 5
4 5 6 7 8
9 7 6 5 2
Smaller Matrix:
7 8
5 2
Result: (row: 1 col: 3)
An example of Smaller matrix which qualifies as a partial match at (1, 3):
7 9
5 2
If More than half of pixels match, then it is taken as partial match.
Thanks.
I recommend doing an internet search on "2d pattern matching algorithms". You'll get plenty of results. I'll just link the first hit on Google, a paper that presents an algorithm for your problem.
You can also take a look at the citations at the end of the paper to get an idea of other existing algorithms.
The abstract:
An algorithm for searching for a two dimensional m x m pattern in a two dimensional n x n text is presented. It performs on the average less comparisons than the size of the text: n^2/m using m^2 extra space. Basically, it uses multiple string matching on only n/m rows of the text. It runs in at most 2n^2 time and is close to the optimal n^2 time for many patterns. It steadily extends to an alphabet-independent algorithm with a similar worst case. Experimental results are included for a practical version.
There are very fast algorithms for this if you are willing to preprocess the matrix and if you have many queries for the same matrix.
Have a look at the papers on Algebraic Databases by the Research group on Multimedia Databases (Prof. Clausen, University of Bonn). Have a look at this paper for example: http://www-mmdb.iai.uni-bonn.de/download/publications/sigir-03.pdf
The basic idea is to generalize inverted list, so they use any kind of algebraic transformation, instead of just shifts in one direction as with ordinary inverted lists.
This means that this approach works whenever the modifications you need to do to the input data can be modelled algebraically. This specifically that queries which are translated in any number of dimensions, rotated, flipped etc can all be retrieved.
The paper is mainly showing this for musical data, since this is their main research interest, but you might be able to find others, which show how to adapt this to image data as well (or you can try to adapt it yourself, if you understand the principle it's quite simple).
Edit:
This idea also works with partial matches, if you define them correctly.
There is no way to do this fast if you only ever need to match one small matrix against one big matrix. But if you need to do many small matrices against big matrices, then preprocess the big matrix.
A simple example, exact match, many 3x3 matrices against one giant matrix.
Make a new "match matrix", same size as "big matrix", For each location in big matrix compute a 3x3 hash for each x,y to x+3,y+3 in big matrix. Now you just scan the match matrix for matching hashes.
You can achieve partial matches with specialized hash functions that give the same hash to things that have the same partial matching properties. Tricky.
If you want to speed up further and have memory for it, create a hash table for the match matrix, and lookup the hashes in the hash table.
The 3x3 solution will work for any test matrix 3x3 or larger. You don't need to have a perfect hash method - you need just something that will reject the majority of bad matches, and then do a full match for potential matches in the hash table.
I think you cannot just guess where the submatrix is with some approach, but you can optimize your searching.
For example, given a matrix A MxN and a submatrix B mxn, you can do like:
SearchSubMatrix (Matrix A, Matrix B)
answer = (-1, -1)
Loop1:
for i = 0 ... (M-m-1)
|
| for j = 0 ... (N-n-1)
| |
| | bool found = true
| |
| | if A[i][j] = B[0][0] then
| | |
| | | Loop2:
| | | for r = 0 ... (m-1)
| | | | for s = 0 ... (n-1)
| | | | | if B[r][s] != A[r+i][s+j] then
| | | | | | found = false
| | | | | | break Loop2
| |
| | if found then
| | | answer = (i, j)
| | | break Loop1
|
return answer
Doing this, you will reduce your search in the reason of the size of the submatrix.
Matrix Submatrix Worst Case:
1 2 3 4 2 4 [1][2][3] 4
4 3 2 1 3 2 [4][3][2] 1
1 3 2 4 [1][3]{2 4}
4 1 3 2 4 1 {3 2}
(M-m+1)(N-n+1) = (4-2+1)(4-2+1) = 9
Although this is O(M*N), it will never look M*N times, unless your submatrix has only 1 dimension.
Related
I have data as follows - please see the fiddle here for all data and code below:
INSERT INTO t VALUES
('|0|34| first zero'),
('|45|0| second zero'),
('|0|0| both zeroes');
I want to SELECT from the start of the line
1st character in the line is a piple (|)
next characters are a valid (possibly negative - one minus sign) INTEGER
after the valid INT, another pipe
then another valid INT
then a pipe
The rest of the line can be anything at all - including sequences with pipe, INT, pipe, INT - but these are not to be SELECTed!
and I'm using a regex to try and SELECT the valid INTEGERs. A single ZERO is also a valid reading - one ZERO and one ZERO only!
The valid integers must be from between the first 3 pipe (|) characters and not elsewhere in the line - i.e.
^|3|3|adfasfadf |555|6666| -- tuple (3, 3) is valid
but
^|--567|-765| adfasdf -- tuple (--567, -765) is invalid - two minus signs!
and
^|This is stuff.... |34|56| -- tuple (34, 56) is invalid - doesn't start pipe, int, pipe, int!
Now, my regexes (so far) are as follows:
SELECT
SUBSTRING(a, '^\|(0{1}|[-+]?[1-9]{1}\d*)\|') AS n1,
SUBSTRING(a, '^\|[-+]?[1-9]{1}\d*\|(0{1}|[-+]?[1-9]{1}\d*)\|') AS n2,
a
FROM t;
and the results I'm getting for my 3 records of interest are:
n1 n2 a
0 NULL |0|34| first zero -- don't want NULL, want 34
45 0 |45|0| second zero -- OK!
0 NULL |0|0| both zeroes -- don't want NULL, want 0
3 3 |3|3| some stuff here
...
... other data snipped - but working OK!
...
Now, the reason why it works for the middle one is that I have (0{1}|.... other parts of the regex in both the upper and lower one!
So, that means take 1 and only 1 zero OR... the other parts of the regex. Fine, I've got that much!
However, and this is the crux of my problem, when I try to change:
'^\|[-+]?[1-9]{1}\d*\|(0{1}|[-+]?[1-9]{1}\d*)\|'
to
'^\|0{1}|[-+]?[1-9]{1}\d*\|(0{1}|[-+]?[1-9]{1}\d*)\|'
Notice the 0{1}| bit I've added near the beginning of my regex - so, this should allow one and only one ZERO at the beginning of the second string (preceded by a pipe literal (|)) OR the rest... the pipe at the end of my 5 character snippet above in this case being part of the regex.
But the result I get is unchanged for the first 3 records - shown above, but it now messes up many records further down - one example a record like this:
|--567|-765|A test of bad negatives...
which obviously fails (NULL, NULL) in the first SELECT now returns (NULL,-765) for the second. If the first fails, I want the second to fail!
I'm at a loss to understand why adding 0{1}|... should have this effect, and I'm also at a loss to understand why my (0, NULL), (45, 0) and (0, NULL) don't give me (0, 0), (45, 0) and (0, 0) as I would expect?
The 0{1}| snippet appears to work fine in the capturing groups, but not outside - is this the problem? Is there a problem with PostgreSQL's regex implementation?
All I did was add a bit to the regex which said as well as what you've accepted before, please accept one and only one leading ZERO!
I have a feeling there's something about regexes I'm missing - so my question is as follows:
could I please receive an explanation as to what's going on with my regex at the moment?
could I please get a corrected regex that will work for INTEGERs as I've indicated. I know there are alternatives, but I'd like to get to the bottom of the mistake I'm making here and, finally
is there an optimum/best method to achieve what I want using regexes? This one was sort of cobbled together and then added to as further necessary conditions became clearer.
I would want any answer(s) to work with the fiddle I've supplied.
Should you require any further information, please don't hesitate to ask! This is not a simple "please give me a regex for INTs" question - my primary interest is in fixing this one to gain understanding!
Some simplifications could be done to the patterns.
SELECT
SUBSTRING(a, '^\|(0|[+-]?[1-9][0-9]*)\|[+-]?[0-9]+\|') AS n1,
SUBSTRING(a, '^\|[+-]?[0-9]+\|(0|[+-]?[1-9][0-9]*)\|') AS n2,
a
FROM t;
n1 | n2 | a
:--- | :--- | :--------------------------------------------------------------
0 | 34 | |0|34| first zero
45 | 0 | |45|0| second zero
0 | 0 | |0|0| both zeroes
3 | 3 | |3|3| some stuff here
null | null | |SE + 18.5D some other stuff
-567 | -765 | |-567|-765|A test of negatives...
null | null | |--567|-765|A test of bad negatives...
null | null | |000|00|A test of zeroes...
54 | 45 | |54|45| yet more stuff
32 | 23 | |32|23| yet more |78|78| stuff
null | null | |This is more text |11|111|22222||| and stuff |||||||
null | null | |1 1|1 1 1|22222|
null | null | |71253412|ahgsdfhgasfghasf
null | null | |aadfsd|34|Fails if first fails - deliberate - Unix philosophy!
db<>fiddle here
Just looking for a bit of direction, I realise that the example given is possible to solve using brute force iteration, but I am looking for a more elegant (ie. mathematical?) solution which could potentially solve significantly larger examples (say 20x20 or 30x30). It is entirely possible that this cannot be done, and I have had very little success in coming up with an approach which does not rely on brute force...
I have a matrix (call it A) which is nxn. I wish to select a subset (call it B) of points from matrix A. The subset will consist of n elements, where one and only one element is taken from each row and from each column of A. The output should provide a solution (B) such that the sum of the elements that make up B is the maximum possible value, given these constraints (eg. 25 in the example below). If multiple instances of B are found (ie. different solutions which give the same maximum sum) the solution for B which has the largest minimum element should be selected.
B could also be a selection matrix which is nxn, but where only the n desired elements are non-zero.
For example:
if A =
|5 4 3 2 1|
|4 3 2 1 5|
|3 2 1 5 4|
|2 1 5 4 3|
|1 5 4 3 2|
=> B would be
|5 5 5 5 5|
However, if A =
|5 4 3|
|4 3 2|
|3 2 1|
B =
|3 3 3|
As the minimum element of B is 3 which is larger than for
|5 3 1|
or
|4 4 1|
which also both sum to 9
Your problem is almost identical to the Assignment problem, which can e.g. be solved by the Hungarian algorithm in polynomial time.
Note that the assignment problem is usually a minimization problem, but multiplying your matrix with -1 and adding some constant should make the method applicable. Further, there is no formal tie-braking condition, for case of multiple optimal solutions. However, the method yields you a solution having the optimal sum. Let m be the minimum summand. Modify your matrix by setting all entries less or equal to m to zero and solve again. Either you get a solution with the same sum that is better than the last one. If not, the previous solution was already optimal.
As Matthias indicated you should use backtracking.
Find a reasonable solution. Select max values from each row and see if they are non-overlapping. If not, then perturb part of the solution so that the result becomes non-overlapping.
Define fitness of a partial solution. Let us say you are picking up value for each row iteratively and you have already picked values from first k rows. The fitness of this solution equals sum of the already picked values + max values from remaining rows and unselected columns
Now recursively start searching for solution. Select the values from first row, calculate their fitness and insert them into a priority queue. Remove all the solutions whose fitness is lower than the current optimal solution (initialized in step 1). Pick the solution at the head of the queue, calculate the next level of solutions and insert them back to the priority queue. Once you have selected values from all columns and rows, calculate the sum, and if it is higher than current optimal, replace it.
Ouch. This algorithm is wrong; there is no proof because it's wrong and therefore it's impossible to prove that it's correct. ;) I'm leaving it here because I'm too attached to delete it entirely, and it's a good demonstration of why you should formally prove algorithms instead of saying "this looks right! There's no possible way this could fail to work!"
I'm giving this solution without proof, for the time being. I have a proof sketch but I'm having trouble proving optimal substructure for this problem. Anyway...
Problem
Given a square array of numbers, select as many "non-overlapping" numbers as possible so that the sum of the selected numbers is maximised. "Non-overlapping" means that no two numbers can be from the same row or the same column.
Algorithm
Let A be a square array of n by n numbers.
Let Aij denote the element of A in the ith row and jth column.
Let S( i1:i2, j1:j2 ) denote the optimal sum of non-overlapping numbers for a square subarray of A containing the intersection of rows i1 to i2 and columns j1 to j2.
Then the optimal sum of non-overlapping numbers is denoted S( 1:n , 1:n ) and is given as follows:
S( 1:n , 1:n ) = max { [ S( 2:n , 2:n ) + A11 ]
[ S( 2:n , 1:n-1 ) + A1n ]
[ S( 1:n-1 , 2:n ) + An1 ]
[ S( 1:n-1 , 1:n-1 ) + Ann ] }
(recursively)
Note that S( i:i, j:j ) is simply Aij.
That is, the optimal sum for a square array of size n can be determined by separately computing the optimal sum for each of the four sub-arrays of size n-1, and then maximising the sum of the sub-array and the element that was "left out".
S for |# # # #|
|# # # #|
|# # # #|
|# # # #|
Is the best of the sums S for:
|# | | #| |# # # | | # # #|
| # # #| |# # # | |# # # | | # # #|
| # # #| |# # # | |# # # | | # # #|
| # # #| |# # # | | #| |# |
Implementation
The recursive algorithm above suggests a recursive solution:
def S(A,i1,i2,j1,j2):
if (i1 == i2) and (j1==j2):
return A[i1][j1]
else:
return max ( S( A, i1+1, i2, j1+1, j2) + A[i1][j1] ],
S( A, i1+1, i2, j1, j2-1) + A[i1][j2] ],
S( A, i1, i2-1, j1+1, j2) + A[i2][j1] ],
S( A, i1, i2-1, j1, j2-1) + A[i2][j2] ], )
Note that this will make O(4^n) calls to S()!! This is much better than the factorial O(n!) time complexity of the "brute force" solution, but still awful performance.
The important thing to note here is that many of the calls are repeated with the same parameters. For example, in solving a 3*3 array, each 2*2 array is solved many times.
This suggests two possible solutions for a speedup:
Make the recursive function S() cache results so that it only needs to S(A,i1,i2,j1,j2) once for each i1,i2,j1,j2. This means that S() only needs to calculate O(n^3) results - all other requests will be fufilled from cache. (This is called memoising.)
Instead of starting at the top, with the large n*n array, and working down through successively smaller subproblems, start at the bottom with the smallest possible subproblems and build up to the n*n case. This is called dynamic programming. This is also O(n^3), but it's a much faster O(n^3) because you don't have to hit a cache all the time.
The dynamic programming solution proceeds somewhat like:
Find optimal solutions to all 1x1 sub-arrays. (Trivial.)
Find optimal solutions for all 2x2 sub-arrays.
Find optimal solutions for all 3x3 sub-arrays.
...
Find optimal solutions for all n-1 * n-1 sub-arrays.
Find optimal solutions for the complete n*n sub-array.
Notes on this solution:
No proof yet. I'm working on it.
You'll note the algorithm above only gives you S(), the optimal sum. It doesn't tell you which numbers actually make up that sum. You get to add in your own method of backtracing your path to the solution.
The algorithm above doesn't guarantee the property that ties like 2,2 vs. 1,3 will be broken in favour of having all the individual numbers be as large as possible (so that 2,2 wins.) I believe you can define max() to break ties in favour of the largest numbers possible, and that will do what you want, but I can't prove it.
General notes:
Dynamic programming is a powerful technique for devising fast algorithms for any problem which exhibits two properties:
Optimal substructure: A problem can be broken down into slightly smaller parts, each of which can be used as part of the solution to the original problem.
Overlapping subproblems means that there are few actual subproblems to solve, and the solutions to the subproblems are re-used many times.
If the problem has optimal substructure, and the problem breaks down into slightly smaller problems - say a problem of size n breaks down into subproblems of size n-1 - then the problem can be solved by dynamic programming.
If you can split the problem into much smaller chunks - say chopping a problem of size n into halves, each of size n/2 - that's divide and conquer, not dynamic programming. Divide and conquer solutions are generally very fast - for example binary search will find an element in a sorted array in O(log n) time.
This is related to the n Queens problem, except that you do not care about the diagonal and you have weighted solutions. As the Queens problem, you can solve it by (multiple) backtracking.
I.e., once you find a solution you remember its weight, mark the soulution as invalid, and start over. The (a) solution with the highest weight wins.
I'm trying to find an algorithm which "breaks the safe" by typing the keys 0-9. The code is 4 digits long. The safe will be open where it identifies the code as substring of the typing. meaning, if the code is "3456" so the next typing will open the safe: "123456". (It just means that the safe is not restarting every 4 keys input).
Is there an algorithm which every time it add one digit to the sequence, it creates new 4 digits number (new combinations of the last 4 digits of the sequence\string)?
thanks, km.
Editing (I post it years ago):
The question is how to make sure that every time I set an input (one digit) to the safe, I generate a new 4 digit code that was not generated before. For example, if the safe gets binary code with 3 digits long then this should be my input sequence:
0001011100
Because for every input I get a new code (3 digit long) that was not generated before:
000 -> 000
1 -> 001
0 -> 010
1 -> 101
1 -> 011
1 -> 111
0 -> 110
0 -> 100
I found a reduction to your problem:
Lets define directed graph G = (V,E) in the following way:
V = {all possible combinations of the code}.
E = {< u,v > | v can be obtained from u by adding 1 digit (at the end), and delete the first digit}.
|V| = 10^4.
Din and Dout of every vertex equal to 10 => |E| = 10^5.
You need to prove that there is Hamilton cycle in G - if you do, you can prove the existence of a solution.
EDIT1:
The algorithm:
Construct directed graph G as mentioned above.
Calculate Hamilton cycle - {v1,v2,..,vn-1,v1}.
Press every number in v1.
X <- v1.
while the safe isn't open:
5.1 X <- next vertex in the Hamilton path after X.
5.2 press the last digit in X.
We can see that because we use Hamilton cycle, we never repeat the same substring. (The last 4 presses).
EDIT2:
Of course Hamilton path is sufficient.
Here in summary is the problem I think you are trying to solve and some explanation on how i might approach solving it. http://www.cs.swan.ac.uk/~csharold/cpp/SPAEcpp.pdf
You have to do some finessing to make it fit into the chinese post man problem however...
Imagine solving this problem for the binary digits, three digits strings. Assume you have the first two digits, and ask your self what are my options to move to? (In regards to the next two digit string?)
You are left with a Directed Graph.
/-\
/ V
\- 00 ----> 01
^ / ^|
\/ ||
/\ ||
V \ |V
/-- 11 ---> 10
\ ^
\-/
Solve the Chinese Postman, you will have all combinations and will form one string
The question is now, is the Chinese postman solvable? There are algorithms which determine weather or not a DAG is solvable for the CPP, but i don't know if this particular graph is necessarily solvable based on the problem alone. That would be a good thing to determine. You do however know you could find out algorithmically weather it is solvable and if it is you could solve it using algorithms available in that paper (I think) and online.
Every vertex here has 2 incoming edges and 2 outgoing edges.
There are 4 (2^2) vertexes.
In the full sized problem there are 19683( 3 ^ 9 ) vertexs and every vertex has 512 ( 2 ^ 9 ) out going and incoming vertexes. There would be a total of
19683( 3 ^ 9 ) x 512 (2 ^ 9) = 10077696 edges in your graph.
Approach to solution:
1.) Create list of all 3 digit numbers 000 to 999.
2.) Create edges for all numbers such that last two digits of first number match first
two digits of next number.
ie 123 -> 238 (valid edge) 123 -> 128 (invalid edge)
3.) use Chinese Postman solving algorithmic approaches to discover if solvable and
solve
I would create an array of subsequences which needs to be updates upon any insertion of a new digit. So in your example, it will start as:
array = {1}
then
array = {1,2,12}
then
array = {1,2,12,3,13,23,123}
then
array = {1,2,12,3,13,23,123,4,14,24,124,34,134,234,1234}
and when you have a sequence that is already at the length=4 you don't need to continue the concatenation, just remove the 1st digit of the sequence and insert the new digit at the end, for example, use the last item 1234, when we add 5 it will become 2345 as follows:
array = {1,2,12,3,13,23,123,4,14,24,124,34,134,234,1234,5,15,25,125,35,135,235,1235,45,145,245,1245,345,1345,2345,2345}
I believe that this is not a very complicated way of going over all the sub-sequences of a given sequence.
I have drawn my answers in paint, are they correct?
(4c) For the alphabet {0, 1} construct finite state automata corresponding to each of the following regular expressions:
(i) 0
(ii) 1 | 0
(iii) 0 * (1 | 0)
The first two are correct, although the first one might be able to be written as (depending on your convention)
(0) -- 0 --> ((1))
The last one is also correct, but can be simplified to (whenever you have ε appearing, there is likely to be a way to compress the edges and states together to remove it)
+- 0 -+
| |
v |
(0) ---+
/ \
1 0
\ /
v
((1))
(Excuse my ascii diagrams. I'm using (..) for each state, and ((..)) for final states.)
Notice that the 0* is basically a loop from a state to itself, since after reading a 0 the remaining regex to match is the same (as long as we aren't at the end of a string).
I have a huge set data records in disk that arranged in sorted order based on some key(s).
The data is read into memory a block (thousands of records) at a time.
I have to search and display all records matching a key.
I was thinking of some binary search based algorithm, but I have some restrictions here.
Records can be only sequentially looked up within a block from the start of the block.
Records with the same key can span multiple blocks (as shown in the figure - 8 spans). In binary search, if I am loading the middle block and if the first record matches, then I have to
scan the blocks previous to the matched block.
Can someone help me devise an efficient strategy that could work in C++. Will it be efficient to go with the linear search method.
+---+
| 1 | Block1
| 3 |
| 3 |
| 4 |
+---+
| 4 | Block2
| 6 |
| 7 |
| 8 |
+---+
| 8 | Block3
| 8 |
| 8 |
| 8 |
+---+
| 8 | Block4
| 14|
| 15|
| 16|
+---+
You could build a secondary array that consists of the first entry in each block then run binary search on that array. The indices for the array should corresponding directly with the block indices making it an O(1) lookup to get the corresponding block.
It cuts the worst case from O(n) to O(logn) and is still relatively simple.
Your idea, using a binary search, is correct. And you can avoid the linear scans alltogether by saving both the minimum and maximum value in each node. In your example the constructed binary-search-tree will look like this:
Block1 <- (1,4)
(1,8)
Block2 <- (4,8)
(1,16)
Block3 <- (8,8)
(8,16)
Block4 <- (8,16)
....
Having both the max and min values makes it efficient to compute the higher nodes. In your example, if you want to search for "8" you will go ...->...->...->(1,16)->(1,8)->(4,8), so you found the correct block without seeking backward and in the most efficent (log(n)) correct way.
It's a well-known problem, and there is a well-known data structure to solve it, employed mostly in databases :)
The idea is to use a B+ Tree.
The idea is to superpose a kind of Binary Search Tree (except that there'll more than 2 children per node) on top of the structure to search in.
If you have any idea of key distribution you could improve a binary search by guesstimating the first location to check. As a sample, with a "Name" key and the value "Bob" you can approximate where "B" is located, either simply based on position in alphabet, or more complex with domain-specific knowledge of the key (distribution frequency of first character in english first names e.g.)
Either way, a binary search is the way to go, optionally with first-key-in-block preloading or caching-as-you-see-them.