I've been looking all over the internet for a simple thinning algorithm and I stumbled across this: Thinning algorithm The problem is, I do not have too much experience with the dereference operator. Also, my project is in python which has a different way of handling this situation. So I have a few questions
1: What is this bit of code doing?
void myThinningInit (CvMat ** kpw, CvMat ** kpb)
{
/ / Kernel for cvFilter2D
/ / The algorithm kpw kernel binary image and it has become a matching white, black,
/ / Convolution is divided into two sets of binary image was inverted kpb kernel, then take the AND
for (int i = 0; i <8; i + +) {
* (Kpw + i) = cvCreateMat (3, 3, CV_8UC1);
* (Kpb + i) = cvCreateMat (3, 3, CV_8UC1);
cvSet (* (kpw + i), cvRealScalar (0), NULL);
cvSet (* (kpb + i), cvRealScalar (0), NULL);
}.....
And 2: How can I translate this kernels creation into python?
He ends up making 8 kernels but I have no idea what their matrix form looks like.
I don't understand what "* (kpw + i)" or "* (kpb + i)" does in the grand scheme of the program.
3) Can I just make the kernels and store them in a list? If so, how could I do that?
UPDATE:
k = [1, 2, 3, 5, 6, 7, 8]
kpw = []
kpb = []
for i in k:
kpw.append [i] = cv.CreateMat (3, 3, cv.CV_8UC1)
kpb.append [i] = cv.CreateMat (3, 3, cv.CV_8UC1)
cv.cvSet (kpw [i], cv.RealScalar (0), cv.NULL)
cv.cvSet (kpb [i], cv.RealScalar (0), cv.NULL)
At first I didn't just had kpw [i] and it was throwing me an error. After a quick google search I found that you needed to index the array first and the way they did that was through append. I tried this bit of code in order to get 8 base kernels of 3x3 in size but I received this error:
Traceback (most recent call last):
File "/home/krtzer/Documents/python_scripts/thinning.py", line 14, in
kpw.append [i] = cv.CreateMat (3, 3, cv.CV_8UC1)
TypeError: 'builtin_function_or_method' object does not support item assignment
Does this mean I cannot have matrices in lists?
That dereference is just creating a Matrix, without initialising its data. The data is manually set to zero by those lines like cvSet (* (kpw + i), cvRealScalar (0), NULL).
In python, you can just do the same thing in one hit with numpy.zeros and then use cv.fromarray. Alternatively, use x = cv.CreateMat(3, 3, cv.CV_8UC1) and then cv.set(x, 0.).
Edit - made a (pretty big) mistake in this answer, will explain
Looks like an array of CvMats in both kpw and kpb.
Suppose I made a list of arrays kpw = [] in Python.
The *(kpw + i) = ... is just like saying kpw[i] = ....
Looks like the other code initialising the list of kernels to 3x3 matrices of 0, so you could do:
# make a list of 8 3x3 matrices of 0.
kpw = []
for i in xrange(8):
kpw.append(np.zeros((3,3)))
kpb.append(np.zeros((3,3)))
Note: I previously had:
kpw = [np.zeros((3,3))] * 8
kpb = [np.zeros((3,3))] * 8
which is wrong ! It produces 8 references to the same matrix within kpw, and so modifying kpw[0] will also modify all the other kpw[i]!
Then the cvSet2D(*(kpb+0), 0, 0, cvRealScalar(0)); can be translated to :
kpb[0][0,0] = 0
Because *(kpb+0) grabs the matrix in kpg[0], the 0,0 means element 0,0 of the matrix, and 0 is the value.
So: every time you see *(kpb+i) just substitute kpb[i] and you should be find translating that code.
I made a new one in python. Thinning(Python)
Related
Hi I have some data and I want to compute the centered moving average or two-sided moving average.
I've understood how easy this can be done with the numpy.convolve function and I wonder if there is an easy or similar way in which this can be done, but when the average needs to be two-sided.
The one sided moving average usually works in the following way if the interval contains three entries, N = 3:
import numpy
list = [3, 4, 7, 8, 9, 10]
N = 3
window = numpy.repeat(1., N)/N
moving_avg = numpy.convolve(list, window, 'valid')
moving_avg = array([ 4.66666667, 6.33333333, 8. , 9. ])
Now what I am aiming to get is the average that is centered, so that if N = 3, the intervals over which the mean is taken are: [[3, 4, 7], [4, 7, 8], [7, 8, 9], [8, 9, 10]]. This is also tricky if N is an even number. Is there a tool to compute this? I'd prefer to do it either by writing a function or using numpy.
Like the commenters, I'm also confused what you're trying to accomplish that's different than the way you demonstrated.
In any case, I did want to offer a solution that lets you write your own convolution operations using Numba's #stencil decorator:
from numba import stencil
#stencil
def ma(a):
return (a[-1] + a[0] + a[1]) / 3
data = np.array([3, 4, 7, 8, 9, 10])
print(ma(data))
[0. 4.66666667 6.33333333 8. 9. 0. ]
Not sure if that's exactly what you're looking for, but the stencil operator is great. The variable you pass it represents a given element, and any indexing you use is relative to that element. As you can see, it was pretty easy to make a 3-element window to calculate a moving average.
Hopefully this gives you what you need.
Using a Large Neighborhood
You can add a parameter to the stencil, which is inclusive. Let's make a neighborhood of 9:
#stencil(neighborhood = ((-4, 4),))
def ma(a):
cumul = 0
for i in range(-4, 5):
cumul += a[i]
return cumul / 9
You can shift the range forward or back with (-8, 0) or (0, 8) and changing the range.
Setting N Neighborhood
Not sure if this is the best way, but I accomplished it with a wrapper:
def wrapper(data, N):
#nb.stencil(neighborhood = ((int(-(N-1)/2), int((N-1)/2)),))
def ma(a):
cumul = 0
for i in np.arange(int(-(N-1)/2), int((N-1)/2)+1):
cumul += a[i]
return cumul / N
return ma(data)
Again, indexing is weird, so you'll have to play with it to get the desired effect.
There is a common algorithm for solving the knapsack problem using dynamic programming. But it's not work for W=750000000, because there is an error of bad alloc. Any ideas how to solve this problem for my value of W?
int n=this->items.size();
std::vector<std::vector<uint64_t>> dps(this->W + 1, std::vector<uint64_t>(n + 1, 0));
for (int j = 1; j <= n; j++)
for (int k = 1; k <= this->W; k++) {
if (this->items[j - 1]->wts <= k)
dps[k][j] = std::max(dps[k][j - 1], dps[k - this->items[j - 1]->wts][j - 1] + this->items[j - 1]->cost);
else
dps[k][j] = dps[k][j - 1];
}
First of all, you can use only one dimension to solve the knapsack problem. This will reduce your memory from dp[W][n] (n*W space) to dp[W] (W space). You can look here: 0/1 Knapsack Dynamic Programming Optimazion, from 2D matrix to 1D matrix
But, even if you use only dp[W], your W is really high, and might be too much memory. If your items are big, you can use some approach to reduce the number of possible weights. First, realize that you don't need all positions of W, only those such that the sum of weight[i] exists.
For example:
W = 500
weights = [100, 200, 400]
You will never use position dp[473] of your matrix, because the items can occupy only positions p = [0, 100, 200, 300, 400, 500]. It is easy to see that this problem is the same as when:
W = 5
weights = [1,2,4]
Another more complicated example:
W = 20
weights = [5, 7, 8]
Using the same approach as before, you don't need all weights from 0 to 20, because the items can occupy only fill up to positions
p = [0, 5, 7, 5 + 7, 5 + 8, 7 + 8, 5 + 7 + 8]
p = [0, 5, 7, 12, 13, 15, 20]
, and you can reduce your matrix from dp[20] to dp[size of p] = M[7].
You do not show n, but even if we assume it is 1, lets see how much data you are trying to allocate. So, it would be:
W*64*2 // Here we don't consider overhead of the vector
This comes out to be:
750000000*64*2 bits = ~11.1758Gb
I am guessing this is more space then your program will allow. You are going to need to take a new approach. Perhaps try to handle the problem as multiple blocks. Consider the first and second half seperatley, then swap.
I want to perform a convolution on a big patch of the image. But I don't want too have to many variables, one solution could be to use the atrous_conv2d function but I would prefer apply first a max_pool on the patch, then the regular conv2d. How can I do this?
I have to keep the same image size between input and output. Here the code with the atrous_conv2d function
x = tf.placeholder('float', shape=[None, size_x*size_y])
image = tf.reshape(x, [-1,size_x , size_y,1])
W = weight_variable([9, 9, 1, n])
conv =tf.nn.atrous_conv2d(image, W, 10, padding='SAME')
If I understand correctly the patch size of the atrous_conv2d convolution is (9*10 X 9*10) but it act on each different pixel at 10 pixel interval and need only 9X9Xn variables.
I would prefer to take the same patch size, apply a max_pool on it, then a conventional conv2d on the (9X9) patch resulting from the max_pool. At the end it would produce the same numbers of variables but it could provide smoother results. The code could look like this :
x = tf.placeholder('float', shape=[None, size_x*size_y])
image = tf.reshape(x, [-1,size_x , size_y,1])
W = weight_variable([9, 9, 1, n])
def maxp(patch):
tf.sum_reduce(tf.nn.max_pool(patch, ksize=[1,10,10,1],
strides=[1,10,10,1], padding='SAME')*W)
conv=conv_func(image,maxp,patch_size=[1,9*10,9*10,1],strides=[1,1,1,1])
where conv_func take as argument the value, a function and a patch_size and apply the function on the patch.
I am a newbie in python and have a very basic knowledge of the language, having said that, I'm trying to get the visualization for all layers both for weights and their filters.For this instead of repeating:
# the parameters are a list of [weights, biases]
filters = net.params['conv1'][0].data
vis_square(filters.transpose(0, 2, 3, 1))
and changing layer name, I tried using a loop like this :
for layer_name, param in net.params.iteritems():
# the parameters are a list of [weights, biases]
filters = net.params[layer_name][0].data
vis_square(filters.transpose(0, 2, 3, 1))
now it works fine for the first layer, but gives this error and stops running:
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-16-cf7d5999a45c> in <module>()
2 # the parameters are a list of [weights, biases]
3 filters = net.params[layer_name][0].data
----> 4 vis_square(filters.transpose(0, 2, 3, 1))
ValueError: axes don't match array
And this is the definition of vis_square() (defined in classification.ipny in example directory of caffe):
def vis_square(data):
"""Take an array of shape (n, height, width) or (n, height, width, 3)
and visualize each (height, width) thing in a grid of size approx. sqrt(n) by sqrt(n)"""
# normalize data for display
data = (data - data.min()) / (data.max() - data.min())
# force the number of filters to be square
n = int(np.ceil(np.sqrt(data.shape[0])))
padding = (((0, n ** 2 - data.shape[0]),
(0, 1), (0, 1)) # add some space between filters
+ ((0, 0),) * (data.ndim - 3)) # don't pad the last dimension (if there is one)
data = np.pad(data, padding, mode='constant', constant_values=1) # pad with ones (white)
# tile the filters into an image
data = data.reshape((n, n) + data.shape[1:]).transpose((0, 2, 1, 3) + tuple(range(4, data.ndim + 1)))
data = data.reshape((n * data.shape[1], n * data.shape[3]) + data.shape[4:])
plt.imshow(data); plt.axis('off')
What is wrong here?
I'd be grateful if anyone could give me a hand on this.
For subsequent layers, the number of channels is > 64. For instance, if you have num_output: 64 in the first layer and num_output: 64 in the second as well, the shape of the 4D matrix that stores the weights is 64 x 64 x height x width. After you do the transpose, it's 64 x height x width x 64.
Your function is not capable of handling a 64 layer object, though it's great for 3-layer objects.
I would just do n = int(np.ceil(np.sqrt(data.shape[0] * data.shape[3]))) and reshape the whole thing into a 1-layer object. I don't think visualising the convolution kernel as RGB will give you any insight.
For anyone having similar problem ("axes don't match array" error): Right before transposing, I put my data in a variable giving the exact size. If my data is Data with the size of 10*12*15:
DataI = Data [0:9, 0:11, 0:14]
DataII = np.transpose(DataI,(0,2,1))
this worked for me.
I would like to try out the Mincemeat map/reduce Python application for matrix multiplication. I am using Python 2.7. I found several web pages that describe how to do matrix multiplication using Hadoop in Java, and I have been referring to this one http://importantfish.com/one-step-matrix-multiplication-with-hadoop/ both because it is simple and because the pseudocode that it displays is very close to Python code already.
I noticed in the Java code that is also included that the matrix dimensions are supplied to the map and reduce functions via an additional argument of type Context. Mincemeat doesn't provide such a thing, but I got a suggestion that I could provide these values to my map and reduce functions using closures. The map and reduce functions I wrote look like this:
def make_map_fn(num_rows_result, num_cols_result):
m = num_rows_result
p = num_cols_result
def map_fn(key, value):
# value is ('A', i, j, a_ij) or ('B', j, k, b_jk)
if value[0] == 'A':
i = value[1]
j = value[2]
a_ij = value[3]
for k in xrange(1, p):
yield ((i, k), ('A', j, a_ij))
else:
j = value[1]
k = value[2]
b_jk = value[3]
for i in xrange(1, m):
yield ((i, k), ('B', j, b_jk))
return map_fn
def make_reduce_fn(inner_dim):
n = inner_dim
def reduce_fn(key, values):
# key is (i, k)
# values is a list of ('A', j, a_ij) and ('B', j, b_jk)
hash_A = {j: a_ij for (x, j, a_ij) in values if x == 'A'}
hash_B = {j: b_jk for (x, j, b_jk) in values if x == 'B'}
result = 0
for j in xrange(1, n):
result += hash_A[j] * hash_B[j]
return (key, result)
return reduce_fn
Then I assign them to Mincemeat like this:
s = mincemeat.Server()
s.mapfn = make_map_fn(num_rows_A, num_cols_B)
s.reducefn = make_reduce_fn(num_cols_A)
When I run this in Mincemeat, I get this error message:
error: uncaptured python exception, closing channel <__main__.Client connected at 0x2ada4d0>
(<type 'exceptions.TypeError'>:arg 5 (closure) must be tuple
[/usr/lib/python2.7/asyncore.py|read|83]
[/usr/lib/python2.7/asyncore.py|handle_read_event|444]
[/usr/lib/python2.7/asynchat.py|handle_read|140]
[/usr/local/lib/python2.7/dist-packages/mincemeat.py|found_terminator|96]
[/usr/local/lib/python2.7/dist-packages/mincemeat.py|process_command|194]
[/usr/local/lib/python2.7/dist-packages/mincemeat.py|set_mapfn|159])
I searched around on the net with search terms like |python closure must be tuple| and the things that I found seemed to be dealing with cases where someone is trying to construct a function using lambda or function() and need to make sure they didn't omit certain things when defining them as closures. In my case, the map_fn and reduce_fn values returned by make_map_fn and make_reduce_fn look like valid function objects, their func_closure values are tuples of cells containing the array dimensions that I want to supply, but something is still missing. What form do I need to pass these functions in to be usable by Mincemeat?
I hate to be the bearer of bad news, but this is just the result of a few off-by-one errors in your code, plus two errors in the input file provided by the site you linked. It is unrelated to your usage of a closure, misleading error messages notwithstanding.
Off-by-one errors
Notice that the innermost loops in the pseudocode look like this:
for k = 1 to p:
for i = 1 to m:
for j = 1 to n:
In pseudocode, this typically indicates that the endpoint is included, i.e. for k = 1 to p means k = 1, 2, ..., p-1, p. On the other hand, the corresponding loops in your code look like this:
for k in xrange(1, p):
for i in xrange(1, m):
for j in xrange(1, n):
And of course, xrange(1, p) yields 1, 2, ..., p-2, p-1. Assuming you indexed the matrices from 0 (as they did on the site you linked), all your xranges should start at 0 (e.g. xrange(0, p)), as their equivalents in the Java code do (for (int k = 0; k < p; k++)). This fixes one of your problems.
Input file errors
In case you didn't catch this, the input file for A and B that the site provides is incorrect - they forgot the (0,0) entries of both matrices. In particular, you should add a line to the beginning of the form A,0,0,0.0, and a line between 9 and 10 of the form B,0,0,0.0. (I guess where exactly you put it doesn't matter, but for consistency, you may as well put them where they naturally fit.)
Once I correct these two errors, mincemeat gives me the result we expect (formatted):
{(0, 1): ((0, 1), 100.0),
(1, 2): ((1, 2), 310.0),
(0, 0): ((0, 0), 90.0),
(0, 2): ((0, 2), 110.0),
(1, 0): ((1, 0), 240.0),
(1, 1): ((1, 1), 275.0)}
I haven't figured out exactly what's going on with the error message, but I think it boils down to the fact that the incorrect loop indices in the map function are resulting in garbage data being passed to the reduce nodes, which is why the error mentions the reduce function.
Basically, what happens is that hash_A and hash_B in the reduce function sometimes don't have the same keys, so when you try to multiply hash_A[j] * hash_B[j], you'll get a KeyError because j is not a key of one or the other, and this gets caught somewhere upstream and rethrown as a TypeError instead.