Related
Hi I have some data and I want to compute the centered moving average or two-sided moving average.
I've understood how easy this can be done with the numpy.convolve function and I wonder if there is an easy or similar way in which this can be done, but when the average needs to be two-sided.
The one sided moving average usually works in the following way if the interval contains three entries, N = 3:
import numpy
list = [3, 4, 7, 8, 9, 10]
N = 3
window = numpy.repeat(1., N)/N
moving_avg = numpy.convolve(list, window, 'valid')
moving_avg = array([ 4.66666667, 6.33333333, 8. , 9. ])
Now what I am aiming to get is the average that is centered, so that if N = 3, the intervals over which the mean is taken are: [[3, 4, 7], [4, 7, 8], [7, 8, 9], [8, 9, 10]]. This is also tricky if N is an even number. Is there a tool to compute this? I'd prefer to do it either by writing a function or using numpy.
Like the commenters, I'm also confused what you're trying to accomplish that's different than the way you demonstrated.
In any case, I did want to offer a solution that lets you write your own convolution operations using Numba's #stencil decorator:
from numba import stencil
#stencil
def ma(a):
return (a[-1] + a[0] + a[1]) / 3
data = np.array([3, 4, 7, 8, 9, 10])
print(ma(data))
[0. 4.66666667 6.33333333 8. 9. 0. ]
Not sure if that's exactly what you're looking for, but the stencil operator is great. The variable you pass it represents a given element, and any indexing you use is relative to that element. As you can see, it was pretty easy to make a 3-element window to calculate a moving average.
Hopefully this gives you what you need.
Using a Large Neighborhood
You can add a parameter to the stencil, which is inclusive. Let's make a neighborhood of 9:
#stencil(neighborhood = ((-4, 4),))
def ma(a):
cumul = 0
for i in range(-4, 5):
cumul += a[i]
return cumul / 9
You can shift the range forward or back with (-8, 0) or (0, 8) and changing the range.
Setting N Neighborhood
Not sure if this is the best way, but I accomplished it with a wrapper:
def wrapper(data, N):
#nb.stencil(neighborhood = ((int(-(N-1)/2), int((N-1)/2)),))
def ma(a):
cumul = 0
for i in np.arange(int(-(N-1)/2), int((N-1)/2)+1):
cumul += a[i]
return cumul / N
return ma(data)
Again, indexing is weird, so you'll have to play with it to get the desired effect.
There is a common algorithm for solving the knapsack problem using dynamic programming. But it's not work for W=750000000, because there is an error of bad alloc. Any ideas how to solve this problem for my value of W?
int n=this->items.size();
std::vector<std::vector<uint64_t>> dps(this->W + 1, std::vector<uint64_t>(n + 1, 0));
for (int j = 1; j <= n; j++)
for (int k = 1; k <= this->W; k++) {
if (this->items[j - 1]->wts <= k)
dps[k][j] = std::max(dps[k][j - 1], dps[k - this->items[j - 1]->wts][j - 1] + this->items[j - 1]->cost);
else
dps[k][j] = dps[k][j - 1];
}
First of all, you can use only one dimension to solve the knapsack problem. This will reduce your memory from dp[W][n] (n*W space) to dp[W] (W space). You can look here: 0/1 Knapsack Dynamic Programming Optimazion, from 2D matrix to 1D matrix
But, even if you use only dp[W], your W is really high, and might be too much memory. If your items are big, you can use some approach to reduce the number of possible weights. First, realize that you don't need all positions of W, only those such that the sum of weight[i] exists.
For example:
W = 500
weights = [100, 200, 400]
You will never use position dp[473] of your matrix, because the items can occupy only positions p = [0, 100, 200, 300, 400, 500]. It is easy to see that this problem is the same as when:
W = 5
weights = [1,2,4]
Another more complicated example:
W = 20
weights = [5, 7, 8]
Using the same approach as before, you don't need all weights from 0 to 20, because the items can occupy only fill up to positions
p = [0, 5, 7, 5 + 7, 5 + 8, 7 + 8, 5 + 7 + 8]
p = [0, 5, 7, 12, 13, 15, 20]
, and you can reduce your matrix from dp[20] to dp[size of p] = M[7].
You do not show n, but even if we assume it is 1, lets see how much data you are trying to allocate. So, it would be:
W*64*2 // Here we don't consider overhead of the vector
This comes out to be:
750000000*64*2 bits = ~11.1758Gb
I am guessing this is more space then your program will allow. You are going to need to take a new approach. Perhaps try to handle the problem as multiple blocks. Consider the first and second half seperatley, then swap.
I am wondering how can I move first N elements from a list and put them at the end.
For example:
[1,2,3,4] and I want to move first 2 elements , so the result will be [3,4,1,2].
rule(List1,N,List2) :- length(List1,Y), ...
I don't know how to start, any advice ?
Since we are speaking of predicates - i.e. true relations among arguments - and Prolog library builtins are written with efficiency and generality in mind, you should know that - for instance - length/2 can generate a list, as well as 'measuring' its length, and append/3 can also split a list in two. Then, your task could be
'move first N elements to the end of the list'(L,N,R) :-
length(-,-),
append(-,-,-),
append(-,-,-).
Replace each dash with an appropriate variable, and you'll get
?- 'move first N elements to the end of the list'([1,2,3,4],2,R).
R = [3, 4, 1, 2].
You could opt to adopt a more general perspective on the task. If you think about it, taking the first N elements of a list and appending them at the end can be seen as a rotation to the left by N steps (just imagine the list elements arranged in a circle). The predicate name rotate/3 in #Willem Van Onsem's answer also indicates this perspective. You can actually define such a predicate as a true relation, that is making it work in all directions. Additionally it would be desirable to abstain from imposing unnecessary restrictions on the arguments while retaining nice termination properties. To reflect the relational nature of the predicate, let's choose a descriptive name. As the third argument is the left rotation by N steps of the list that is the first argument, let's maybe call it list_n_lrot/3 and define it like so:
:- use_module(library(lists)).
:- use_module(library(clpfd)).
list_n_lrot([],0,[]). % <- special case
list_n_lrot(L,N,LR) :-
list_list_samelength(L,LR,Len), % <- structural constraint
NMod #= N mod Len,
list_n_heads_tail(L,NMod,H,T),
append(T,H,LR).
list_list_samelength([],[],0).
list_list_samelength([_X|Xs],[_Y|Ys],N1) :-
N1 #> 0,
N0 #= N1-1,
list_list_samelength(Xs,Ys,N0).
list_n_heads_tail(L,N,H,T) :-
if_(N=0,(L=T,H=[]),
(N0#=N-1,L=[X|Xs],H=[X|Ys],list_n_heads_tail(Xs,N0,Ys,T))).
Now let's step through the definition and observe some of its effects by example. The first rule of list_n_lrot/3 is only included to deal with the special case of empty lists:
?- list_n_lrot([],N,R).
N = 0,
R = [] ;
false.
?- list_n_lrot(L,N,[]).
L = [],
N = 0 ;
false.
?- list_n_lrot(L,N,R).
L = R, R = [],
N = 0 ;
...
If you don't want to include these cases in your solution just omit that rule. Throughout the predicates CLP(FD) is used for arithmetic constraints, so the second argument of list_n_lrot/3 can be variable without leading to instantiation errors. The goal list_list_samelength/2 is a structural constraint to ensure the two lists are of same length. This helps avoiding an infinite loop after producing all answers in the case that only the third argument is ground (to see this, remove the first two goals of list_n_lrot/3 and replace the third with list_n_heads_tail(L,N,H,T) and then try the query ?- list_n_lrot(L,N,[1,2,3]).). It's also the reason why the most general query is listing the solutions in a fair order, that is producing all possibilities for every list length instead of only listing the rotation by 0 steps:
?- list_n_lrot(L,N,R).
... % <- first solutions
L = R, R = [_G2142, _G2145, _G2148], % <- length 3, rotation by 0 steps
N mod 3#=0 ;
L = [_G2502, _G2505, _G2508], % <- length 3, rotation by 1 step
R = [_G2505, _G2508, _G2502],
N mod 3#=1 ;
L = [_G2865, _G2868, _G2871], % <- length 3, rotation by 2 steps
R = [_G2871, _G2865, _G2868],
N mod 3#=2 ;
... % <- further solutions
Finally, it also describes the actual length of the two lists, which is used in the next goal to determine the remainder of N modulo the length of the list. Consider the following: If you rotate a list of length N by N steps you arrive at the initial list again. So a rotation by N+1 steps yields the same list as a rotation by 1 step. Algebraically speaking, this goal is exploiting the fact that congruence modulo N is partitioning the infinite set of integers into a finite number of residue classes. So for a list of length N it is sufficient to produce the N rotations that correspond to the N residue classes in order to cover all possible rotations (see the query above for N=3). On the other hand, a given N > list length can be easily computed by taking the smallest non-negative member of its residue class. For example, given a list with three elements, a rotation by 2 or 5 steps respectively yields the same result:
?- list_n_lrot([1,2,3],2,R).
R = [3, 1, 2].
?- list_n_lrot([1,2,3],5,R).
R = [3, 1, 2].
And of course you could also left rotate the list by a negative number of steps, that is rotating it in the other direction:
?- list_n_lrot([1,2,3],-1,R).
R = [3, 1, 2].
On a side note: Since this constitutes rotation to the right, you could easily define right rotation by simply writing:
list_n_rrot(L,N,R) :-
list_n_lrot(L,-N,R).
?- list_n_rrot([1,2,3],1,R).
R = [3, 1, 2].
The predicate list_n_heads_tail/4 is quite similar to splitAt/4 in Willem's post. However, due to the use of if_/3 the predicate succeeds deterministically (no need to hit ; after the only answer since no unnecessary choicepoints are left open), if one of the lists and the second argument of list_n_lrot/3 are ground:
?- list_n_lrot2([a,b,c,d,e],2,R).
R = [c, d, e, a, b].
?- list_n_lrot2(L,2,[c,d,e,a,b]).
L = [a, b, c, d, e].
You can observe another nice effect of using CLP(FD) with the second solution of the most general query:
?- list_n_lrot(L,N,R).
L = R, R = [],
N = 0 ;
L = R, R = [_G125], % <- here
N in inf..sup ; % <- here
...
This answer states, that for a list with one element any rotation by an arbitrary number of steps yields the very same list again. So in principle, this single general answer summarizes an infinite number of concrete answers. Furthermore, you can also ask questions like: What lists are there with regard to a rotation by 2 steps?
?- list_n_lrot2(L,2,R).
L = R, R = [_G19] ;
L = R, R = [_G19, _G54] ;
L = [_G19, _G54, _G22],
R = [_G22, _G19, _G54] ;
...
To finally come back to the example in your question:
?- list_n_lrot([1,2,3,4],2,R).
R = [3, 4, 1, 2].
Note how this more general approach to define arbitrary rotations on lists subsumes your use case of relocating the first N elements to the end of the list.
Try this
despI([C|B],R):-append(B,[C|[]],R).
desp(A,0,A).
desp([C|B],N,R):-N1 is N - 1, despI([C|B],R1), desp(R1,N1,R),!.
The first predicate moves one element to the end of the list, then the only thing I do is "repeat" that N times.
?-de([1,2,3,4],2,R).
R = [3, 4, 1, 2].
?- de([1,2,3,4,5,6,7],4,R).
R = [5, 6, 7, 1, 2, 3, 4].
We can do this with a predicate that works in two phases:
a collect phase: we collect the first N items of the list; and
an emit phase: we construct a list where we add these elements at the tail.
Let is construct the two phases with separate predicate. For the collect phase, we could use the following predicate:
% splitAt(L,N,L1,L2).
splitAt(L,0,[],L).
splitAt([H|T],N,[H|T1],L2) :-
N > 0,
N1 is N-1,
splitAt(T,N1,T1,L2).
Now for the emit phase, we could use append/3. So then the full predicate is:
rotate(L,N,R) :-
splitAt(L,N,L1,L2),
append(L2,L1,R).
This gives:
?- rotate([1,2,3,4],0,R).
R = [1, 2, 3, 4] .
?- rotate([1,2,3,4],1,R).
R = [2, 3, 4, 1] .
?- rotate([1,2,3,4],2,R).
R = [3, 4, 1, 2] .
?- rotate([1,2,3,4],3,R).
R = [4, 1, 2, 3] .
?- rotate([1,2,3,4],4,R).
R = [1, 2, 3, 4].
The algorithm works in O(n).
I have recently been trying to figure out Prolog and been messing with lists of lists in Prolog. I am trying to create a sort of mask I suppose in p
Prolog. I have a predicate that determines the difference between two lists of lists (L1 and L2 lets say) in Prolog and saves them as a list of a list(Lets say R). I have another predicate that simply states if the difference is equal to zero(noDifference). I would like to have two resulting lists of lists (M1 and M2) based off of L1 and L2 compared to the R. For example I would like to compare all values of L1 and L2 to R, if a negative value is at a location of R then the value in the same location of L1 is saved into M1. And if a positive value is at a location of R then the value in the same location of L2 is saved into M2 if that makes sense. Before all of this I check with my noDifference function to see if the difference is 0 and if so all values of M1 and M2's lists of lists will be 0.
This is what I have so far(I'm not sure if I started it right)
masker(L1,L2,R,M1,M2):- noDifference(R1), M1=R, M2=R1;
and for the rest of it here are what some example values should look like under the hood
L1=[[1,5,3,8],[1,5,3,8]]
L2=[[5,4,7,4],[5,4,7,4]]
R=[[4,-1,4,-4],[4,-1,4,-4]]
M1=[[0,5,0,8],[0,5,0,8]]Neg values of L1 at R are stored rest are 0)
M2=[[5,0,7,0],[5,0,7,0]](Pos values of L2 at R are stored rest are 0)
Any insight if what I am doing so far is right and how to properly formulate the subgoals/where I should go next would be awesome!
edit with ex predicate
?- masker([[1,5,3,8],[1,5,3,8]],
[[5,4,7,4],[5,4,7,4]],
[[4,-1,4,-4],[4,-1,4,-4]], M1, M2).
M1=[[0,5,0,8],[0,5,0,8]].
M2=[[5,0,7,0],[5,0,7,0]].
Think what your predicate should describe. It is a relation between five lists of lists which, according to the example you provided, are of same length. This suggests the base case with five empty lists. Otherwise the heads of all five lists are lists themselves, that are in a specific relation to each other, let's call it lists_mask_mlists/5. And of course the same should be true for the tails, which can be realized by a recursive goal. So your predicate masker/5 could look something like that:
masker([],[],[],[],[]).
masker([X|Xs],[Y|Ys],[M|Ms],[R1|R1s],[R2|R2s]) :-
lists_mask_mlists(X,Y,M,R1,R2),
masker(Xs,Ys,Ms,R1s,R2s).
The actual masking relation also has a base case with five empty lists. Otherwise there are two further cases:
1) The current masking element (head of the third list) is negative: The head of the first list is the head of the fourth list and the head of the fifth list is 0
2) The current masking element is positive: The head of the second list is the head of the fifth list and the head of the fourth list is 0
You can express that like so:
lists_mask_mlists([],[],[],[],[]).
lists_mask_mlists([X|Xs],[_Y|Ys],[M|Ms],[X|R1s],[0|R2s]) :- % 1)
M < 0,
lists_mask_mlists(Xs,Ys,Ms,R1s,R2s).
lists_mask_mlists([_X|Xs],[Y|Ys],[M|Ms],[0|R1s],[Y|R2s]) :- % 2)
M >= 0,
lists_mask_mlists(Xs,Ys,Ms,R1s,R2s).
With this predicate your example query yields the desired result:
?- masker([[1,5,3,8],[1,5,3,8]],[[5,4,7,4],[5,4,7,4]],[[4,-1,4,-4],[4,-1,4,-4]],M1,M2).
M1 = [[0,5,0,8],[0,5,0,8]],
M2 = [[5,0,7,0],[5,0,7,0]] ? ;
no
Note however, that due to < and >= this only works, if the third list is variable free. Replacing the first 4 in the third argument by a variable yields an instantiation error:
?- masker([[1,5,3,8],[1,5,3,8]],[[5,4,7,4],[5,4,7,4]],[[X,-1,4,-4],[4,-1,4,-4]],M1,M2).
ERROR at clause 2 of user:masked/5 !!
INSTANTIATION ERROR- =:=/2: expected bound value
If you intend to use the predicate with a third argument that is not variable free, you might like to consider using clpfd. Include the line
:-use_module(library(clpfd)).
in your source file and alter lists_mask_mlists/5 like so:
lists_mask_mlists([],[],[],[],[]).
lists_mask_mlists([X|Xs],[_Y|Ys],[M|Ms],[X|R1s],[0|R2s]) :-
M #< 0, % <- here
lists_mask_mlists(Xs,Ys,Ms,R1s,R2s).
lists_mask_mlists([_X|Xs],[Y|Ys],[M|Ms],[0|R1s],[Y|R2s]) :-
M #>= 0, % <- here
lists_mask_mlists(Xs,Ys,Ms,R1s,R2s).
Now the second query works too:
?- masker([[1,5,3,8],[1,5,3,8]],[[5,4,7,4],[5,4,7,4]],[[X,-1,4,-4],[4,-1,4,-4]],M1,M2).
M1 = [[1,5,0,8],[0,5,0,8]],
M2 = [[0,0,7,0],[5,0,7,0]],
X in inf.. -1 ? ;
M1 = [[0,5,0,8],[0,5,0,8]],
M2 = [[5,0,7,0],[5,0,7,0]],
X in 0..sup ? ;
no
#tas has presented a good solution and explanation (+1).
Building on this code, I would like to improve the space efficiency of the program. Consider again the example query with the CLP(FD) based solution:
?- masker([[1,5,3,8],[1,5,3,8]],[[5,4,7,4],[5,4,7,4]],[[4,-1,4,-4],[4,-1,4,-4]],M1,M2).
M1 = [[0, 5, 0, 8], [0, 5, 0, 8]],
M2 = [[5, 0, 7, 0], [5, 0, 7, 0]] ;
false.
We see from the ; false that choice points were accumulated during the execution of this program, because it was not apparent to the Prolog engine that the clauses were in fact mutually exclusive.
Obviously, such programs use more memory than necessary to keep track of remaining choices.
Resist the temptation to impurely cut away branches of the computation, because that will only lead to even more problems.
Instead, consider using if_/3.
All it takes to apply if_/3 in this case is a very easy reification of the CLP(FD) constraint (#<)/2, which is easy to do with zcompare/3:
#<(X, Y, T) :-
zcompare(C, X, Y),
less_true(C, T).
less_true(<, true).
less_true(>, false).
less_true(=, false).
With this definition, the whole program becomes:
:- use_module(library(clpfd)).
masker([], [], [], [], []).
masker([X|Xs], [Y|Ys], [M|Ms], [R1|R1s], [R2|R2s]) :-
lists_mask_mlists(X, Y, M, R1, R2),
masker(Xs, Ys, Ms, R1s, R2s).
lists_mask_mlists([], [], [], [], []).
lists_mask_mlists([X|Xs], [Y|Ys], [M|Ms], R1s0, R2s0) :-
if_(M #< 0,
(R1s0 = [X|R1s], R2s0 = [0|R2s]),
(R1s0 = [0|R1s], R2s0 = [Y|R2s])),
lists_mask_mlists(Xs, Ys, Ms, R1s, R2s).
And now the point:
?- masker([[1,5,3,8],[1,5,3,8]],[[5,4,7,4],[5,4,7,4]],[[4,-1,4,-4],[4,-1,4,-4]],M1,M2).
M1 = [[0, 5, 0, 8], [0, 5, 0, 8]],
M2 = [[5, 0, 7, 0], [5, 0, 7, 0]].
This example query is now deterministic!
And the program is still general enough to also handle the second example query!
I have a (i x j) sized matrix mat which contains values from an experiment.
If I use ListPlot3D[mat] I can visualise this in a 3D plot.
I also have two arrays of size i (aRow) and of size j (aCol) which I determined from my experiment.
How do I replace the default x and y axis shown with ListPlot3D[mat] with aRow and aCol?
Please take a look at the Ticks option and it example uses in the docs.
Here's one way to do it. First generate some example data:
mat = Table[Exp[-(x^2 + y^2)], {x, -2, 2, .1}, {y, -2, 2, .1}];
aCol = aRow = Round[mat[[20]], 0.01];
Plot it in 3D. I chose to show every 10th tick mark out of all possible ones. list[[;; ;; 10]] selects every 10th element of a list.
ListPlot3D[mat,
Ticks -> {
Transpose[{Range#Length[aRow], aRow}][[;; ;; 10]],
Transpose[{Range#Length[aCol], aCol}][[;; ;; 10]],
Automatic}]
Plot it in 2D too. ListDensityPlot has a Frame (not Axes) by default, so we use FrameTicks
ListDensityPlot[mat,
FrameTicks -> {
Transpose[{Range#Length[aRow], aRow}][[;; ;; 10]],
Transpose[{Range#Length[aCol], aCol}][[;; ;; 10]],
None, None},
Mesh -> Automatic]
Update
If you don't need arbitrary ticks, just a different range for the usual, linearly spaces tick marks, then you can use the DataRange option like this:
ListPlot3D[mat, DataRange -> {{0, 1}, {0, 1}}]
If you still need the data in an {x,y,z} format (because the coordinates are not evenly spaced), you can build it using
Join ## MapThread[Append, {Outer[List, aRow, aCol], mat}, 2]
If the differences between consecutive elements in aRow and bRow are constant, you could do something like
ListPlot3D[mat, DataRange -> (Through[{Min, Max}[#]] & /# {aCol, aRow})]
If not then you could create a list with elements {aCol[[i]], aRow[[j]], mat[[j,i]]} and plot that. There are different ways to do this, for example
list = Flatten[Table[{aCol[[i]], aRow[[j]], mat[[j, i]]},
{i, Length[aCol]}, {j, Length[aRow]}], 1];
ListPlot3D[list]
Edit
A faster way to create list is to do something like
list = ConstantArray[0, {Length[aCol] Length[aRow], 3}];
list[[All, {2, 1}]] = Tuples[{aRow, aCol}];
list[[All, 3]] = Flatten[mat];