I have the following MPI_AllToAllv call. All of the variables are vectors
MPI_Alltoallv(
&elements[0],
&send_counts[0],
&send_displacements[0],
MPI_INT,
&receiving_vector[0],
&receiving_counts[0],
&receiving_displacements[0],
MPI_INT,
MPI_COMM_WORLD
);
Here are the contents of the vectors:
Elements : [6, 5, 4, ]
# 0
Elements : [3, 2, 1, ]
# 1
send_counts : [3, 0, ]
# 1
send_displacements : [0, 3, ]
# 1
receiving_vector : [0, 0, 0, ]
# 0
elements : [6, 5, 4, ]
# 0
send_counts : [0, 3, ]
# 0
send_displacements : [0, 0, ]
# 0
receiving_vector : [0, 0, 0, ]
# 1
receiving_counts : [0, 3, ]
# 1
receiving_displacements : [0, 0, ]
# 1
[lawn-143-215-98-238:1182] *** An error occurred in MPI_Alltoallv
[lawn-143-215-98-238:1182] *** reported by process [2332229633,0]
[lawn-143-215-98-238:1182] *** on communicator MPI_COMM_WORLD
[lawn-143-215-98-238:1182] *** MPI_ERR_TRUNCATE: message truncated
[lawn-143-215-98-238:1182] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[lawn-143-215-98-238:1182] *** and potentially your MPI job)
receiving_counts : [3, 0, ]
# 0
receiving_displacements : [0, 0, ]
# 0
I don't understand why I am getting this error. Any help would be deeply appreciated.
I have googled this error and it's probably my receiving vector's size but I have tried many sizes and haven't gotten anywhere.
There is a mismatch in the amount of data send and the amount of data received. Since you only have two ranks, it is easy to draw a table of who send how much and to whom. Each row of the table is the content of send_counts[] at the corresponding rank:
receiver
s | 0 | 1 |
e ---+---+---+
n 0 | 0 | 3 | (send_counts[] # 0)
d ---+---+---+
e 1 | 3 | 0 | (send_counts[] # 1)
r ---+---+---+
To match the amount of data sent, the receive counts at each rank should be equal to the column-vector from the table above that corresponds to that rank:
receiving_counts[] # 0 should be { 0, 3 } while you have [3, 0, ];
receiving_counts[] # 1 should be { 3, 0 } while you have [0, 3, ].
Hence the truncation error.
Related
In Raku, how does one write the equivalent of Haskell's span function?
In Haskell, given a predicate and a list, one can split the list into two parts:
the longest prefix of elements satisfying the predicate
the remainder of the list
For example, the Haskell expression …
span (< 10) [2, 2, 2, 5, 5, 7, 13, 9, 6, 2, 20, 4]
… evaluates to …
([2,2,2,5,5,7],[13,9,6,2,20,4])
How does one write the Raku equivalent of Haskell's span function?
Update 1
Based on the answer of #chenyf, I developed the following span subroutine (additional later update reflects negated predicate within span required to remain faithful to the positive logic of Haskell's span function) …
sub span( &predicate, #numberList )
{
my &negatedPredicate = { ! &predicate($^x) } ;
my $idx = #numberList.first(&negatedPredicate):k ;
my #lst is Array[List] = #numberList[0..$idx-1], #numberList[$idx..*] ;
#lst ;
} # end sub span
sub MAIN()
{
my &myPredicate = { $_ <= 10 } ;
my #myNumberList is Array[Int] = [2, 2, 2, 5, 5, 7, 13, 9, 6, 2, 20, 4] ;
my #result is Array[List] = span( &myPredicate, #myNumberList ) ;
say '#result is ...' ;
say #result ;
say '#result[0] is ...' ;
say #result[0] ;
say #result[0].WHAT ;
say '#result[1] is ...' ;
say #result[1] ;
say #result[1].WHAT ;
} # end sub MAIN
Program output is …
#result is ...
[(2 2 2 5 5 7) (13 9 6 2 20 4)]
#result[0] is ...
(2 2 2 5 5 7)
(List)
#result[1] is ...
(13 9 6 2 20 4)
(List)
Update 2
Utilizing information posted to StackOverflow concerning Raku's Nil, the following updated draft of subroutine span is …
sub span( &predicate, #numberList )
{
my &negatedPredicate = { ! &predicate($^x) } ;
my $idx = #numberList.first( &negatedPredicate ):k ;
if Nil ~~ any($idx) { $idx = #numberList.elems ; }
my List $returnList = (#numberList[0..$idx-1], #numberList[$idx..*]) ;
$returnList ;
} # end sub span
sub MAIN()
{
say span( { $_ == 0 }, [2, 2, 5, 7, 4, 0] ) ; # (() (2 2 5 7 4 0))
say span( { $_ < 6 }, (2, 2, 5, 7, 4, 0) ) ; # ((2 2 5) (7 4 0))
say span( { $_ != 9 }, [2, 2, 5, 7, 4, 0] ) ; # ((2 2 5 7 4 0) ())
} # end sub MAIN
I use first method and :k adverb, like this:
my #num = [2, 2, 2, 5, 5, 7, 13, 9, 6, 2, 20, 4];
my $idx = #num.first(* > 10):k;
#num[0..$idx-1], #num[$idx..*];
A completely naive take on this:
sub split_on(#arr, &pred) {
my #arr1;
my #arr2 = #arr;
loop {
if not &pred(#arr2.first) {
last;
}
push #arr1: #arr2.shift
}
(#arr1, #arr2);
}
Create a new #arr1 and copy the array into #arr2. Loop, and if the predicate is not met for the first element in the array, it's the last time through. Otherwise, shift the first element off from #arr2 and push it onto #arr1.
When testing this:
my #a = [2, 2, 2, 5, 5, 7, 13, 9, 6, 2, 20, 4];
my #b = split_on #a, -> $x { $x < 10 };
say #b;
The output is:
[[2 2 2 5 5 7] [13 9 6 2 20 4]]
Only problem here is... what if the predicate isn't met? Well, let's check if the list is empty or the predicate isn't met to terminate the loop.
sub split_on(#arr, &pred) {
my #arr1;
my #arr2 = #arr;
loop {
if !#arr2 || not &pred(#arr2.first) {
last;
}
push #arr1: #arr2.shift;
}
(#arr1, #arr2);
}
So I figured I'd throw my version in because I thought that classify could be helpful :
sub span( &predicate, #list ) {
#list
.classify({
state $f = True;
$f &&= &predicate($_);
$f.Int;
}){1,0}
.map( {$_ // []} )
}
The map at the end is to handle the situation where either the predicate is never or always true.
In his presentation 105 C++ Algorithms in 1 line* of Raku (*each) Daniel Sockwell discusses a function that almost answers your question. I've refactored it a bit to fit your question, but the changes are minor.
#| Return the index at which the list splits given a predicate.
sub partition-point(&p, #xs) {
my \zt = #xs.&{ $_ Z .skip };
my \mm = zt.map({ &p(.[0]) and !&p(.[1]) });
my \nn = mm <<&&>> #xs.keys;
return nn.first(?*)
}
#| Given a predicate p and a list xs, returns a tuple where first element is
#| longest prefix (possibly empty) of xs of elements that satisfy p and second
#| element is the remainder of the list.
sub span(&p, #xs) {
my \idx = partition-point &p, #xs;
idx.defined ?? (#xs[0..idx], #xs[idx^..*]) !! ([], #xs)
}
my #a = 2, 2, 2, 5, 5, 7, 13, 9, 6, 2, 20, 4;
say span { $_ < 10 }, #a; #=> ((2 2 2 5 5 7) (13 9 6 2 20 4))
say span { $_ < 5 }, [6, 7, 8, 1, 2, 3]; #=> ([] [6 7 8 1 2 3])
Version 6.e of raku will sport the new 'snip' function:
use v6.e;
dd (^10).snip( * < 5 );
#«((0, 1, 2, 3, 4), (5, 6, 7, 8, 9)).Seq»
All
I'm trying to simulate a 100 users (breed) each of them having users-own attributes which are tracked over various two runs of ticks say 0 & 1. I store these values as a list within a list and sorted them. They appear like this [[tick 0, user 2, attrib 1 ... attrib 9] [tick 1, user 2, attrib 1 ... attrib 9] [tick 0, user 3, attrib 1 ... attrib 9] [tick 1, user 3, attrib 1 ... attrib 9] ... [tick 0, user 99, attrib 1 ... attrib 9] [tick 1, user 99, attrib 1 ... attrib 9] ]
As I randomly 'ask users' to reassign certain attributes, how do I get the subset of matching values that matches the current user referred by 'who'? for example if the current user in context is 'user 3', how do I get the sublist of the matching entries from the list of lists?
Once I obtain the sublist matching 'user 3'[tick 0, user 3, attrib 1 ... attrib 9] [tick 1, user 3, attrib 1 ... attrib 9], how do I get the index of each line & the items within that list? for example if I need to access the last item on the sublist (corresponding to tick 1) so that I can obtain the value of attrib 9 to process it further?
Thanks in advance for your help!
if your list is structured: [ [ TICK# TURTLE# ATT1 ... ATT9 ] ... ]
[
[ 0 0 1 .. 9 ]
[ 0 1 1 .. 9 ]
[ 0 2 1 .. 9 ]
...
[ 0 99 1 .. 9 ]
[ 1 0 1 .. 9 ]
[ 1 1 1 .. 0 ]
... and so on ...
And the number of turtles is always CONSTANT
Then you can calculate the index of the item you need.
;; attribute data is in list "data"
LET total-turtles COUNT TURTLES
LET desired-tick 1 ;; number of the tick
LET desired-turtle 7 ;; WHO of turtle
LET desired-attribute 4
;; calculate the sublist index
LET sublist-index desired-tick * total-turtles + desired-turtle
;; get the sublist
LET attribute-sublist ITEM sublist-index data
;; get the value from the sublist
LET attribute-value ITEM (desired-attribute + 2) attribute-sublist
If you set desired-turtle to [WHO] OF ONE-OF TURTLES you are working with a random turtle. Or you can ASK a random turtle to do the above, and it can just use its own WHO.
If you add the info to the list strictly in order, then you don't need to store the tick# or who#, and you don't need to sort it.
If you let the turtle store its own list of prior attributes, it might be even easier to store and access, depending on your application.
Now you have the pieces needed for a random turtle to access the history list and grab the values from its own history, or some other turtle's history.
How do I print a line from a 2-dimensional mixed array (containing integers and floating data types) without the square brackets and the spacing in the floating types and decimal point in integers?
I want to use:
for line in xd:
print line,
to get the final output
The code i tried is as follows:
import numpy
x =[[1.456, 2, 3],
[4, 5.231, 6],
[7, 8, 9.145]]
x=numpy.array(x)
xd=numpy.array2string(x, separator='\t')
for line in xd:
print line,
This is the output from the code
[ [ 1 . 4 5 6 2 . 3 . ]
[ 4 . 5 . 2 3 1 6 . ]
[ 7 . 8 . 9 . 1 4 5 ] ]
Your x is a list of lists
In [14]: x =[[1.456, 2, 3],
...: [4, 5.231, 6],
...: [7, 8, 9.145]]
In [15]: x
Out[15]: [[1.456, 2, 3], [4, 5.231, 6], [7, 8, 9.145]]
In [16]: for row in x:
...: print(row)
...:
[1.456, 2, 3]
[4, 5.231, 6]
[7, 8, 9.145]
We could set up a Python formatted print expression for each row.
But first look at what happens when you make an array:
In [17]: arr=np.array(x)
In [18]: arr
Out[18]:
array([[ 1.456, 2. , 3. ],
[ 4. , 5.231, 6. ],
[ 7. , 8. , 9.145]])
This is a 2d array - of the dtype float, the dtype that can hold both the ints and the floats. Formally you've lost the distinction between float and ints.
Trying to display this array without decimals and without [] is a lot harder than for the original list. The array formatting puts a lot of effort into lining up the columns, in other words making a pretty table like display. If the array is large lines will wrap, and it will start to add ellipsis. So from a display stand point, you actually loose control when making an array.
np.savetxt may help; it can be used to write the rows without [];
Drawing inspiration from savetxt (and how it formats rows)
In [21]: for row in x:
...: print('%6s %6s %6s'%tuple(row))
...:
1.456 2 3
4 5.231 6
7 8 9.145
In [22]: for row in arr:
...: print('%6s %6s %6s'%tuple(row))
...:
1.456 2.0 3.0
4.0 5.231 6.0
7.0 8.0 9.145
So if you are picky about the format of the numbers, stick with the list of lists, and study up on the Python formatting system (whether the % or .format version). For example %g works with the array as well as with the list
In [29]: for row in arr:
...: print('%6.2g %6.3g %6g'%tuple(row))
...:
1.5 2 3
4 5.23 6
7 8 9.145
Here is my minimal working example:
list1 = [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20] #len = 21
list2 = [1,1,1,0,1,0,0,1,0,1,1,0,1,0,1,0,0,0,1,1,0] #len = 21
list3 = [0,0,1,0,1,1,0,1,0,1,0,1,1,1,0,1,0,1,1,1,1] #len = 21
list4 = [1,0,0,1,1,0,0,0,0,1,0,1,1,1,1,0,1,0,1,0,1] #len = 21
I have four lists and I want to "clean" my list 1 using the following rule: "if any of list2[i] or list3[i] or list4[i] are equal to zero, then I want to eliminate the item I from list1. SO basically I only keep those elements of list1 such that the other lists all have ones there.
here is the function I wrote to solve this
def clean(list1, list2,list3,list4):
for i in range(len(list2)):
if (list2[i]==0 or list3[i]==0 or list4[i]==0):
list1.pop(i)
return list1
however it doesn't work. If you apply it, it give the error
Traceback (most recent call last):line 68, in clean list1.pop(I)
IndexError: pop index out of range
What am I doing wrong? Also, I was told Pandas is really good in dealing with data. Is there a way I can do it with Pandas? Each of these lists are actually columns (after removing the heading) of a csv file.
EDIT
For example at the end I would like to get: list1 = [4,9,11,15]
I think the main problem is that at each iteration, when I pop out the elements, the index of all the successor of that element change! And also, the overall length of the list changes, and so the index in pop() is too large. So hopefully there is another strategy or function that I can use
This is definitely a job for pandas:
import pandas as pd
df = pd.DataFrame({
'l1':list1,
'l2':list2,
'l3':list3,
'l4':list4
})
no_zeroes = df.loc[(df['l2'] != 0) & (df['l3'] != 0) & (df['l4'] != 0)]
Where df.loc[...] takes the full dataframe, then filters it by the criteria provided. In this example, your criteria are that you only keep the items where l2, l3, and l3 are not zero (!= 0).
Gives you a pandas dataframe:
l1 l2 l3 l4
4 4 1 1 1
9 9 1 1 1
12 12 1 1 1
18 18 1 1 1
or if you need just list1:
list1 = df['l1'].tolist()
if you want the criteria to be where all other columns are 1, then use:
all_ones = df.loc[(df['l2'] == 1) & (df['l3'] == 1) & (df['l4'] == 1)]
Note that I'm creating new dataframes for no_zeroes and all_ones and that the original dataframe stays intact if you want to further manipulate the data.
Update:
Per Divakar's answer (far more elegant than my original answer), much the same can be done in pandas:
df = pd.DataFrame([list1, list2, list3, list4])
list1 = df.loc[0, (df[1:] != 0).all()].astype(int).tolist()
Here's one approach with NumPy -
import numpy as np
mask = (np.asarray(list2)==1) & (np.asarray(list3)==1) & (np.asarray(list4)==1)
out = np.asarray(list1)[mask].tolist()
Here's another way with NumPy that stacks those lists into rows to form a 2D array and thus simplifies things quite a bit -
arr = np.vstack((list1, list2, list3, list4))
out = arr[0,(arr[1:] == 1).all(0)].tolist()
Sample run -
In [165]: arr = np.vstack((list1, list2, list3, list4))
In [166]: print arr
[[ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20]
[ 1 1 1 0 1 0 0 1 0 1 1 0 1 0 1 0 0 0 1 1 0]
[ 0 0 1 0 1 1 0 1 0 1 0 1 1 1 0 1 0 1 1 1 1]
[ 1 0 0 1 1 0 0 0 0 1 0 1 1 1 1 0 1 0 1 0 1]]
In [167]: arr[0,(arr[1:] == 1).all(0)].tolist()
Out[167]: [4, 9, 12, 18]
I have a data frame that is production data for a factory. The factory is organised into lines. The structure of the data is such that one of the columns contains repeating values that properly thought of are headers. I need to reshape the data. So in the following DataFrame the 'Quality' column contains 4 measures, that are then measured for each hour. Clearly this gives us four observations per line.
The goal here is to transpose this data, but such that some of the columns are single index and some are multi index. The row index should remain ['Date', 'ID']. The single index columns should be 'line_no', 'floor', 'buyer' and the multi index columns should be the hourly measures for each of the quality measures.
I know that this is possible because I accidentally stumbled across the way to do it. Basically as my code will show, I put everything in the index except the hourly data and then unstacked the quality column from the index. Then by chance, I tried to reset the index and it created this amazing dataframe where some columns were single index and some multi. Of course its highly impractical to have loads of columns in the index, because we might want to do stuff with them, like change them. My question is how to achieve this type of thing without having to go through this (what I feel is a) workaraound.
import random
import pandas as pd
d = {'ID' : [1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3] * 2,
'Date' : ['2013-05-04' for x in xrange(12)] + \
['2013-05-06' for x in xrange(12)],
'line_no' : [1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3] * 2,
'floor' : [5, 5, 5, 5, 6, 6, 6, 6, 5, 5, 5, 5] * 2,
'buyer' : ['buyer1', 'buyer1', 'buyer1', 'buyer1',\
'buyer2', 'buyer2', 'buyer2', 'buyer2',\
'buyer1', 'buyer1', 'buyer1', 'buyer1'] * 2,
'Quality' : ['no_checked', 'good', 'alter', 'rejected'] * 6,
'Hour1' : [random.randint(1000, 15000) for x in xrange(24)],
'Hour2' : [random.randint(1000, 15000) for x in xrange(24)],
'Hour3' : [random.randint(1000, 15000) for x in xrange(24)],
'Hour4' : [random.randint(1000, 15000) for x in xrange(24)],
'Hour5' : [random.randint(1000, 15000) for x in xrange(24)],
'Hour6' : [random.randint(1000, 15000) for x in xrange(24)]}
DF = pd.DataFrame(d, columns = ['ID', 'Date', 'line_no', 'floor', 'buyer',
'Quality', 'Hour1', 'Hour2', 'Hour3', 'Hour4',
'Hour5', 'Hour6'])
DF.set_index(['Date', 'ID'])
So this is how I achieved what I wanted, but there must be a way to do this without having to go through all these steps. Help please...
# Reset the index
DF.reset_index(inplace = True)
# Put everything in the index
DF.set_index(['Date', 'ID', 'line_no', 'floor', 'buyer', 'Quality'], inplace = True)
# Unstack Quality
DFS = DF.unstack('Quality')
#Now this was the accidental workaround - gives exactly the result I want
DFS.reset_index(inplace = True)
DFS.set_index(['Date', 'ID'], inplace = True)
All help appreciated. Sorry for the long question, but at least there is some data riiiight!
In general inplace operations are not faster and IMHO less readable.
In [18]: df.set_index(['Date','ID','Quality']).unstack('Quality'))
Out[18]:
line_no floor buyer Hour1 Hour2 Hour3 Hour4 Hour5 Hour6
Quality alter good no_checked rejected alter good no_checked rejected alter good no_checked rejected alter good no_checked rejected alter good no_checked rejected alter good no_checked rejected
Date ID
2013-05-04 1 1 5 buyer1 6920 8681 9317 14631 5739 2112 4211 12026 13577 1855 13884 12710 7250 2540 1948 7116 9874 7302 10961 8251 3070 2793 14293 10895
2 2 6 buyer2 7943 7501 13725 1648 7178 9670 6278 6888 9969 11766 9968 4722 7242 4049 6704 2225 6546 8688 11513 14550 2140 11941 1142 6683
3 3 5 buyer1 5155 2449 13648 2183 14184 7309 1185 10454 11742 14102 2242 14297 6185 5554 12505 13312 3062 7426 4421 5693 12342 11622 10431 13375
2013-05-06 1 1 5 buyer1 14563 1343 14419 3350 8526 1185 5244 14777 2238 3640 6717 1109 7777 13136 1732 8681 14454 1059 10606 6942 9349 4524 13931 11799
2 2 6 buyer2 14837 9524 8453 6074 11516 12356 9651 10650 15000 11374 4690 10914 1857 3231 14627 6590 6503 9268 13108 8581 8448 12013 14175 10783
3 3 5 buyer1 9032 12959 4613 6793 7918 2827 6027 13002 11771 13370 12767 11080 12624 13269 11740 10543 8609 14709 11921 12484 8670 12706 8001 8991
[6 rows x 27 columns]
is a quite reasonable idiom for what you are doing