Getting matrix from data using pandas or python - python-2.7

How can I write pandas or python code to obtain matrix from my data? I have the following table:
Item Route Order
R124 A1 1
R124 A2 2
R124 A3 3
R124 A4 4
R124 A4 4
R126 A5 1
R126 A6 2
R126 A7 3
R126 A7 3
My required output is:
A1 A2 A3 A4 A5 A6 A7
R124 1 1 1 2 0 0 0
R126 0 0 0 0 1 1 2
To obtain the matrix, a unique 'Item' value becomes a row name. For example R124 has 1 entry each in the 'Order' column mapping to A1, A2 and A3 in the 'Route' column, and 2 entries mapping to A4 in the 'Route' column. The number of mapped entries are recorded accordingly. Since R124 has no 'Order' entry mapping to 'Route' column for A5, A6 and A7, zeros are recorded as shown in the output matrix.


You just need pivot_table.
If your data frame is df:
df.pivot_table(index="Item",columns="Route",values="Order",aggfunc='count')
gives:
Route A1 A2 A3 A4 A5 A6 A7
Item
R124 1.0 1.0 1.0 2.0 NaN NaN NaN
R126 NaN NaN NaN NaN 1.0 1.0 2.0
and to completely mimic your desired answer just add fillna and astype:
df.pivot_table(index="Item",columns="Route",values="Order",aggfunc='count').fillna(0).astype(int)
gives
Route A1 A2 A3 A4 A5 A6 A7
Item
R124 1 1 1 2 0 0 0
R126 0 0 0 0 1 1 2

Related

Packing and de-interleaving two __m256 registers

I have a row-wise array of floats (~20 cols x ~1M rows) from which I need to extract two columns at a time into two __m256 registers.
...a0.........b0......
...a1.........b1......
// ...
...a7.........b7......
// end first __m256
A naive way to do this is
__m256i vindex = _mm256_setr_epi32(
0,
1 * stride,
2 * stride,
// ...
7 * stride);
__m256 colA = _mm256_i32gather_ps(baseAddrColA, vindex, sizeof(float));
__m256 colB = _mm256_i32gather_ps(baseAddrColB, vindex, sizeof(float));
However, I was wondering if I would get better performance by retrieving a0, b0, a1, b1, a2, b2, a3, b3 in one gather, and a4, b4, ... a7, b7 in another because they're closer in memory, and then de-interleave them. That is:
// __m256 lo = a0 b0 a1 b1 a2 b2 a3 b3 // load proximal elements
// __m256 hi = a4 b4 a5 b5 a6 b6 a7 b7
// __m256 colA = a0 a1 a2 a3 a4 a5 a6 a7 // goal
// __m256 colB = b0 b1 b2 b3 b4 b5 b6 b7
I can't figure out how to nicely interleave lo and hi. I basically need the opposite of _mm256_unpacklo_ps. The best I've come up with is something like:
__m256i idxA = _mm256_setr_epi32(0, 2, 4, 6, 1, 3, 5, 7);
__m256i idxB = _mm256_setr_epi32(1, 3, 5, 7, 0, 2, 4, 6);
__m256 permLA = _mm256_permutevar8x32_ps(lo, idxA); // a0 a1 a2 a3 b0 b1 b2 b3
__m256 permHB = _mm256_permutevar8x32_ps(hi, idxB); // b4 b5 b6 b7 a4 a5 a6 a7
__m256 colA = _mm256_blend_ps(permLA, permHB, 0b11110000); // a0 a1 a2 a3 a4 a5 a6 a7
__m256 colB = _mm256_setr_m128(
_mm256_extractf128_ps(permLA, 1),
_mm256_castps256_ps128(permHB)); // b0 b1 b2 b3 b4 b5 b6 b7
That's 13 cycles. Is there a better way?
(For all I know, prefetch is already optimizing the naive approach as best as possible, but lacking that knowledge, I was hoping to benchmark the second approach. If anyone already knows what the result of this would be, please do share. With the above de-interlacing method, it's about 8% slower than the naive approach.)
Edit Even without the de-interlacing, the "proximal" gather method is about 6% slower than the naive, constant-stride gather method. I take that to mean that this access pattern confuses hardware prefetch too much to be a worthwhile optimization.

// __m256 lo = a0 b0 a1 b1 a2 b2 a3 b3 // load proximal elements
// __m256 hi = a4 b4 a5 b5 a6 b6 a7 b7
// __m256 colA = a0 a1 a2 a3 a4 a5 a6 a7 // goal
// __m256 colB = b0 b1 b2 b3 b4 b5 b6 b7
It seems we can do this shuffle even faster than my orginal answer:
void unpack_cols(__m256i lo, __m256i hi, __m256i& colA, __m256i& colB) {
const __m256i mask = _mm256_setr_epi32(0, 2, 4, 6, 1, 3, 5, 7);
// group cols crossing lanes:
// a0 a1 a2 a3 b0 b1 b2 b3
// a4 a5 a6 a7 b4 b5 b6 b7
auto lo_grouped = _mm256_permutevar8x32_epi32(lo, mask);
auto hi_grouped = _mm256_permutevar8x32_epi32(hi, mask);
// swap lanes:
// a0 a1 a2 a3 a4 a5 a6 a7
// b0 b1 b2 b3 b4 b5 b6 b7
colA = _mm256_permute2x128_si256(lo_grouped, hi_grouped, 0 | (2 << 4));
colB = _mm256_permute2x128_si256(lo_grouped, hi_grouped, 1 | (3 << 4));
}
While both instructions have a 3 cycles latency on Haswell (see Agner Fog) they have a single cycle throughput. This means it has a throughput of 4 cycles and 8 cycles latency. If you have a spare register which can keep the mask this should be better. Doing only two of these in parallel allows you to completly hide its latency. See godbolt and rextester.
Old answer, kept for reference:
The fastest way to do this shuffle is the following:
void unpack_cols(__m256i lo, __m256i hi, __m256i& colA, __m256i& colB) {
// group cols within lanes:
// a0 a1 b0 b1 a2 a3 b2 b3
// a4 a5 b4 b5 a6 a7 b6 b7
auto lo_shuffled = _mm256_shuffle_epi32(lo, _MM_SHUFFLE(3, 1, 2, 0));
auto hi_shuffled = _mm256_shuffle_epi32(hi, _MM_SHUFFLE(3, 1, 2, 0));
// unpack lo + hi a 64 bit
// a0 a1 a4 a5 a2 a3 a6 a7
// b0 b1 b4 b5 b2 b3 b6 b7
auto colA_shuffled = _mm256_unpacklo_epi64(lo_shuffled, hi_shuffled);
auto colB_shuffled = _mm256_unpackhi_epi64(lo_shuffled, hi_shuffled);
// swap crossing lanes:
// a0 a1 a2 a3 a4 a5 a6 a7
// b0 b1 b2 b3 b4 b5 b6 b7
colA = _mm256_permute4x64_epi64(colA_shuffled, _MM_SHUFFLE(3, 1, 2, 0));
colB = _mm256_permute4x64_epi64(colB_shuffled, _MM_SHUFFLE(3, 1, 2, 0));
}
Starting with Haswell this has a throughput of 6 cycles (sadly six instructions on port 5). According to Agner Fog _mm256_permute4x64_epi64 has a latency of 3 cycles. This means unpack_cols has a latency of 11 8 cycles.
You can check the code on godbolt.org or test it at rextester which has AVX2 support but sadly no permalinks like godbolt.
Note that this is also very close to the problem I had where I gathered 64 bit ints and needed the high and low 32 bits separated.
Note that gather performance is really bad in Haswell but according to Agner Fog Skylake got a lot better at it (~12 cycles throughput down to ~5). Still shuffling around such simple patterns should still be a lot faster than gathering.

In order to load columns of 32-bit float type you could use intrinsics _mm256_setr_pd and _mm256_shuffle_ps (it takes 10 cycles):
#include <iostream>
#include <immintrin.h>
inline void Print(const __m256 & v)
{
float b[8];
_mm256_storeu_ps(b, v);
for (int i = 0; i < 8; i++)
std::cout << b[i] << " ";
std::cout << std::endl;
}
int main()
{
const size_t stride = 100;
float m[stride * 8];
for (size_t i = 0; i < stride*8; ++i)
m[i] = (float)i;
const size_t stride2 = stride / 2;
double * p = (double*)m;
__m256 ab0145 = _mm256_castpd_ps(_mm256_setr_pd(p[0 * stride2], p[1 * stride2], p[4 * stride2], p[5 * stride2]));
__m256 ab2367 = _mm256_castpd_ps(_mm256_setr_pd(p[2 * stride2], p[3 * stride2], p[6 * stride2], p[7 * stride2]));
__m256 a = _mm256_shuffle_ps(ab0145, ab2367, 0x88);
__m256 b = _mm256_shuffle_ps(ab0145, ab2367, 0xDD);
Print(a);
Print(b);
return 0;
}
Output:
0 100 200 300 400 500 600 700
1 101 201 301 401 501 601 701
Concerning to performance of intrinsic _mm256_i32gather_ps I would recommend to see here.

I assume that a and b are placed in 0,10, then 1,11 to 9,19 if not chnge the vindexm[] as you want ;
If you want to use gather instruction:
//#includes
#define Distance 20 // number of columns.
float a[32][20]__attribute__(( aligned(32)))= {{1.01,1.02,1.03,1.04,1.05,1.06,1.07,1.08,1.09,1.10,1.11,1.12,1.13,1.14,1.15,1.16},
{2.01,2.02,2.03,2.04,2.05,2.06,2.07,2.08,2.09,2.10,2.11,2.12,2.13,2.14,2.15,2.16},
{3.01,3.02,3.03,3.04,3.05,3.06,3.07,3.08,3.09,3.10,3.11,3.12,3.13,3.14,3.15,3.16},
{4.01,4.02,4.03,4.04,4.05,4.06,4.07,4.08,4.09,4.10,4.11,4.12,4.13,4.14,4.15,4.16},
{5.01,5.02,5.03,5.04,5.05,5.06,5.07,5.08,5.09,5.10,5.11,5.12,5.13,5.14,5.15,5.16},
{6.01,6.02,6.03,6.04,6.05,6.06,6.07,6.08,6.09,6.10,6.11,6.12,6.13,6.14,6.15,6.16},
{7.01,7.02,7.03,7.04,7.05,7.06,7.07,7.08,7.09,7.10,7.11,7.12,7.13,7.14,7.15,7.16},
{8.01,8.02,8.03,8.04,8.05,8.06,8.07,8.08,8.09,8.10,8.11,8.12,8.13,8.14,8.15,8.16},
{9.01,9.02,9.03,9.04,9.05,9.06,9.07,9.08,9.09,9.10,9.11,9.12,9.13,7.14,9.15,9.16},
{10.1,10.2,10.3,10.4,10.5,10.6,10.7,10.8,10.9,10.10,10.11,10.12,10.13,10.14,10.15,10.16},
{11.1,11.2,11.3,11.4,11.5,11.6,11.7,11.8,11.9,11.10,11.11,11.12,11.13,11.14,11.15,11.16},
{12.1,12.2,12.3,12.4,12.5,12.6,12.7,12.8,12.9,12.10,12.11,12.12,12.13,12.14,12.15,12.16},
{13.1,13.2,13.3,13.4,13.5,13.6,13.7,13.8,13.9,13.10,13.11,13.12,13.13,13.14,13.15,13.16},
{14.1,14.2,14.3,14.4,14.5,14.6,14.7,14.8,14.9,14.10,14.11,14.12,14.13,14.14,14.15,14.16},
{15.1,15.2,15.3,15.4,15.5,15.6,15.7,15.8,15.9,15.10,15.11,15.12,15.13,15.14,15.15,15.16},
{16.1,16.2,16.3,16.4,16.5,16.6,16.7,16.8,16.9,16.10,16.11,16.12,16.13,16.14,16.15,16.16}};
float tempps[8];
void printVecps(__m256 vec)
{
_mm256_store_ps(&tempps[0], vec);
printf(", [0]=%3.2f, [1]=%3.2f, [2]=%3.2f, [3]=%3.2f, [4]=%3.2f, [5]=%3.2f, [6]=%3.2f, [7]=%3.2f \n",
tempps[0],tempps[1],tempps[2],tempps[3],tempps[4],tempps[5],tempps[6],tempps[7]) ;
}
int main() {
__m256 vec1;
int vindexm [8]={0, Distance/2, Distance, Distance + Distance/2, Distance*2, Distance*2 +Distance/2, Distance*3, Distance*3 + Distance/2};
__m256i vindex = _mm256_load_si256((__m256i *) &vindexm[0]);
//loops
vec1 = _mm256_i32gather_ps (&a[0][0],vindex, 4);//place it in your loop as you want
printVecps(vec1);
return 0;
}
the out put is
[0]=1.01, [1]=1.11, [2]=2.01, [3]=2.11, [4]=3.01, [5]=3.11, [6]=4.01, [7]=4.11

CUDA - check repeated values and add two values

I have two group of arrays
a1 a2 a3 a4 a5 a6 a7 a8 <= name it as key1
b1 b2 b3 b4 b5 b6 b7 b8 <= val1
c1 c2 c3 c4 c5 c6 c7 c8
and
d1 d2 d3 d4 d5 d6 d7 d8 <= key2
e1 e2 e3 e4 e5 e6 e7 e8 <= val2
f1 f2 f3 f4 f5 f6 f7 f8
The arrays a1,...,an and d1,...,dn are sorted and might be repeated. i.e. their values might be something like 1 1 2 3 4 6 7 7 7 ... I want to check if for each Tuple di,ei check if it is equal to any of ai,bi. If it is (di==ai,bi==ei) then I have to combine fi and ci using some function e.g. add and store in fi.
Firstly, is it possible to do this using zip iterators and transformation in thurst library to solve this efficiently?
Secondly, the simplest method that I can imagine is to count occurance of number of each keys (ai) do prefix sum and use both to get start and end index of each keys and then for each di use above counting to iterate through those indices and check if ei==di. and perform the transformation.
i.e. If I have
1 1 2 3 5 6 7
2 3 4 5 2 4 6
2 4 5 6 7 8 5
as first array, I count the occurance of 1,2,3,4,5,6,7,...:
2 1 1 0 1 1 1 <=name it as count
and then do prefix sum to get:
2 3 4 4 5 6 7 <= name it as cumsum
and use this to do:
for each element di,
for i in (cumsum[di] -count[di]) to cumsum[di]:
if ei==val1[i] then performAddition;
What I fear is that since not all threads are equal, this will lead to warp divergence, and I may not have efficient performance.

You could treat your data as two key-value tables.Table1: (a,b) -> c and Table2: (d,e)->f, where pair (a,b) and (d,e) are keys, and c, f are values.
Then your problem simplifies to
foreach key in Table2
if key in Table1
Table2[key] += Table1[key]
Suppose a and b have limited ranges and are positive, such as unsigned char, a simple way to combine a and b into one key is
unsigned short key = (unsigned short)(a) * 256 + b;
If the range of key is still not too large as in the above example, you could create your Table1 as
int Table1[65536];
Checking if key in Table1 becomes
if (Table1[key] != INVALID_VALUE)
....
With all these restrictions, implementation with thrust should be very simple.
Similar combining method could still be used if a and b have larger range like int.
But if the range of key is too large, you have to go to the method suggested by Robert Crovella.

How to Merge Two Dataframes Without Losing Any Rows

I have two data frames:
df1 =
Id ColA ColB ColC
1 aa bb cc
3 11 ww 55
5 11 bb cc
df2 =
Id ColD ColE ColF
1 ff ee rr
2 ww rr 55
3 hh 11 22
4 11 11 cc
5 cc bb aa
I need to merge these two data frames to get the following result:
result =
Id ColA ColB ColC ColD ColE ColF
1 aa bb cc ff ee rr
2 NaN NaN NaN ww rr 55
3 11 ww 55 hh 11 22
4 NaN NaN NaN 11 11 cc
5 11 bb cc cc bb aa
I do the merging this way:
import pandas as pd
result = pd.merge(df1,df2,on='Id')
However my result looks as follows instead of the expected above-shown result:
result =
Id ColA ColB ColC ColD ColE ColF
1 aa bb cc ff ee rr
3 11 ww 55 hh 11 22
5 11 bb cc cc bb aa

According to the documentation of merge, you need to specify the 'how' parameter as outer (the default is inner, which is consistent with what you're getting):
outer: use union of keys from both frames (SQL: full outer join)
inner: use intersection of keys from both frames (SQL: inner join)

Merge on one key after the other

I often end up with the following situation. I have a dataframe with two IDs
A = pd.DataFrame([[1,'a', 'a1'], [2, None, 'a2'], [3,'c', 'a3'], [4,'None', 'a3'], [None, 'e', 'a3'], ['None', 'None', 'None']], columns = ['id1', 'id2', 'colA'])
id1 id2 colA
0 1 a a1
1 2 None a2
2 3 c a3
3 4 None a3
4 None e a3
5 None None None
and I have another dataframe with additional info I want to add to the first dataframe
B = pd.DataFrame([[1,'a', 'b1', 'c1'], [2, 'b', 'b2', 'c2'], [3,'c', 'b3', 'c3'], [4, 'd', 'b4', 'c4'], [5, 'e', 'b5', 'c5'], [6, 'e', 'b5', 'c5']], columns = ['id1', 'id2', 'colB', 'colC'])
Out[15]:
id1 id2 colB colC
0 1 a b1 c1
1 2 b b2 c2
2 3 c b3 c3
3 4 d b4 c4
4 5 e b5 c5
5 6 e b5 c5
I want to merge on id1, like this
A.merge(B, how='left', on='id1')
id1 id2_x colA id2_y colB colC
0 1 a a1 a b1 c1
1 2 None a2 b b2 c2
2 3 c a3 c b3 c3
3 4 None a3 d b4 c4
4 None e a3 NaN NaN NaN
5 None None None NaN NaN NaN
This is close to what I want. However for the failed lookups (that is when id1 is not available) I would like to merge on id2, so the result looks like
id1 id2_x colA id2_y colB colC
0 1 a a1 a b1 c1
1 2 None a2 b b2 c2
2 3 c a3 c b3 c3
3 4 None a3 d b4 c4
4 None e a3 NaN b5 c5
5 None None None NaN NaN NaN
What's the best way to achieve this? Note I don't really want 2 id2 columns in the result and id2 may have duplicates.

IIUC you use fillna. But it fill last row too.
print df
id1 id2_x colA id2_y colB colC
0 1 a a1 a b1 c1
1 2 None a2 b b2 c2
2 3 c a3 c b3 c3
3 4 None a3 d b4 c4
4 None e a3 NaN NaN NaN
5 None None None NaN NaN NaN
df = df.fillna(B)
print df
id1 id2_x colA id2_y colB colC
0 1 a a1 a b1 c1
1 2 None a2 b b2 c2
2 3 c a3 c b3 c3
3 4 None a3 d b4 c4
4 None e a3 NaN b5 c5
5 None None None NaN b5 c5
As EdChum mentioned in comments, next solution is use combine_first, but output is different:
print A.combine_first(B)
colA colB colC id1 id2
0 a1 b1 c1 1 a
1 a2 b2 c2 2 b
2 a3 b3 c3 3 c
3 a3 b4 c4 4 None
4 a3 b5 c5 5 e
5 None b5 c5 None None
Difference is:
In [142]: %timeit A.combine_first(B)
100 loops, best of 3: 3.44 ms per loop
In [143]: %timeit A.merge(B, how='left', on='id1').fillna(B)
100 loops, best of 3: 2.89 ms per loop

SAS: How do I pick conditional observations per by-group

I have a SAS data set, which I have sorted according to my needs. I want to split it into BY groups and, for each group, output each observation until the first occurrence of a particular value in a particular column.
ID No C1 Year2 C3 Date (DD/MM/YYYY)
---------------------------------------------------------
AB123 4 B4 2008E OC 09/04/2008
AB123 3 B4 2008E EL 09/04/2008
AB123 2 B4 2008E ZZ 09/04/2008
AB123 1 B4 2008E OC 09/04/2008
AB123 0 B4 2008E ZZ 09/04/2008
AB123 1 B4 2008E OC 06/02/2008
AB123 0 B4 2008E ZZ 06/02/2008
This is one BY group: the data set is grouped by ID, C1, Year2 and sorted by ID, C1, Year2, Date(desc), No(desc). Further instances of each of ID, C1 and Year2 could occur anywhere in the data set, but the 3 variables define each BY group.
I want to output all observations per BY group up to and including the first occurrence of ZZ in C3. So above I would want the first 3 observations output (or flagged) and then move on to the next BY group.
Any help would be greatly appreciated. Please let me know if you need any more details of the problem. Thanks.

Here's one way that should work.
data have;
input ID $ No C1 $ Year2 $ C3 $ Date :DDMMYY10.;
format date DDMMYY10.;
cards;
AB123 4 B4 2008E OC 09/04/2008
AB123 3 B4 2008E EL 09/04/2008
AB123 2 B4 2008E ZZ 09/04/2008
AB123 1 B4 2008E OC 09/04/2008
AB123 0 B4 2008E ZZ 09/04/2008
AB123 1 B4 2008E OC 06/02/2008
AB123 0 B4 2008E ZZ 06/02/2008
;
run;
data want (drop=stopflag);
set have;
by id c1 year2;
retain stopflag;
if max(first.id,first.c1,first.year2)=1 then stopflag=0;
if c3='ZZ' and stopflag=0 then do;
output;
stopflag=1;
end;
if stopflag=0 then output;
run;