MultiIndexing in pandas based on column conditions - python-2.7

Question

I have a huge GPS datasets in a csv file.
It is something like this.
12,1999-09-08 12:12:12, 116.3426, 32.5678
12,1999-09-08 12:12:17, 116.34234, 32.5678
.
.
.
where each column is in the form of
id, timestamp, longitude, latitude
Now, I am using pandas and importing the file into a dataframe, I have so far written this code.
import pandas as pd
import numpy as np
#this imports the columns and making the timestamp values as row indexes
df = pd.read_csv('/home/abc/Downloads/..../366.txt',delimiter=',',
index_col=1,names=['id','longitude','latitude'])
#removes repeated entries due to gps errors.
df = df.groupby(df.index).first()
Sometimes, there will be 2 or 3 multiple entries for same date which should be removed
I get something like this
id longitude latitude
1999-09-08 12:12:12 12 116.3426 32.5678
1999-09-08 12:12:17 12 116.34234 32.5678
# and so on with redundant entries removed
Now I want rows which have same latitude and longitude to be indexed serially..
i.e., my visualization is
id longitude latitude
0 1999-09-08 12:12:12 12 116.3426 32.5678
1 1999-09-08 12:12:17 12 116.34234 32.5678
2 1999-09-08 12:12:22 12 116.342341 32.5678
1999-09-08 12:12:27 12 116.342341 32.5678
1999-09-08 12:12:32 12 116.342341 32.5678
....
1999-09-08 12:19:37 12 116.342341 32.5678
3 1999-09-08 12:19:42 12 116.34234 32.56123
and so on..
i.e., rows with same latitude and longitude values are to be indexed serially. How can i achieve that? i am a beginner in pandas so i don't know much about it. pls help!

Answer 1

You should leverage the DataFrame.duplicated and do some math with it:
idx = df.duplicated(['longitude', 'latitude'])
idx *= -1
idx += 1
idx.ix[0] = 0
df = df.set_index(idx.cumsum(), append=True).swaplevel(0,1)
How the code works
Starting with the df you get:
In [215]: df
Out[215]:
id longitude latitude
stamp
1999-09-08T12:12:12 12 116.342600 32.56780
1999-09-08T12:12:17 12 116.342340 32.56780
1999-09-08T12:12:22 12 116.342341 32.56780
1999-09-08T12:12:27 12 116.342341 32.56780
1999-09-08T12:12:32 12 116.342341 32.56780
1999-09-08T12:19:37 12 116.342341 32.56780
1999-09-08T12:19:42 12 116.342340 32.56123
First calculate the consecutive duplicated (longitude, latitude) tuples:
In [216]: idx = df.duplicated(['longitude', 'latitude'])
In [217]: idx
Out[217]:
stamp
1999-09-08T12:12:12 False
1999-09-08T12:12:17 False
1999-09-08T12:12:22 False
1999-09-08T12:12:27 True
1999-09-08T12:12:32 True
1999-09-08T12:19:37 True
1999-09-08T12:19:42 False
Then we use cumsum to create a zero-based index that does not increment on duplicaes.
Put some math with it to obtain zeros on duplicated rows and ones for others:
In [218]: idx *= -1
In [219]: idx += 1
In [220]: idx
Out[220]:
stamp
1999-09-08T12:12:12 1
1999-09-08T12:12:17 1
1999-09-08T12:12:22 1
1999-09-08T12:12:27 0
1999-09-08T12:12:32 0
1999-09-08T12:19:37 0
1999-09-08T12:19:42 1
As we want a zero-based index, we set the first cell to 0, and we append that column to the index of df to create the MultiIndex:
In [221]: idx.ix[0] = 0
In [222]: df = df.set_index(idx.cumsum(), append=True)
By default, set_index adds the index at an inferior level than the existing one. We must finish by swapping the levels between the timestamps and our additional index:
In [223]: df = df.swaplevel(0,1)
In [224]: df
Out[224]:
id longitude latitude
stamp
0 1999-09-08T12:12:12 12 116.342600 32.56780
1 1999-09-08T12:12:17 12 116.342340 32.56780
2 1999-09-08T12:12:22 12 116.342341 32.56780
1999-09-08T12:12:27 12 116.342341 32.56780
1999-09-08T12:12:32 12 116.342341 32.56780
1999-09-08T12:19:37 12 116.342341 32.56780
3 1999-09-08T12:19:42 12 116.342340 32.56123