How to handle the duplicate Timestamps in route map? - powerbi

I am using route map, everything is working fine, even the route is accurate.
Only issue before filtering specific path map look somthing like
I am guessing the issue is Timestamp, I am giving 1 to n rank for each path as timestamp.
table views of map
My question is How to create measure that change No column this
No Container_No Events Lat Lon
1 HLXU111 Dub-Rot 53.33306 -6.24889
1 HLXU111 Rot-Kara 51.9225 4.47917
1 HLXU222 Rot-Kara 51.225 4.7917
2 HLXU111 Dub-Rot 52.84235105 -6.16893985
2 HLXU111 Rot-Kara 51.79675051 4.429912263
2 HLXU222 Rot-Kara 51.925 4.4917
3 HLXU111 Dub-Rot 50.55237595 -5.79583915
3 HLXU111 Rot-Kara 51.533278 3.430639
3 HLXU222 Rot-Kara 51.922 4.4791
4 HLXU111 Dub-Rot 50.061667 -5.715889
4 HLXU111 Rot-Kara 49.727861 -1.940028
4 HLXU222 Rot-Kara 49.72786 -1.94002
5 HLXU111 Dub-Rot 50.08522105 -5.4050098
To
No Container_No Events Lat Lon
1.1 HLXU111 Dub-Rot 53.33306 -6.24889
2.1 HLXU111 Rot-Kara 51.9225 4.47917
3.1 HLXU222 Rot-Kara 51.225 4.7917
1.2 HLXU111 Dub-Rot 52.84235105 -6.16893985
2.2 HLXU111 Rot-Kara 51.79675051 4.429912263
3.2 HLXU222 Rot-Kara 51.925 4.4917
1.3 HLXU111 Dub-Rot 50.55237595 -5.79583915
2.3 HLXU111 Rot-Kara 51.533278 3.430639
3.3 HLXU222 Rot-Kara 51.922 4.4791
1.4 HLXU111 Dub-Rot 50.061667 -5.715889
2.4 HLXU111 Rot-Kara 49.727861 -1.940028
4.4 HLXU222 Rot-Kara 49.72786 -1.94002
1.5 HLXU111 Dub-Rot 50.08522105 -5.4050098


I have tried to overcome this issue by creating new column, which generates unique No
!no = RANKX(His_data, COMBINEVALUES(",", His_data[Container_No],COMBINEVALUES("-",[From],[To])),, ASC, DENSE) + ([no]/10)

Related

How to Pivot data in Power BI and then show a line chart for the pivot-ed data

I have interest rates curves data for different dates and i want to compare them. In excel I create a pivot and then from pivot a chart. How do I the same in power bi?
data example:
example of data pivoted in excel (note the filter here chart comparing the series):
Example of PivotChart
I want to create this chart in Power BI
data in text format
SeriesName
SeqId
Data
Value
EUROIS
1
31-Dec-21
1.1
EUROIS
2
31-Dec-21
1.2
EUROIS
3
31-Dec-21
1.3
EUROIS
4
31-Dec-21
1.4
EUROIS
5
31-Dec-21
1.5
EUREURIBOR3M
1
31-Dec-21
3.2
EUREURIBOR3M
2
31-Dec-21
3.3
EUREURIBOR3M
3
31-Dec-21
3.4
EUREURIBOR3M
4
31-Dec-21
3.5
EUREURIBOR3M
5
31-Dec-21
3.6
EUROIS
1
31-Jan-22
0.1
EUROIS
2
31-Jan-22
0.2
EUROIS
3
31-Jan-22
0.3
EUROIS
4
31-Jan-22
0.4
EUROIS
5
31-Jan-22
0.5
EUREURIBOR3M
1
31-Jan-22
2.2
EUREURIBOR3M
2
31-Jan-22
2.3
EUREURIBOR3M
3
31-Jan-22
2.4
EUREURIBOR3M
4
31-Jan-22
2.5
EUREURIBOR3M
5
31-Jan-22
2.6

How to only add common index pandas data frame? [duplicate]

This question already has answers here:
Adding two Series with NaNs
(3 answers)
Closed 5 years ago.
Suppose I have two data frame. I would like to add both values if there is a common index otherwise take the value. Let me illustrate this with an example
import pandas as pd
In [118]: df1 = pd.DataFrame([1, 2, 3, 4], index=pd.date_range('2018-01-01', periods=4))
In [119]: df2 = pd.DataFrame(10*np.ones_like(df1.values[1:3]), index=df1.index[1:3])
In [120]: df1.add(df2)
Out[120]:
0
2018-01-01 NaN
2018-01-02 12.0
2018-01-03 13.0
2018-01-04 NaN
However, I wanted to get
0
2018-01-01 1.0
2018-01-02 12.0
2018-01-03 13.0
2018-01-04 4.0
How can I achieve this? Moreover, is it even possible if df2.index is not a proper subset of df1.index, i.e. if
df2 = pd.DataFrame(10*np.ones_like(df1.values[1:3]), index=pd.DatetimeIndex([df1.index[1], pd.Timestamp('2019-01-01')]))
In [131]: df2
Out[131]:
0
2018-01-02 10
2019-01-01 10
In [132]: df1.add(df2)
Out[132]:
0
2018-01-01 NaN
2018-01-02 12.0
2018-01-03 NaN
2018-01-04 NaN
2019-01-01 NaN
But what I wanted is
0
2018-01-01 1.0
2018-01-02 12.0
2018-01-03 3.0
2018-01-04 4.0
2019-01-01 10.0

Combine with fillna
df1.add(df2).fillna(df1)
Out[581]:
0
2018-01-01 1.0
2018-01-02 12.0
2018-01-03 13.0
2018-01-04 4.0
Ok,
pd.concat([df1,df2]).sum(level=0)
Out[591]:
0
2018-01-01 1
2018-01-02 12
2018-01-03 3
2018-01-04 4
2019-01-01 10

calculates the average of identical columns of several dataframes

I am trying to write a function that calculates the average of identical columns of different dataframes stored in a list:
def mean(dfs):
# declare an empty dataframe
df_mean = pd.DataFrame()
# assign the first column from each raw data framework to df
for i in range(len(dfs)):
dfs[i].set_index(['Time'], inplace=True)
for j in dfs[0].columns:
for i in range(len(dfs)):
df_mean[j] = pd.concat([df_mean,dfs[i][j]], axis=1).mean(axis=1)
return df_mean
dfs = []
l1 = [[1,6,2,6,7],[2,3,2,6,8],[3,3,2,8,8],[4,5,2,6,8],[5,3,9,6,8]]
l2 = [[1,7,2,5,7],[2,3,0,6,8],[3,3,3,6,8],[4,3,7,6,8],[5,3,0,6,8]]
dfs.append(pd.DataFrame(l1, columns=['Time','25','50','75','100']))
dfs.append(pd.DataFrame(l2, columns=['Time','25','50','75','100']))
mean(dfs)
However, I got out only the mean of the first column right!

Option 1
Use python's sum, which well default to reducing the list based on the individual object's __add__ method. Then just divide by the length of the list.
sum(dfs) / len(dfs)
Time 25 50 75 100
0 1.0 6.5 2.0 5.5 7.0
1 2.0 3.0 1.0 6.0 8.0
2 3.0 3.0 2.5 7.0 8.0
3 4.0 4.0 4.5 6.0 8.0
4 5.0 3.0 4.5 6.0 8.0
Option 2
Reconstruct while using numpy's mean function
pd.DataFrame(
np.mean([d.values for d in dfs], 0),
dfs[0].index, dfs[0].columns)
Time 25 50 75 100
0 1.0 6.5 2.0 5.5 7.0
1 2.0 3.0 1.0 6.0 8.0
2 3.0 3.0 2.5 7.0 8.0
3 4.0 4.0 4.5 6.0 8.0
4 5.0 3.0 4.5 6.0 8.0

Use concat on Time indexed list of dataframes, and groupby larger dataframe on Time and take mean
In [275]: pd.concat([d.set_index('Time') for d in dfs]).groupby(level='Time').mean()
Out[275]:
25 50 75 100
Time
1 6.5 2.0 5.5 7.0
2 3.0 1.0 6.0 8.0
3 3.0 2.5 7.0 8.0
4 4.0 4.5 6.0 8.0
5 3.0 4.5 6.0 8.0
Or, since Time column is anyway common for both, atleast in this usecase
In [289]: pd.concat(dfs).groupby(level=0).mean()
Out[289]:
Time 25 50 75 100
0 1.0 6.5 2.0 5.5 7.0
1 2.0 3.0 1.0 6.0 8.0
2 3.0 3.0 2.5 7.0 8.0
3 4.0 4.0 4.5 6.0 8.0
4 5.0 3.0 4.5 6.0 8.0
Details
In [276]: dfs
Out[276]:
[ Time 25 50 75 100
0 1 6 2 6 7
1 2 3 2 6 8
2 3 3 2 8 8
3 4 5 2 6 8
4 5 3 9 6 8, Time 25 50 75 100
0 1 7 2 5 7
1 2 3 0 6 8
2 3 3 3 6 8
3 4 3 7 6 8
4 5 3 0 6 8]
In [277]: pd.concat([d.set_index('Time') for d in dfs])
Out[277]:
25 50 75 100
Time
1 6 2 6 7
2 3 2 6 8
3 3 2 8 8
4 5 2 6 8
5 3 9 6 8
1 7 2 5 7
2 3 0 6 8
3 3 3 6 8
4 3 7 6 8
5 3 0 6 8

pandas using melt to create lookup table

I have a dataframe df of size 24x13 which appears as (I have displayed truncated version of 24x13 array which represents 12 months and 24 hours):
HE 1 2 3 4
0 1 1.8 2.5 3.5 8.5
1 2 2.6 2.9 4.3 8.7
2 3 4.4 2.3 5.3 4.3
3 4 2.6 2.1 4.2 5.3
How do I change this to look up table for each combination of hour and month and display the value in third column as follows:
Hour Month Value
1 1 1.8
1 2 2.5
1 3 3.5
I am trying the following and variation of it but this is not working:
pd.melt(df, id_vars=range(1,24), value_vars=range(1,12))
Edit 1:
df.columns
Index([u'HE', 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12], dtype='object')
df.shape
(24, 13)

df.set_index('HE').stack().reset_index()
Output:
HE level_1 0
0 1 1 1.8
1 1 2 2.5
2 1 3 3.5
3 1 4 8.5
4 2 1 2.6
OR using melt
df.melt(id_vars='HE').sort_values(by=['HE','variable']
Output:
HE variable value
0 1 1 1.8
4 1 2 2.5
8 1 3 3.5
12 1 4 8.5
1 2 1 2.6

Broadcasting pandas dataframe to two dimensional matrix

I have a dataframe of size 12x24 which appears as (I have displayed truncated version):
HE 1 2 3 4 5
0 1 1.8 2.5 3.5 8.5
1 2 2.6 2.9 4.3 8.7
2 3 4.4 2.3 5.3 4.3
3 4 2.6 2.1 4.2 5.3
The column names are number 1 through 12 (representing months) and rows are numbered 1 through 24 (representing hours).
I have another DateTable dataframe which has data as follows:
Date Month Hour
2001-01-01 1 1
2001-02-01 2 4
2001-01-05 1 3
2011-01-31 3 2
2012-01-01 1 5
I want to broadcast the values from 12x24 array into the DateTable to get the following:
Date Month Hour Values
2001-01-01 1 1 3.5
2001-02-01 2 4 5.3
2001-01-05 1 3 2.5
2011-01-31 3 2 2.6
2012-01-01 1 5 1.8
I envision creating some kind of multiindex from the 12x24 table and using against the DateTable but not quite sure about the syntax as I am struggling with syntax.