Calculating results pro rata over several months with PowerQuery - powerbi

I am currently stuck on below issue:
I have two tables that I have to work with, one contains financial information for vessels and the other contains arrival and departure time for vessels. I get my data combining multiple excel sheets from different folders:
financialTable
voyageTimeTable
I have to calculate the result for above voyage, and apportion the result over June, July and August for both estimated and updated.
Time in June : 4 hours (20/06/2020 20:00 - 23:59) + 10 days (21/06/2020 00:00 - 30/06/2020 23:59) = 10.1666
Time in July : 31 full days
Time in August: 1 day + 14 hours (02/08/2020 00:00 - 14:00) = 1.5833
Total voyage duration = 10.1666 + 31 + 1.5833 = 42.7499
The result for the "updated" financialItem would be the following:
Result June : 100*(10.1666/42.7499) = 23.7816
Result July : 100*(31/42.7499) = 72.5148
Result August : 100*(1.5833/42.7499) = 3.7036
sum = 100
and then for "estimated" it would be twice of everything above.
This is the format I ideally would like to get:
prorataResultTable
I have to do this for multiple vessels, with multiple timespans and several voyage numbers.
Eagerly awaiting responses, if any. Many thanks in advance.
Brds,

Not sure if you're still looking for an answer, but code below gives me your expected output:
let
financialTable = Table.FromRows({{"A", 1, "profit/loss", 200, 100}}, type table [vesselName = text, vesselNumber = Int64.Type, financialItem = text, estimated = number, updated = number]),
voyageTimeTable = Table.FromRows({{"A", 1, #datetime(2020, 6, 20, 20, 0, 0), #datetime(2020, 8, 2, 14, 0, 0)}}, type table [vesselName = text, vesselNumber = Int64.Type, voyageStartDatetime = datetime, voyageEndDatetime = datetime]),
joined =
let
joined = Table.NestedJoin(financialTable, {"vesselName", "vesselNumber"}, voyageTimeTable, {"vesselName", "vesselNumber"}, "$toExpand", JoinKind.LeftOuter),
expanded = Table.ExpandTableColumn(joined, "$toExpand", {"voyageStartDatetime", "voyageEndDatetime"})
in expanded,
toExpand = Table.AddColumn(joined, "$toExpand", (currentRow as record) =>
let
voyageInclusiveStart = DateTime.From(currentRow[voyageStartDatetime]),
voyageExclusiveEnd = DateTime.From(currentRow[voyageEndDatetime]),
voyageDurationInDays = Duration.TotalDays(voyageExclusiveEnd - voyageInclusiveStart),
createRecordForPeriod = (someInclusiveStart as datetime) => [
inclusiveStart = someInclusiveStart,
exclusiveEnd = List.Min({
DateTime.From(Date.EndOfMonth(DateTime.Date(someInclusiveStart)) + #duration(1, 0, 0, 0)),
voyageExclusiveEnd
}),
durationInDays = Duration.TotalDays(exclusiveEnd - inclusiveStart),
prorataDuration = durationInDays / voyageDurationInDays,
estimated = prorataDuration * currentRow[estimated],
updated = prorataDuration * currentRow[updated],
month = Date.MonthName(DateTime.Date(inclusiveStart)),
year = Date.Year(inclusiveStart)
],
monthlyRecords = List.Generate(
() => createRecordForPeriod(voyageInclusiveStart),
each [inclusiveStart] < voyageExclusiveEnd,
each createRecordForPeriod([exclusiveEnd])
),
toTable = Table.FromRecords(monthlyRecords)
in toTable
),
expanded =
let
dropped = Table.RemoveColumns(toExpand, {"estimated", "updated", "voyageStartDatetime", "voyageEndDatetime"}),
expanded = Table.ExpandTableColumn(dropped, "$toExpand", {"month", "year", "estimated", "updated"})
in expanded
in
expanded
The code tries to:
join financialTable and voyageTimeTable, so that for each vesselName and vesselNumber combination, we know: estimated, updated, voyageStartDatetime and voyageEndDatetime.
generate a list of months for the period between voyageStartDatetime and voyageEndDatetime (which get expanded into new table rows)
for each month (in the list), do all the arithmetic you mention in your question
get rid of some columns (like the old estimated and updated columns)
I recommend testing it with different vesselNames and vesselNumbers from your dataset, just to see if the output is always correct (I think it should be).
You should be able to manually inspect the cells in the $toExpand column (of the toExpand step/expression) to see the nested rows before they get expanded.

Related

PowerBI Create List of Month Dates

Hi in powerbi I am trying to create a list of dates starting from a column in my table [COD], and then ending on a set date. Right now this is just looping through 60 months from the column start date [COD]. Can i specify an ending variable for it loop until?
List.Transform({0..60}, (x) =>
Date.AddMonths(
(Date.StartOfMonth([COD])), x))
Assuming
start=Date.StartOfMonth([COD]),
end = #date(2020,4,30),
One way is to add column, custom column with formula
= { Number.From(start) .. Number.From(end) }
then expand and convert to date format
or you could generate a list with List.Dates instead, and expand that
= List.Dates(start, Number.From(end) - Number.From(start)+1, #duration(1, 0, 0, 0))
Assuming you want start of month dates through June 2023. In the example below, I have 2023 and 6 hard coded, but this could easily come from a parameter Date.Year(DateParameter) or or column Date.Month([EndDate]).
Get the count of months with this:
12 * (2023 - Date.Year([COD]) )
+ (6 - Date.Month([COD]) )
+ 1
Then just use this column in your formula:
List.Transform({0..[Month count]-1}, (x) =>
Date.AddMonths(Date.StartOfMonth([COD]), x)
)
You could also combine it all into one harder to read formula:
List.Transform(
{0..
(12 * ( Date.Year(DateParameter) - Date.Year([COD]) )
+ ( Date.Month(DateParameter) - Date.Month([COD]) )
)
}, (x) => Date.AddMonths(Date.StartOfMonth([COD]), x)
)
If there is a chance that COD could be after the End Date, you would want to include error checking the the Month count formula.
Generate list:
let
Start = Date1
, End = Date2
, Mos = ElapsedMonths(End, Start) + 1
, Dates = List.Transform(List.Numbers(0,Mos), each Date.AddMonths(Start, _))
in
Dates
ElapsedMonths(D1, D2) function def:
(D1 as date, D2 as date) =>
let
DStart = if D1 < D2 then D1 else D2
, DEnd = if D1 < D2 then D2 else D1
, Elapsed = (12*(Date.Year(DEnd)-Date.Year(DStart))+(Date.Month(DEnd)-Date.Month(DStart)))
in
Elapsed
Of course, you can create a function rather than hard code startdate and enddate:
(StartDate as date, optional EndDate as date, optional Months as number)=>
let
Mos = if EndDate = null
then (if Months = null
then error Error.Record("Missing Parameter", "Specify either [EndDate] or [Months]", "Both are null")
else Months
)
else ElapsedMonths(StartDate, EndDate) + 1
, Dates = List.Transform(List.Numbers(0, Mos), each Date.AddMonths(StartDate, _))
in
Dates

Power Query convert of 3 byte binary date getting variable wasn't recognized

Date displays as 3byte hex value example = 72, 02, 11
Converts to 72 to decimal = 114 + 1900 = 2014, 02 = 02, 11 = 17
So date is 02/17/2014
Bdate comes in as a three byte field, binary so defined
1st byte = year, 2nd byte = mth, 3rd byte = day.
Building a function to call because several dates in data base are of this format.
Secondary function to convert hex to decimal is fnhex2Dec from a library.
Here is code for FixDateBn
let
FixDateBn = (Bdate as binary) =>
let
FixDateBn = BinaryFormat.ByteOrder(
BinaryFormat.Record ([
Yr= BinaryFormat.Byte, //<- the Yr variable is getting Name Yr wasn't recognized.
Mth = BinaryFormat.Byte,
Day = BinaryFormat.Byte ]),
ByteOrder.LittleEndian),
Yrnum = fnhex2Dec(Yr,16)+1900,
Mthnum = fnhex2Dec(Mth,16),
Daynum = fnhex2Dec(Day,16),
Gooddate = #date(Yrnum,Mthnum,Daynum) as date
in
if Bdate is null then null else Gooddate
in
FixDateBn
Appreciate any help.
Here is the sql that converts from hex to regular date. Don't know how to do this in Power Query M language.
CREATE FUNCTION "GetDateFromWDate" (:wDate CHAR(3)) RETURNS DATE AS
BEGIN
DECLARE :tmpdate DATE
SET :tmpdate = '1900-01-01';
IF (:wDate <> null) and (:wDate <> '') THEN
SET :tmpdate = DATEFROMPARTS (ascii(substring(:wDate, 1, 1)) + 1900, ascii(substring(:wDate, 2, 1)), ascii(substring(:wDate, 3, 1)));
END IF;
RETURN :tmpdate;
END
Thanks
Sammy

Pinescript - "Compare" Indicator with Percentage Change Function takes only last bar data

need help pls.
In Tradingview I use "Compare" to see the BTCUSDT vs. ETHUSDT on Binance and it's basically OK. But lines on the chart are too "up & down" and I want to see the SMA or EMA for those tickers.
I'm trying to do it step by step but I can't pass through the issue that my code takes only last calculated value in consideration and "percentage change line" starts from 0 with each new data. So it makes no sence. Meaning, my last data doesn't add upon prior value, but always starts from zero.
So, data (value) that comes out is good (same as when I put same tickers on Tradingview "Compare") but Tradingview "Compare" calculates / adds data on historical data, while my code starts from 0.
Here is the Pine script code:
//#version=4
study(title="Compare", shorttitle="Compare", overlay=false, max_bars_back=0)
perc_change = (((close[0] - open[0]) / open [0]) * 100)
sym1 = "BINANCE:BTCUSDT", res1 = "30", source1 = perc_change
plot(security(sym1, res1, source1), color=color.orange, linewidth=2)
sym2 = "BINANCE:ETHUSDT", res2 = "30", source2 = perc_change
plot(security(sym2, res2, source2), color=color.blue, linewidth=2)
Sounds like the delta between the two ROCs is what you are looking for. With this you can show only the 2 ROCs, but also columns representing the delta between the two. you can also change the ROC's period:
//#version=4
study(title="Compare", shorttitle="Compare")
rocPeriod = input(1, minval = 1)
showLines = input(true)
showDelta = input(true)
perc_change = roc(close, rocPeriod)
sym1 = "BINANCE:BTCUSDT"
sym2 = "BINANCE:ETHUSDT"
res = "30"
s1 = security(sym1, res, perc_change)
s2 = security(sym2, res, perc_change)
delta = s1 - s2
plot(showLines ? s1 : na, "s1", color.orange)
plot(showLines ? s2 : na, "s2", color.blue)
hline(0)
plot(showDelta ? delta : na, "delta", delta > 0 ? color.lime : color.red, 1, plot.style_columns)

Year over Year Variance Calculation Error "EARLIER/EARLIEST" refers to an earlier row context which doesn't exist

While trying to calculate Year over Year variance (unsuccessfully for 2 days now), I get the following error message.
EARLIER/EARLIEST refers to an earlier row context which doesn't exist.
YOY Variance = var PreviousYearPrinBal = CALCULATE(SUM(Deals[Principal Balance]),FILTER(ALL(Deals[Close Date].[Year]),Deals[Close Date].[Year] = EARLIER(Deals[Close Date].[Year])))
return
if(PreviousYearPrinBal = BLANK(), BLANK(), Deals[PrincipalBalance] - PreviousYearPrinBal)
In a different SO question, there is a different approach which gives me the following error:
A column specified in the call to function 'SAMEPERIODLASTYEAR' is not of type DATE. This is not supported.
yoy = CALCULATE([PrincipalBalance], SAMEPERIODLASTYEAR(Deals[Close Date].[Year]))
While I have some idea of what these mean, I do not have an idea of how to fix them. Here is my table.
And Here is what I expect as the result.
I've tried posting this question in Power BI community but haven't received an answer yet. Calculate Year over Year Variance.
ACTUAL DATA SAMPLE:
1) Created Year and Year Difference Column (Calculated Column)
Year = YEAR(Table1[Date])
Year Difference = Table1[Year] - Min(Table1[Year])
2) Created the Variance (Measure)
Variance =
Var current_YearDifference = SELECTEDVALUE(Table1[Year Difference])
Var Current_PrincipalBalance = CALCULATE(SUM(Table1[Principal Balance]),FILTER(ALL(Table1), Table1[Year Difference] = (current_YearDifference)))
Var Previous_PrincipalBalance = CALCULATE(SUM(Table1[Principal Balance]),FILTER(ALL(Table1), Table1[Year Difference] = (current_YearDifference - 1)))
Return if(current_YearDifference <> 0, (Current_PrincipalBalance - Previous_PrincipalBalance), 0)
3) Finally Created the Variance in terms of Percentage (Measure),
Variance in terms of Percentage =
Var current_YearDifference = SELECTEDVALUE(Table1[Year Difference])
Var Current_PrincipalBalance = CALCULATE(SUM(Table1[Principal Balance]),FILTER(ALL(Table1), Table1[Year Difference] = (current_YearDifference)))
Var Previous_PrincipalBalance = CALCULATE(SUM(Table1[Principal Balance]),FILTER(ALL(Table1), Table1[Year Difference] = (current_YearDifference - 1)))
Return if(current_YearDifference <> 0, ((Current_PrincipalBalance - Previous_PrincipalBalance) / Previous_PrincipalBalance), 0)
My Final Output
The Principal Balance has the function SUM selected on the Values Pane of the Output Table, where as the Year is Don't Summarize.
My Best Practice
Always Use Vars when creating Complex Measures to simplify the
formula.
Then return only a part of the Measure to check if the output is as expected.
Kindly let me know, if it helps or not.

Merge two annotated results into one dictionary

I have to annotated results:
cred_rec = Disponibilidad.objects.values('mac__mac', 'int_mes').annotate(tramites=Count('fuar'), recibidas=Count('fecha_disp_mac')
cred14 = Disponibilidad.objects.filter(int_disponible__lte=14).values('mac__mac', 'int_mes').annotate(en14=Count('fuar'))
Both have the same keys 'mac__mac' and 'int_mes', what I want is to create a new dictionary with the keys in cred_red, plus en14 from cred14.
I try some answers found here, but I missing something.
Thanks.
EDIT. After some tries and errors, I got this:
for linea in cred_rec:
clave = (linea['mac__mac'], linea['int_mes'])
for linea2 in cred14:
clave2 = (linea2['mac__mac'], linea2['int_mes'])
if clave2 == clave:
linea['en14'] = linea2['en14']
linea['disp'] = float(linea2['en14'])/linea['recibidas']*100
Now, I have to ask for a better solution. Thanks again.
=======
EDIT
This is how the input looks like:
fuar, mac_id, int_mes, int_disponible, int_exitoso, fecha_tramite, fecha_actualiza_pe, fecha_disp_mac
1229012106349,1,7,21,14,2012-07-02 08:33:54.0,2012-07-16 17:33:21.0,2012-07-23 08:01:22.0
1229012106350,1,7,25,17,2012-07-02 09:01:25.0,2012-07-19 17:45:57.0,2012-07-27 17:45:59.0
1229012106351,1,7,21,14,2012-07-02 09:15:12.0,2012-07-16 19:14:35.0,2012-07-23 08:01:22.0
1229012106352,1,7,24,16,2012-07-02 09:25:19.0,2012-07-18 07:52:18.0,2012-07-26 16:04:11.0
... a few thousand lines dropped ...
The fuar is like an order_id; mac__mac is like the site_id, mes is month; int_disponible is the timedelta between fecha_tramite and fecha_disp_mac; int_exitoso is the timedelta between fecha_tramite and fecha_actualiza_pe.
The output is like this:
mac, mes, tramites, cred_rec, cred14, % rec, % en 14
1, 7, 2023, 2006, 1313, 99.1596638655, 65.4536390828
1, 8, 1748, 1182, 1150, 67.6201372998, 97.2927241963
2, 8, 731, 471, 441, 64.4322845417, 93.6305732484
3, 8, 1352, 840, 784, 62.1301775148, 93.3333333333
tramites is the sum of all orders (fuar) in a month
cred_rec cred is our product, in theory for each fuar there is a cred, cred_rec is the sum of all cred produced in a month
cred_14 is the sum of all cred made in 14 days
% rec the relationship between fuar received and cred produced, in %
% en 14 is the relationship between the cred produced and the cred produced in time
I will use this table in a Annotated Time Line chart or a Combo Chart from Google Charts to show the performance of our manufacturation process.
Thanks for your time.
One immediate improvement to the current code you have would be to have the en14 and disp values precalculated and indexed by key. This will reduce the scans on the cred14 list, but will use memory to store the precalculated values.
def line_key(line):
return (line['mac__mac'], line['int_mes'])
cred14_calcs = {}
for line in cred14:
cred14_calcs[line_key(line)] = {
'en14': line['en14'],
'disp': float(line['en14'])/line['recibidas']*100
}
for line in cred_rec:
calc = cred14_calcs.get(line_key(line))
if calc:
line.update(calc)