Power Query: Date Time Zone operations - powerbi

I am building Power BI report based on data from SQL Server's table User which contains a column CreatedDate of type DateTypeOffset(7).
I am looking for Power Query equivalent to .NET [TimeZoneInfo.FindSystemTimeZoneById]. My Report data has a Time Zone info which is stored as standard ID (like Pacific Standard Time) and I need to convert values in my DateTimeOffset column to the time zone I was talking about above, this is how I do that in C#:
TimeSpan timeZoneOffsetSpan = TimeZoneInfo.FindSystemTimeZoneById(settings.TimeZ​one).GetUtcOffset(UserCreatedDateTime);
This gives me a difference between the time in the time zone stored in my Settings and UTC for UserCreatedDateTime. And I am executing that line of the code just once for all rows in my User table because the purpose of the code above is to find out current offset taking into account such timezone's features like DayLightSaving.
And now I can simply add that offset (which could be positive or negative) against every value in my User.CreateDateTime column, in C# I am doing that by using [DateTimeOffset.ToOffset]:
DateTimeOffset convertedOffset = UserCreatedDateTime.ToOffset(timeZoneOffsetSpan)
So I need to be able to do the same conversions in my Power BI report but I couldnot find any Power Query function which can do the same what [TimeZoneInfo.FindSystemTimeZoneById] does, if that function existed it would cover my first line of C# code and then I would need to find an equivalent to [TimeZoneInfo.FindSystemTimeZoneById]. [DateTimeZone.SwitchZone] is probably what I need but I first need to know the offset corresponding to my time zone and then I would be able to supply that offset as 2nd and 3rd parameters to that function.
So, to finalize, I need Power BI analog for [TimeZoneInfo.FindSystemTimeZoneById].
Can anybody help?

You could add a table to the data model with columns for time zone ID and time zone offset. In your main query you can join this table and then create a calculation to add or subtract the offset into a new column. After that, remove the joined column.
See here How can I perform the equivalent of AddHours to a DateTime in Power Query?
Or you could use the DateTimeZone functions built into M.

You can take advantage of the AT TIME ZONE feature of TSQL to create a table of offsets (in minutes) for each day, for each timezone, and import the resultant table into PowerBI, so you can add/subtract relevant offsets to your base time.
If I was in the OP's situation, I would import (convert) all times to UTC, and then add the offset within PowerBI, dependent on the timezone you want to display. (maybe using this method?)
The example below gives a table the offsets for Australian timezones against UTC.
If my target table is in UTC and I want to change the time in PowerBI, I just have to add the specific offset to my time...
+------------+-----------+----------+----------+----------+
| TargetDate | QLDOffset | NTOffset | SAOffset | WAOffset |
+------------+-----------+----------+----------+----------+
| 2018-04-02 | -600 | -570 | -570 | -480 |
| 2018-04-01 | -600 | -570 | -570 | -480 |
| 2018-03-31 | -600 | -570 | -630 | -480 |
| 2018-03-30 | -600 | -570 | -630 | -480 |
+------------+-----------+----------+----------+----------+
You can see the offset changing for SA on 1st April
WITH MostDays AS ( --MostDays is a table of 10 years worth of dates going back from today.
SELECT Today = CAST(DATEADD(DAY, -days, GETUTCDATE()) AS DATETIMEOFFSET)
FROM (SELECT TOP 3650 Days = ROW_NUMBER() OVER (ORDER BY message_id ) -- this will give a rolling window of 10 years
FROM sys.messages
)x
)
-- This section takes the datetime from above, and calculates how many minutes difference there is between each timezone for each date.
Select TargetDate=CAST(Today AS DATE)
,QLDOffset = DATEDIFF(minute,CAST(Today at TIME ZONE 'UTC' AT TIME ZONE 'E. Australia Standard TIME' AS DateTime),CAST(Today at TIME ZONE 'UTC' AS DATETIME))
,NTOffset = DATEDIFF(minute,CAST(Today at TIME ZONE 'UTC' AT TIME ZONE 'AUS Central Standard Time' AS DateTime),CAST(Today at TIME ZONE 'UTC' AS DATETIME))
,SAOffset = DATEDIFF(minute,CAST(Today at TIME ZONE 'UTC' AT TIME ZONE 'Cen. Australia Standard TIME' AS DateTime),CAST(Today at TIME ZONE 'UTC' AS DATETIME))
,WAOffset = DATEDIFF(minute,CAST(Today at TIME ZONE 'UTC' AT TIME ZONE 'W. Australia Standard TIME' AS DateTime),CAST(Today at TIME ZONE 'UTC' AS DATETIME))
FROM MostDays
Obviously the table gets pretty big if you have no limits on dates or timezones

Related

Amazon CloudWatch Logs Insights: Aggregating logs using bin(1w) always starts on a Sunday

I want to display a bar diagram where each bar represents a week starting on a Monday.
The below meta code does that except that each week starts on a Sunday.
I realize that the Query syntax only mentions time periods with m for minutes and h for hours, but it seems to work fine also using d for day and w for week (except that I cannot set the starting day).
Any idea how to make weeks starting on Mondays instead of Sundays?
fields #timestamp, #message
| fields strcontains(#message, 'the start') as start
| fields strcontains(#message, 'the result') as result
| stats sum(start) as startCalls, sum(result) as resultCalls by bin(1w) as t
| sort t asc

How to get uptime total and percentage of GCP compute vm instance through MQL?

I am trying to get total uptime of a single GCP compute vm instance inclusive of restarts. I've seen multiple posts not one with using MQL.
Eg: In the past 24 hours if instance is not running for 1hr , i expect the mql query to return 23 hrs
In the below snap, code snippet the graph reqpresents the max uptime but doesn't consider the restarts . I've tried using secondary aggregator with max but still query doesn't report the exact value.
If you have any idea on how to get information of total uptime in the past 1 day through MQL that would be very helpful. Any pointers are much appreciated. Thank you.
fetch gce_instance
| metric 'compute.googleapis.com/instance/uptime_total'
| group_by 1d, [value_uptime_total_max: max(value.uptime_total)]
| every 1d
you can try with the uptime metric instead :
fetch gce_instance
| metric 'compute.googleapis.com/instance/uptime'
| filter (metric.instance_name == 'instance-1')
| align delta(1d)
| every 1d
| group_by [], [value_uptime_mean: mean(value.uptime)]
so you get a graph similar to this one:
Using sliding in the group_by and sum aggregator for the calculation.
fetch gce_instance
| metric 'compute.googleapis.com/instance/uptime_total'
| filter (metric.instance_name = "the instance name you need")
| group_by [], sliding(1d), [value_uptime_total_sum: sum(value.uptime_total)]
GCP compute VM metrics instace/uptime , instance/uptime_total are not reliable. Rather tracking uptime through uptime check and using following MQL query gives the exact values for historical uptime.
Please replace 30d with appropriate value 1d , 1h
fetch uptime_url
| metric 'monitoring.googleapis.com/uptime_check/check_passed'
| filter (metric.check_id == 'dev-uptime-test')
| group_by 30d,
[value_check_passed_fraction_true: fraction_true(value.check_passed)]
| every 30d | mean

Power BI calculate sum only last value of duplicate ID

I'm struggling to create a Measure that sums a column and have it filter out duplicate IDs while taking only the latest row.
For example, there is a table as such:
UID | Quantity | Status | StatusDate
aaa | 3 | Shipped | 11/1/2020
aaa | 3 | Delivered | 11/5/2020
bbb | 5 | Ordered | 10/29/2020
ccc | 8 | Shipped | 11/4/2020
So the idea would be to sum the quantity, but I would only want to count quantity for id "aaa" once and only count towards the latest status ("Delivered" in this case). I would make a visual that shows the quantities with status as its axis. I also need to add a date Slicer so I could go back in time. So, when I go before 11/5/2020, instead of "Delivered," it would switch back to "Shipped."
I tried several methods:
SUMMARIZE to a table filtering "MAX" date value if UID is the same. I found this doesn't work with the date slicer since it seems like it is not actually recalculating the filtering and just slicing away rows outside of the dates. Seems to be the same whether the SUMMARIZE is set as a new table or VAR in the Measure.
CALCULATE seems promising but I can't seem to figure out a syntax
that filters that I need. Example of one that doesn't work (I also tried SUMX instead of SUM but that doesn't work either):
CALCULATE(
SUM(Table1[Quantity]),
FILTER(Table1, [StatusDate]=MAXX(FILTER(Table1,[UID]=EARLIER([UID])),[StatusDate])
)
I also tried adding a column that states whether if the row is "old" as well as a numerical "rank" to the different statuses. But once again, I run into the issue where the date slicer is not exactly filtering to recalculate those columns. For example, if the date slicer is set to 11/3/2020, it should add "3" to "Shipped" instead of "Delivered." But instead of that, it just removes the row which tells me that it is not actually recalculating the columns (like #1).
Any help would be appreciated :-) Thank you!
You can try something like this:
Measure =
VAR d = LASTDATE(Table1[StatusDate])
VAR tb = SUMMARIZE(FILTER(Table1, Table1[StatusDate] <= d),
Table1[UID],
"last", LASTDATE(Table1[StatusDate]))
RETURN CALCULATE(SUM(Table1[Quantity]), TREATAS(tb, Table1[UID], Table1[StatusDate]))
The tb variable contains a table which has the latest date per UID. You then use that to filter your main table with the TREATAS function.
One other alternative is to create a table with the RANK function ordered by date and then doing a SUM over that table, where Rank = 1.

AWS Dynamo DB: High read latency using query on GSI

I have a dynamo db table which contains Date, City and other attributes as the columns. I have configured GSI with Date as the hash key. The table contains 27 attributes from 350 cities recorded daily.
| Date | City | Attribute1 | Attribute27|
+------------+------------+-------------+------------+
| 25-06-2020 | Boston | someValue | someValue |
| 25-06-2020 | NY | someValue | someValue |
| 25-06-2020 | Chicago | someValue | someValue |
+------------+------------+-------------+------------+
I have a Lambda proxy integration setup in API Gateway. The lambda function receives a 7 day date range as the request. Each of the date, in this range is used query the dynamodb (using query input) to get all the items for a given day. The result for each day is consolidated for a week, and is then sent back as a JSON response.
The latency seen in POSTMAN is around 1.5s, after increasing the lambda memory to 1024MB (Even though, only 76MB is being consumed).
Is there any way to improve the performance? The dynamo db is already running in On-Demand Capacity.
You don't say if you are using parallel queries or not.
If not do so.
You also don't say what cloudwatch is showing for Query latency, as mentioned by Marcin, DAX can help reduce that.
You also don't mention what cloudwatch is showing for lambda execution. There's various articles about optimizing lambda.
Whatever's left is networking...not much you can do about that..one piece to consider is reusing Db connections in your lambda

PowerBi: incremental data load by using OData feed

is there any possibility to save previous data before overriding because of refreshing data?
Steps i have done:
I Created a table and appended to table A
Created a Column called DateTime with the function
DateTime.LocalNow()
Now i have a problem how to save previous data before the refreshing phase. I need to preserve the timestamp of previous data and actually data.
Example giving:
Before refreshing:
Table A:
|Columnname x| DateTime | ....
| value | 23.03.2016 23:00
New Table:
|Columnname x| DateTime | ....
| value | 23.03.2016 23:00
After refreshing:
Table A:
|Columnname x| DateTime | ....
| value | 23.03.2016 23:00
| value 2 | 23.03.2016 23:01
New Table:
|Columnname x| DateTime | ....
| value | 23.03.2016 23:00
| value 2 | 23.03.2016 23:01
kind regards
Incremental refreshes in the Power BI Service or Power BI Desktop aren't currently supported. But please vote for this feature. (update: see that link for info on a preview feature that does this)
If you need this behavior you need to load these rows to a database then incrementally load the database. The load to Power BI will still be a full load of the table(s).
This is now available in PowerBI Premium
From the docs
Incremental refresh enables very large datasets in the Power BI Premium service with the following benefits:
Refreshes are faster. Only data that has changed needs to be refreshed. For example, refresh only the last 5 days of a 10-year dataset.
Refreshes are more reliable. For example, it is not necessary to maintain long-running connections to volatile source systems.
Resource consumption is reduced. Less data to refresh reduces overall consumption of memory and other resources.