I have a process where I have relationship between my sales table and my custom date table. My sales data gets its' filter from the custom date table. I have a process where I have 3 refreshes in 3 consecutive days starting from EOM of "previous month" and show last 12 calendar months.
I start this process end of every month i.e. For February data report I start on 02/28/2023 that will be my first refresh to filter sales data for last 12 calendar months (03/01/2022-02/28/2023). The next following day (03/01/2023) the second refresh need to filter (03/01/2022-02/28/2023).Then the final refresh is the next following day (03/02/2023) also need to filter (03/01/2022-02/28/2023). I couldn't use the standard filters in PBI since the eom is an exception. I have tried to create custom table using a formula where I add 7 days to today(02/28/2023) so I can go forward in time for my first day(02/28) and come back a month so I can get whole 12 calendar months. This wouldn't matter for the next month(03) since 7 days is not going to make any difference for month filter. I tried below custom table formula however it didn't work. Perhaps there is a better way to handle the exception indicated below
Table = CALENDAR(DATE(YEAR(TODAY())-1,MONTH(TODAY()),DAY(TODAY()+7)),TODAY())
Please, I created my table using hour time partition. Please, I would like would like to fetch data that was stored in my table in the last X minutes, eg last 5 minutes.
I tried using this command
SELECT *
FROM mydataset.mytable
FOR SYSTEM_TIME AS OF TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 5 MINUTE);
But, it returns a lot more rows than what is expected. I typically store 500 rows every 2 minutes, but this query is returning more than 30000 rows
As #Samuel mentioned in the comment, below example query can be considered to fetch data for a specific time period in BigQuery.
Select * from `dataset.table`
WHERE col_timestamp < TIMESTAMP_SUB(CURRENT_TIMESTAMP(),
INTERVAL 5 MINUTE)
Posting the answer as community wiki for the benefit of the community that might encounter this use case in the future.
Feel free to edit this answer for additional information.
Run into a problem.
I need to have a yearly reset on an entitlement value but the reset point is a different date for each person. A person gets entitlement 6 months after their start date, so that +6months date is when they should have a reset. Here is the +6months column i made:
+6m = DATE(YEAR('Employee List'[Emp. Dates]),
MONTH('Employee List'[Emp. Dates])+6,
DAY('Employee List'[Emp. Dates]))
I ran into this problem when i saw that the excess was taking away a persons entitlement by the amount of days took throughout their entire time at the company. Here is the measure for Excess:
Excess Days = CALCULATE(SUM('Employee List'[Entitlement]) -
SUM('Leave combined'[Duration1]))
This is taking away all recorded days even from a few years ago.
Can i make a measure that pretty much says if we are past the +6months date (DD/MM(Year doesnt matter i just need the same time every year)), then it will take away their entitlement from that +6months date and will not pick up any days from before the +6months reset?
e.g. if the +6months is March 1 2015, currently it is taking all the days took from their start to today, i want it to reset at every year on March 1 so it would only take data from March 1 2019 - today and will reset once we past March 1 2020. If a person has took 25 days a year on average and they only have 20 days entitlement, it should show they 5 excess days but for me now it is taking the total days took away from their entitlement.
I understand this is hard to explain and you may not get it but i am happy enough to explain.
Dummy Sheet
It is easier to create a calculated column to do this. Perhaps something like:
Calc =
VAR Person = TableA[Person]
VAR Date1 = EDATE(Entitlement[Start Date],6)
RETURN IF(TableA[Start]<Date1,0,TableA[Duration])
This column will make the duration 0, wherever the start date is lesser than the +6m date from entitlement. Once this column is created you can simply sum up the field at whatever level required, to get the desired output. Hope this helps.
I want to delete data from red shift table older than 10 days - is there any script to automate this process
There is no in-built function for deleting data older than a given period.
However, you could regularly run an SQL command that deletes rows based upon the date range of a particular column. For example:
DELETE FROM table WHERE date_field < CURRENT_DATE() - INTERVAL '10 days'
Need help in Data Analytics Calculations.
Currently, I am getting historical data for consumption as follows:
on above data, I am adding custom columns for calculating exact consumption(gallons) in no. of days. like:
Now, I have to plot month wise bar chart for consumption of respective Meter ID in 2016 year. But problem here is, I will have to calculate Every months consumption by dividing it in days in each respective month of 2016, and then only I will able to plot them monthly like:
y axis = consumption in every month
x axis = Jan Feb March Apr May Jun Jul Aug Sep Oct Nov Dec
so, in jan month, consumption should be = 10 + 100 + ((115/38) * 7) gallons
Notes: here, in ((115/38) * 7) : we are calculating avg consumption of single day 7 days in Jan and whole march and then getting last 7 day consumption of Jan so that we can add it in calculation of total consumption of Jan month
but how to add measure/custom column/new table for these calcualtions?
Thanks
What you need to do is relatively complicated, but the summary of my solution is:
Calculate the per-day consumption
Calculate the start and end date of each reading (e.g. the previous reading date plus one day, and the reading date)
Expand your data to have 1 row per day rather than 1-row per reading
You want to do these steps before you load the data into your data model (i.e. in your source system, or as the data is loaded using the Query Editor/Power Query).
Below, I assume you're using the Query Editor/Power Query. However, if you can use your source system, it's often the better choice (since the source system may be a database that is vastly faster than your desktop).
Note that your No. of Days calculation doesn't make sense to me. There are more than 38 days between 24 Jan 2016 and 31 Mar 2016. There are also more than 13 days between 10 Jan and 24 Jan. For this reason, it was difficult to tell whether you wanted a new reading to count on the day the previous reading was taken, or on the next full day. I assume the former. Also note, I've proceeded on the basis that your No. of Days calculation is correct
Calculate the Per Day Consumption
This is the easiest step, given that you have already calculated the Consumption and the No. of Days. Just divide one by the other. In the Query Editor, you can click in the Consumption (gallons) column and select Add Column > Standard > Divide. Under Value, choose Use values in a column and then select the No. of Days column.
Calculate the Start & End Date of Each Reading
The date of the reading is the end date, so you can rename Date to be End Date (since a reading is applied retroactively).
For the start date, in the Query Editor, you will need to add an index column (Add Column > Index Column). You will want to make sure your data is sorted by Meter ID and Date Ascending before doing this. Call the column Index.
Next, Add Column > Custom Column and pull the reading date from the prior row. Call the new column Previous End Date for now.
// A try is necessary because we can't get the previous row if there is no previous row (we'll get an error, which we can handle in the 'otherwise' block)
try
if
// See if the previous row is for the same Meter ID
[Meter ID] = #"Reordered Columns"{[Index] - 1}[Meter ID]
then
// If it is, grab the Reading Date from the previous row
#"Reordered Columns"{[Index]-1}[End Date]
else
// If this is the first reading for a meter, calculate the Start Date by subtracting the No. of Days from the End Date
Date.AddDays([End Date], -[No. of Days])
otherwise
// If this is the first row in the table, also calculate the Start Date by subtracting the No. of Days from the End Date
Date.AddDays([End Date], -[No. of Days])
Next, you'll want to add 1 to the Start Date, as we want the reading to apply to the day after the previous reading, not on the day of the previous reading.
Note, if you want the reading date to count in the prior period, subtract 1 from the End Date rather than add 1 to the start date (previous end date).
Expand your data to have 1 row per day
At this point, you should have a Meter ID, Start Date, End Date, and per day consumption column that reflects what you expect (i.e. the per day consumption is correct for the date range).
The final step is to duplicate each row for each date in the date range. There are several solutions to this outlined in this thread (https://community.powerbi.com/t5/Desktop/Convert-date-ranges-into-list-of-dates/td-p/129418), but personally, I recommend the technique (and video) posted by MarcelBeug (https://youtu.be/QSXzhb-EwHM).
You should end up with something more like this (after some removing & renaming of columns):
Finally
Now that you have one row per meter & date, with a per day consumption already calculated, you can build a visual. For example, you could do a column chart with Date on the Axis, and Consumption per Day as the value. By default, Power BI will recognize that Date is a date, and will roll it up by Year-Quarter-Month-Day. Press the little 'x' by Year and Quarter, and you'll have a chart that sums up the per day consumption by month. You can also drill down to individual date.
Further Reading
Reading a value from a previous row in Power Query
If Statements in Power Query
The AddDays function in Power Query
Adding Comments in Power Query
Catching Errors in Power Query
Converting a date range into a list of dates (Marcel Beug's solution)
A similar problem I previously answered