I am working on creating a trend line for a daily count I am getting from a SQL query. Each time the query runs, I get a count and the current date.
I need a way to record the output of the query into a new table and continue to append each time the query runs. The appended table would looke like this:
Count
Date
250
10/12/2022
257
10/13/2022
220
10/14/2022
This table would allow me to create a trend line. I am also open to a different approach if there is a better way.
Have you tried to use power query append option.
Should solve your issue.
Related
I'm using Power Query in Power BI to retrieve tables from Dynamics 365 using the Dataverse connector in Import mode. I'm currently having a problem where I have a table that has 23 columns and 6M+ rows, and a lot of the keys are alphanumeric GUIDS (which I intend to replace with numeric keys). So far I have retrieved a base table from D365 (TableBase), which I referenced to another table (Table0) to do some basic transformation like changing text types to integers, replacing nulls with 0, and changing column labels. My next step would be to create 3 smaller fact tables referencing Table0 (Table1, Table2, Table 3) by filtering on a category code for each table.
I have two problems occurring:
When I create a dimension table and left join to the fact table on the GUID, then expand the table to only include the numeric key I created in the dim table, it gives an error saying the file size is too large.
I tried filtering Table1 to return a decreased number of rows (the current table has 6,213,553 rows, I want to get it down to 567,458) it also gives an error saying the file size is too large.
The error message I get is this: Microsoft SQL: Return records size cannot exceed 83886080. Make sure to filter result set to tailor it to your report.
Of course my alternative is to use the OData connector instead of the Dataverse connector for Dynamics 365, but it's dreadfully slow and one simple model I have build with the OData connector is not even refreshing anymore due to the refresh taking so long. Also, I'm confused as to why filtering a table to have fewer rows throws this error, as I'm trying to reduce the file size.
Do you all have any suggestions on why this is happening, and how I can get around it? I know the most obvious answer is that I should remove some columns from the fact table, and I'm going to have a talk with one of my consumers about doing this because that is the only solution I can figure out at this point.
I appreciate any help you can provide!
I need to calculate the time difference between two timestamp columns in my table and create a new column with the calculated values but as I'm doing this in the direct query mode, I have some limitations for it.
I tried datediff, duration.totalseconds etc but nothing really worked..
A header
Another header
2022-07-02T21:09:13
2022-07-02T18:10:00
2022-07-03T06:09:35
2022-07-03T03:10:00
2022-07-03T03:08:41
2022-07-03T00:10:00
2022-07-02T15:09:29
2022-07-02T12:10:00
2022-07-05T00:09:22
2022-07-04T21:10:00
2022-07-02T12:09:34
022-07-02T08:30:00
I was able to achieve the result I want through my sql query
select (unix_timestamp(timestamp1)-unix_timestamp(timestamp2))/3600 as hourly_diff from table1
hourly_diff
2.9869444444444446
2.9930555555555554
2.9780555555555557
2.991388888888889
2.9894444444444446
3.6594444444444445
2.9716666666666667
3.0055555555555555
This is the result I'm getting with my sql query and I want the same result with DAX..
Anyone could help me with this query?
Thanks in advnance
I am connecting PBI to Snowflake using DirectQuery. To keep it simple, I have two tables, a product dimension table and a sales fact table. There are 3.7M rows in the product dimension table and 100M in the sales fact table. I also have a measure that calculates total sales which uses SUM to sum a column in the fact table.
I create a table visual in PBI and put the product description as the first column. The query generated by PBI is good. It retrieves 501 rows and displays them. So far, so good. Next I put the total sales measure as the second column. Now PBI generates several queries retrieving 1,000,001 rows. Of course I get an error stating the 1M row limit for DirectQuery has been reached.
This should not be happening. Has anyone run into something like this? Is there anything I can do?
I had a dig around and there is a capability to adjust the limit if you have a premium license
https://powerbi.microsoft.com/en-gb/blog/five-new-power-bi-premium-capacity-settings-is-available-on-the-portal-preloaded-with-default-values-admin-can-review-and-override-the-defaults-with-their-preference-to-better-fence-their-capacity/
I have a SQL query to get the data into Power BI. For example:
select a,b,c,d from table1
where a in ('1111','2222','3333' etc.)
However, the list of variables ('1111','2222','3333' etc.) will change every day so I would like the SQL statement to be updated before refreshing the data. Is this possible?
Ideally, I would like to keep a spreadsheet with a list of a values (in this example) so before refresh, it will feed those parameters into this script.
Another problem I have is the list will have a different nr of parameters so the last variable needs to be without a comma.
Another option I was considering is to run the script without the where a in ('1111','2222','3333' etc.) and then load the spreadsheet with a list of those a's and filter the report down based on that list however this will be a lot of data to import into Power BI.
It's my first post ever, although I was sourcing help from Stackoverflow for years, so hopefully, it's all clear.
I would create a new Query to read the "a values" from your spreadsheet. I would set the Load To / Import Data option to Only Create Connection (to avoid duplicating the data).
Then in your SQL query I would remove the where clause. With that gone you actually don't need to write custom SQL at all - just select the table/view from the Navigation UI.
Then from the the "table1" query I would add a Merge Queries step, connecting to the "a values" Query on the "a" column, using the Join Type: Inner. The resulting rows will be only those with a matching "a" column value (similar to your current SQL where clause).
Power Query wont be able to send this to your SQL Server as a single query, so it will first select all the rows from table1. But it is still fairly quick and efficient.
I have 40 million rows in my dataset. Each day I may get an extra 100 rows. Obviously I don't want to have to import the whole 40 million each time I do a data refresh. Is it possible to do an incremental refresh where only the new rows are added?
I don't think incremental update as you describe it is possible yet.
It looks like you can push rows with Power BI REST API, if you're happy to switch to that.
However, you might find this workaround useful:
Split your table and query into two: where date <= 'somedate' and where date >'somedate'
Add an "empty query", use Table.Combine to join your two subtables. Use this as your main table.
Whenever you need to refresh, only refresh the second query (the one with where date >'somedate').
Every once in a while, when that second query starts taking a long time, change somedate to the current date and do a full refresh.
The feature has now been implemented and is called Incremental refresh. Currently it is a premium only feature.