I'm new in the BI field and I have to develop the BI of over 25 million records.
I'm using DirectQuery to import the data to PowerBi and to web. But the dashboard loading is too slow. And sometimes the dashboard won't load.
I'd like to know what's the best way to import big amount of data to PowerBI and to publish in the web after the fact.
DirectQuery, Analysis Service, what else?
Thanks for the answers.
you can do
1) make aggregation on data and reduce size of data.
2) bring only useful column only- that reduce size again
3) data is (real time/permutation -combination) only then use direct qry
4) if spark --you can connect to directly to spark
Related
Just looking for a pointer as to the best way to go about this.
I'm comfortable with Power BI Report Builder (SSRS experience), but am pretty much a Power BI novice.
Basically, we have to create a Paginated (non-interactive) report for client consumption. It's going to be large, have multiple datasets, and use parameters / presence of data in the data sets to group data and/or turn sections on or off.
Not too much visualisation - some illustrative graphs and tables here and there - and quite a bit of text, some of it with data / text inserted via placeholders from the various datasets.
There are 3 Azure SQL databases I need to combine data from for this, (split roughly into config, data and results).
In SSRS / SQL Server, I would have used one of my databases as the data source, and written a stored procedure per SSRS data set, joining to tables in other databases in the stored procedure query.
Then in Report builder just set up the data sets joining to the stored procs and gone from there.
On Azure SQL Server, I think I've got 2 options:
write elastic queries so I can bring in the data I need from each database, but just query on one database.
Build a Power BI Model / Dataset that joins the relevant tables from the 3 databases together, publish to power bi service and use that as my datasource.
What's the best solution for my reporting scenario?
Cheers
In my scenario, Databricks is performing read and writing transformations in Delta tables. We have PBI connected to the Databricks cluster that needs to be running most of the time, which is expensive.
Knowing that delta tables are in a container, what would be the best way in terms of cost x performance to feed PBI from delta tables?
If your set size is under max allowed size in PowerBI (100 GB I guess) and daily refresh is enough you can just load everything to your PowerBI model.
https://blog.gbrueckl.at/2021/01/reading-delta-lake-tables-natively-in-powerbi/
If you want to save the costs maybe you don't need transactions and can save it in csv in data lake, than loading everything to PowerBI and refresh daily is really easy.
If you want to save the costs and query new incoming data all the time using DirectQuery consider using Azure SQL. It has really competitive prices starting from 5 eur/usd. Integration with databricks is also perfect write in append mode do all magic.
Another option to consider is to create an Azure Synapse workspace and use serverless SQL compute to query the delta lake files. This is a pay-per-the-TB consumed pricing model so you don’t have to have your Databricks cluster running all the time. It’s a great way to load Power BI import models.
I'm new to Data visualization and currently I'm migrating couple of dashboards from tableau to powerBi.
Both tableau and powerBi imports data from sql server.
Which is the best way to create an efficient data model in powerBi while taking into consideration of views from tableau?
The best way to create a data model in Power BI is to create a Star Schema. If you are not familiar with a Star Schema do some research on the web. It will be something that it is well worth your while to learn.
You may end up having to use Power Query to deal with tables from both sources that hold similar data.
I’m pretty new to Power BI and am still at the point of assessing whether it will meet our needs.
I’ve got as far as realising that when creating a new report I can either Import tables (I’m using SQL Server) and use a Direct Query.
The particular report I’m trying to report is quite resource intensive. To create the report in TSql requires iterating through hundreds of thousands of rows in multiple tables in a cursor and then storing some data in a temp table which is the output of the query. I’m very concerned about using the Direct Query option for this because of potential performance degradation on the server.
Is it possible in Power BI Desktop to Import the 5 tables that are used in my query, and then somehow write my query against these tables? That way (in-theory) the query wouldn’t be sent directly to our server each time someone views the report.
My question is based on my lack of knowledge of Power Bi so I may be asking something that is completely impossible!!
Thanks in advance for your help
Regards
Dotdev
That's exactly what Import option does. It imports the tables only once (unless you refresh or change your query). The viewer would be looking at the data that was extracted upon import and packaged into the PBIX file rather than a direct connection to the database.
I have connected to Azure Document DB by Power BI ,but it is taking too much time for data to load and even more time to apply the queries ...Is there any way to reduce this data loading time??
This is more of a generic question around Power BI and even broadly BI tools. But, in general, you have to specify more filters to the queries (yes, they can be edited even in plain SQL). Azure Cosmos DB is super-fast, it all depends on how much data you're trying to query. Also, make sure data is in the region from which users are accessing it in Power BI.