Live connections
When connecting to SQL Server Analysis Services,
there's an option to either import data from or connect live to, the
selected data model. If you use import, you define a query against
that external SQL Server Analysis Services source, and the data is
imported as normal. If you use connect live, there's no query defined,
and the entire external model is shown in the field list.
My understanding was that import/direct query can be used to query a data source like SQL server. Where as live mode is used to connect to an existing power bi dataset, or SSAS, or azure analysis service.
The above quote says When connecting to SQL Server Analysis Services, there's an option to either import data from or connect live to, the selected data model. - So does this mean that the live mode allows us to choose between import/direct query to the live model?
When connecting to SQL Server Analysis Services, there's an option to
either import data from or connect live to, the selected data model.
This quote means that you do not have to use a live connection with SSAS models. Instead you can construct a query and import the data. The dataset would be limited to whatever was in the query. If you use a live connection, you have access to the entire model.
Related
I can connect to shared dataset using live connection.
Suppose I want to use more than 1 shared dataset, or want to add additional data sources along with shared dataset, then there is a new feature recently launched for such functionality where-in the connection changes from live to direct query mode.
However, this link https://learn.microsoft.com/en-us/power-bi/transform-model/desktop-many-to-many-relationships#considerations-and-limitations mentions Power BI dataset as a limitation.
I want to ask- what exactly is the limitation they are referring to? Does it mean that - when there is a m:m relationship in a model, and if we use DQ to connect to that model with the intention to also connect to another data source (or model) in import or DQ mode, then this isn't supported when the dataset we have connected to has got a m:m relationship?
Does the limitation apply when connecting to another dataset in live mode?
as per your link:
The following Live Connect (multidimensional) sources can't be used
with composite models
Analysis Services can host both tabular and multidimensional models (see here: https://learn.microsoft.com/en-us/analysis-services/comparing-tabular-and-multidimensional-solutions-ssas?view=asallproducts-allversions). The way I understand the article you posted, you cannot use composite models (and m:m) with the multidimensional models.
I've connected to the tabular models hosted in AAS via DirectQuery (adding tables and m:m relationships) and never had issues, so if you have a tabular model these limitations shouldn't apply to you
This is what I am trying to do: I have various SQL server databases with data. I created views in all of them. All views will need to be imported, and I specify their relationships. I want this to be refreshed nightly. I want to build various reports of the same data source.
Do I have to use a PowerBI desktop application to import data into PowerBI Report Service? [I have done this so far, but then can create new reports in the cloud on existing data. It would make sense to connect directly from PowerBI report service to my SQL servers.]
Once I uploaded data using a desktop application (as I have done so far), how can I view the data model in the report service once it is uploaded in the cloud?
In order to get routinely refreshed data I need to setup a gateway. Is the local PowerBI desktop application still involved in this process, or could I [in theory] delete the local desktop application that pushed the data in initially?
For your questions:
You have two options, use PBI Desktop to connect to the data using import/direct query, then load it to the service. You can use dataflows to create an import based on your views, but you will then need to create reports from those. Using dataflows, you'll have to set up a refresh schedule, then for the dataset(s) built on top of those, you'll have to set another refresh schedule.
You will be limited to the dataset sizes of 1GB for the workspace if importing data. You cannot use direct query on dataflows (unless you have enhanced compute with PBI premium). Once the dataset is loaded, you can then create new reports in the service or via desktop on top of that dataset. If possible it is recommended to use direct query.
To see the data model, you can use desktop to connect to PBI Service Dataset. This will connect in 'Live Connection' mode, and will be limited to that one dataset, you can't add others to it, Excel, CSV, SQL etc. You can also use Analyse in Excel, a plugin for Excel, that can connect to the data model. You can create new reports in the service for existing data models as well.
When creating the report in PBI Desktop it does not use the Gateway, you connect to your data sources as normal, then once you load the dataset to Power BI it will match the data sources in the file to the ones set up in the Gateway Admin settings. So you will still need PBI Desktop to create reports, but the gateway is there for the refreshing. The Desktop is not used in the process for refreshing. You could delete the workbook or application, but if you have to make changes, what will you refer to? (You could download a copy of the report from the service).+ It is easier to make changes in the desktop app, then the service, as there is a feature difference between dataset creation in the desktop vs service.
Currently we have a problem with loading data when updating the report data with respect to the DB, since it has too many records and it takes forever to load all the data. The issue is how can I load only the data from the last year to avoid taking so long to load everything. As I see, trying to connect to the COSMO DB in the box allows me to place an SQL query, but I don't know how to do it in this type of non-relational database.
Example
Power BI has an incremental refresh feature. You should be able to refresh the current year only.
If that still doesn’t meet expectations I would look at a preview feature called Azure Synapse Link which automatically pulls all Cosmos DB updates out into analytical storage you can query much faster in Azure Synapse Analytics in order to refresh Power BI faster.
Depending on the volume of the data you will hit a number of issues. First is you may exceed your RU limit, slowing down the extraction of the data from CosmosDB. The second issue will be the transforming of the data from JSON format to a structured format.
I would try to write a query to specify the fields and items that you need. That will reduce the time of processing and getting the data.
For SQL queries it will be some thing like
SELECT * FROM c WHERE c.partitionEntity = 'guid'
For more information on the CosmosDB SQL API syntax please see here to get you started.
You can use the query window in Azure to run the SQL commands, or Azure Storage Explorer to test the query, then move it to Power BI.
What is highly recommended is to extract the data into a place where is can be transformed into a strcutured format like a table or csv file.
For example use Azure Databricks to extract, then turn the JSON format into a table formatted object.
You do have the option of using running Databricks notebook queries in CosmosDB, or Azure DataBricks in its own instance. One other option would to use change feed to send the data and an Azure Function to send and shred the data to Blob Storage and query it from there, using Power BI, DataBricks, Azure SQL Database etc.
In the Source of your Query, you can make a select based on the CosmosDB _ts system property, like:
Query ="SELECT * FROM XYZ AS t WHERE t._ts > 1609455599"
In this case, 1609455599 is the timestamp which corresponds to 31.12.2020, 23:59:59. So, only data from 2021 will be selected.
After setting up my ETL, now I'm preparing the reporting phase. I want to know the diffrence between Import and Connect live a SQL Server analysis Services database methods knowing that I'll be working with measures.
In my example : I need to create a measure which count the emails with failed status by department.
I have (after importing the cube) :
Fact Mailing Count
Mail Status Dimension
Message Template Dimension (having the application name which is the
same name of the department)
Live connect: This will connect power bi to analysis services data model directly.so you will be building your model completely on analysis services project and deploying it frequently. So the power bi will have a live connection to the model and make updates when necessary. when new data processed or new measures or tables created. Here the limitation is you cannot combine multiple sources of data and you have to rely on the SASS Model you have already connected with.
Import: This will import tables to the Power bi and you will be allowed to create and manipulate your facts and dimensions as per your wish inside the power bi(in live connect mode you have to do it in analysis services model itself).
The main obstacle of import mode is When you import large tables will a large number of data, power bi has limitations(1GB) on that.
Creating measures and calculated tables are allowed in both modes.the difference is on which side you create them.
A detailed comparison of both Live connect and Import
https://community.powerbi.com/t5/Community-Blog/Power-BI-Live-connection-vs-Import-comparison-and-limitations/ba-p/84377
I am planning to automate the KILL of user queries running over a configurable amount of time. I see that
sys.dm_pdw_exec_sessions
has the login_name, but the client_id column looks to me like an Azure IP (104.*) and not the IP (10.*) from my company's servers/PCs.
Is it possible to know the client context for an ADW query?
If you connect from your laptop to Azure SQL DW the IP will usually match your public IP. Compare to http://www.myipaddress.com and see if it matches. It won't be your internal 10.* IP address because you can't put Azure SQL DW on a VNET at the moment.
sys.dm_pdw_exec_sessions also has app_name, which gives some good details, for example:
'Mashup Engine' in the above example is Power BI Desktop.
You could also instigate a policy that all queries had to use the LABEL syntax to give additional query context, kill queries that don't comply? See below for additional details:
https://learn.microsoft.com/en-us/azure/sql-data-warehouse/sql-data-warehouse-develop-label