I have two platforms and what I want to do is, when I am using Qlik Sense, I want to tell SAS to do the some regression analysis on the table that I loaded on Qlik Sense and show me the results in Qlik Sense. Is it possible interconneting these two software?
Note: SAS ODBC connector is available but I am not sure if installing this will allow me to use SAS scripts on Qlik Sense Editor. It seems this tool only allows me to see SAS table on Qlik View. I am not sure if it allows me to send commands to SAS like regression, decision tree etc..
Related
Years ago I used a BI product called Hyperion Interactive Reporting. It allowed me to connect to a data source and create data models from which I would create reports. So far, sounds like Power BI right?
It also had the ability to connect to a metadata repository database. This database would contain data that mapped the actual, often cryptic, table and column names in the database to human-readable, business terms. For example a column that I saw in Hyperion as "Cost Center" may have been in the database as costCenter, work_order, or PROJECT-NUMBER. (It would also allow me to define the default join paths, but let's keep this question smaller.) This provided a way to make report development easier.
In Power BI, I see that I can manually rename columns, one-at-a-time. (And each time I touch something minor like this, Power BI takes several seconds to validate the entire file.) But I also see the need for many models that use the same data sources. So I may be defining the "Cost Center" column a few hundred times (a handful of reports per data set to answer a specific type of question, a few data sets that need Cost Center because the transformations in the model will be different for each type of question, several different combinations of data sources that include Cost Center, etc.)
Is there a way to connect Power BI to a metadata repository? Is there a way in Power BI to say, "Across all of my models/datasets, if I'm using the costCenter column from the Financial database, display Cost Center to the user"?
With about 20,000 columns in my data warehouse and 20,000 reports in my current reporting system, this could become a big deal if we intend to migrate to Power BI.
TLDR; There isn't an easy way to achieve this. What you have now is probably better than you could achieve without a ton of work. If you do try it, use SSAS instead of Power BI Desktop to author models.
Does Power BI have a metadata repository? No. There are tools that can get metadata from Power BI models, but you would have to manually build the metadata repository. If you want a centrally managed environment like this, I would highly recommend using SQL Server Analysis Services (SSAS) for on premise, or even better, Azure SSAS in the cloud. (Azure SSAS will get new features sooner than SSAS installed on premise.) While Power BI Desktop is a great self-service tool, I wouldn't author in it if I needed to control or report across the environment. There just aren't easy ways to corral all of the Power BI models in a report and it's a much more manual process. SSAS will need IT Support and is a higher cost and you will hit more issues than Power BI Desktop, but you will need it if you need central control. It's possible that more management controls will be added to the PowerBI.com service over time, but as of November 2021, you can't do this easily.
So what's the difference between Power BI Desktop and SSAS? The same Power BI engine in Power BI Desktop also exists in SSAS. When you start Power BI Desktop, it's actually starting a SSAS instance behind the scenes. Using SSAS directly just makes it easier for you to connect to the database behind the scenes and see all the models in the environment from one place, while Power BI Desktop doesn't let you peak behind the scenes and it only loads a single model at a time.
How do you get the metadata? It is an easy thing to get SSAS metadata using Power Query (or any SQL tool) to pull Direct Management View (DMV) data. DMVs are management tables that hold all of the metadata of the model, and you just use SQL commands to get the data. Search on "SSAS DMV" to learn more. I have a Excel file that uses Power Query to pull all the key DMV views for all our models in our servers. It makes it easy to do the kind of analysis as in your example.
For Power BI Desktop, you can connect to the hidden SSAS instance and do the same thing, but the report has to be open to do it, and there is no easy way to refresh the data--you pretty much just repeat the process each time. You will connect via localhost:port_number, and the port number is randomly created each time you start Power BI making it impossible to refresh the data pull. There are External Tools such as DAX Studio, Power BI Helper, and dataMarc's Document Model that make that easier, but there's no easy way to automate building the metadata repository for Power BI Desktop files. I would use SSAS directly rather than trying to automate building a large metadata repository.
What about making changes to models? To my knowledge, there aren't any tools that make it easy to make changes across models, though again, you could manually build them. I don't think I would trust my own tool to automate changes across models though. There's just too much that could go wrong. But if you had the need and the budget, you could build it. Look at tools like Tabular Editor, ALM Toolkit, and Microsoft's SSMS, and read on DevOps pipelines for automating updates. These tools work against SSAS and Power BI Desktop, but again, you have to open the Power BI files to work with those models, which makes automation that much harder to do.
Note that all the external tools I've mentioned except Tabular Editor v3 are free (though Tabular Editor v2 is free). PowerBI.tips is a great place to install all these tools from a single installer.
I need to automatically extract raw data of a PowerBI visualisation across multiple published reports.
Why not just pull the underlying dataset? Because the visualisations are using anomaly detection features of PowerBI, which include anomaly flags not available in the underlying dataset (basically, the visualisations contain calculated columns that are not included in main PowerBI data model)
Ideally a REST API solution would be best, but dumping CSV files or other more roundabout methods are ok.
So far, the closest functionality I can see is in the Javascript API here - https://learn.microsoft.com/en-us/javascript/api/overview/powerbi/export-data, which allows a website to communicate with an embedded PowerBI report and pass in and out information. But this doesn't seem to match my implementation needs.
I have also seen this https://learn.microsoft.com/en-us/azure/cognitive-services/anomaly-detector/tutorials/batch-anomaly-detection-powerbi which is to manually implement anomaly detection via Azure Services rather than the native PowerBI functionality, however this means abandoning the simplicity of the PowerBI anomaly function that is so attractive in the first place.
I have also seen this StackOverflow question here PowerBI Report Export in csv format via Rest API and it mentions using XMLA endpoints, however it doesn't seem like the client applications have the functionality to connect to visualisations - for example I tried DAX Studio and it doesn't seem to have any ability to query the data on a visualisation level.
I'm afraid all information on PowerBI says this is not possible. The API only supports PDF, PPTX and PNG options, and as such the integration with Power Automate doesn't do any better.
The StackOverflow question you link has some information on retrieving the Dataset but that's before the anomaly detection has processed the data.
I'm afraid your best bet is to, indeed, use the Azure service. I'd suggest ditching PowerBI and going to an ETL tool like DataFactory or even into the AzureML propositions Microsoft offers. You'll be more flexible than in PowerBI as well since you'll have the full power of Python/R notebooks at your disposal.
Sorry I can't give you a better answer.
I wanted to know like which concepts/topics I need to learn in order to work for a BigQuery DWH project? Along with Big Query, what other programming languages I need to get acquainted or expertise with(like python)? I am currently working as data enginner with ssis, informatica, power bi skills with strong sql. Please give your valuable suggestions.
Thanks,
Ven.
BigQuery has an SQL interface so if you don't already know SQL, learn it.
See the query reference.
Also, you can interact with BigQuery using Bash, with the bq CLI provided as a Google Cloud component in the gcloud CLI, or with Python, Go, Java, node.js... (choose your favorite).
Actually if you are not planning a long term project, or become an expert of BigQuery, the more complex concepts are not needed. In case you want to know more about it I link a pretty interesting blog.
To sum up:
Learn SQL
Take into account that BigQuery is optimized for reading and performing analysis, it is not a common database (do not exceed with writes)
Most common languages has a bigquery client, so you don't need to learn any new language.
Is it possible to create simplified version of Tableau (Just basic visualization) using Google Chart Tools? If so, what are main challenges in imitating Tableau?
What are the advantages of Tableau over Google Chart Tools?
I would recommend reading a bit what Tableau and Google Chart actually does.
Google Chart is a library to create visualisations. Basically you need to have your data ready in the correct format and then with the API you can create and customiza a graph to publish on the web.
Tableau on the other hand gives you the possibility to directly connect to different data sources, blend and join these sources and then create dashboards that give you an overview of whatever data you would like to view.
So to answer your first question
Is it possible to create simplified version of Tableau (Just basic visualization) using Google Chart Tools?
Yes it is possible to create basic visualisation with Google Charts (that's what it's made for) but that's also possible with Excel (although that won't give you the interactivity online).
If so, what are main challenges in imitating Tableau?
You will not be able to join and blend data. You also won't have a fairly easy to understand user interface, since Google Charts is handled on code level. You will also not be able to just "play around" until you see something interesting, at least not as easily as in Tableau
What are the advantages of Tableau over Google Chart Tools?
that depends on your use case. If you want to get some pretty graphs on your website, Tableau won't be able to do the job and you shoudl use Google Charts.
If you want to have complex, interactive dashboards to examine your data from different sources, you will struggle getting all of that together just with Google Charts and you should probably have a look at Tableau.
We have datasets that are created and stored in SAS. I'm curious if it is possible to access those datasets from a remote SQL client. Why? To make those datasets more broadly accessible within our organization.
Yes, you can license a product called SAS/SHARE that includes something called SHARE*NET. This is a very useful product that typically is installed in a BI server environment but I suppose it's possible to run on a local desktop.
Basically, you "register" SAS libraries to a service which then makes the data available to external clients over ODBC. This makes the data sets available as "tables" for applications like Excel, so I'm sure you can use other clients as well.
The SAS ODBC driver itself does not require a license, but the SAS/SHARE software does. I use it to make data available to many users who do not have direct access to my UNIX server.
It might be possible through SAS/ACCESS (or something similar), but SAS datasets typically cannot be understood by third-party software.