I created a report in Power BI Desktop, connected to an AWS RDS database, and published it to the Power BI Web App, with an intent to refresh the dataset from the web app.
I tried doing so with both MySQL as well as SQL Server (on RDS). However, Power BI web does not let me refresh the dataset and instead wants me to install an on-premise Gateway.
I am not sure why this is a requirement as my database is on the cloud and not on-premise, and on a public VPC.
Is it possible to refresh an AWS RDS dataset in the Power BI web app? How?
To refresh the Power Bi Visuals on the Web Application, it needs to have a successful connection with the Database. Now, in the case of connecting to an AWS RDS instance, since the AWS database server is installed on an AWS Virtual Machine, this acts as an on-premise source on that machine [6]. Therefore you would have to install the on-premises gateway on the AWS server with an access to the RDS possibly on the same VPC or with the help of best security practices.
The steps to create an on-premise data gateway an AWS:
Create an EC2 Windows instance, please see this AWS documentation [3] for more details. Refer to this public Microsoft link [4] for details around the instance(s) based on the operating system requirements. Following are the instance configurations that have worked for me and can be different based on your requirements:
a. AMI Name: Windows_Server-2019-English-Full-Base-2021.10.13
b. Instance Type: t2.2xlarge
Please check the required network ports you will need to allow in your instance security group inbound and outbound rules. Please see this AWS documentation [5] to know how to work with security groups.
Ensure that you add a “Key Pair” to the instance and have access to the PEM file. This will be needed to RDP into the machine.
Once the instance is created, RDP into it and install any required software.
For instance in my case to connect to an AWS Postgres Server, I had to install the Npgsql version 4.0.9. The latest version unfortunately didn’t work so be mindful of version compatibility.
Also though not essential, I wanted to install the Chrome browser on the Remote Server as a preferred choice of my browser. To do so I had to run the following command on the PowerShell CLI
$Path = $env:TEMP; $Installer = "chrome_installer.exe"; Invoke-WebRequest "http://dl.google.com/chrome/install/375.126/chrome_installer.exe" -OutFile $Path$Installer; Start-Process -FilePath $Path$Installer -Args "/silent /install" -Verb RunAs -Wait; Remove-Item $Path$Installer
With the Remote Desktop open, install the Power BI on-Premises Gateway [4].
Now search for the “On-Premises Data Gateway” on the Remote Server and register your account. Please use the account that is connected to your PowerBi Web App. This could be either the role based email to which you and the Power Bi has access to … or your own email address associated with the PowerBi Web App.
For details on creating a data source, please see the reference link [6] for more details.
Data Refresh Scheduling
Now back on the Power Bi Web app, when you click on your settings button > manage gateways, on the left hand side you should be able to see the gateway showing up. Add the credentials and test if the connection is working. If you get a green tick, you should be able to schedule the Data Refresh on the web.
You can configure PowerBi to refresh data. Please see the reference link [7] for more details.
Troubleshooting
Pay attention to the error that is given when you try to connect to the database on the web. It would usually have a useful hint of what might be missing.
Reference links:
[3] Launch an instance using the Launch Instance Wizard https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/launching-instance.html
[4] Install PowerBi gateway - https://learn.microsoft.com/en-us/data-integration/gateway/service-gateway-install
[5] Work with security groups - https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/working-with-security-groups.html
[6] Connecting the Microsoft Power BI service to AWS data sources - https://docs.aws.amazon.com/en_us/whitepapers/latest/using-power-bi-with-aws-cloud/connecting-the-microsoft-power-bi-service-to-aws-data-sources.html
[7] Data refresh in Power BI - https://learn.microsoft.com/en-us/power-bi/connect-data/refresh-data
[8] https://learn.microsoft.com/en-us/power-bi/connect-data/service-gateway-onprem
[9] credits / author myself: https://www.smart5.co.uk/en_gb/article/25/on-premises-data-gateway-installation-aws-power-bi
Related
I am exploring Google Dialogflow and creating chatbot for learning purpose. I would like to store/fetch the details into the SQL Server DB in an local machine taken from the dialogflow chat session. Similar to Workato & Celonis tools, we have an On-Prem agent to install it in the respective machine which will create a tunneling to access it without affecting the machine firewall.
I tried looking in google documentation, but unable to get proper result based on my analysis. It would be great, if I get guidance/support on how to connect SQL DB hosted in local machine from Dialogflow Inline editor using On-Prem agent.
Please let me know if I need to add any other details on the mentioned scenario.
NOTE: Based on my google search, I came to know that we can write NodeJs code and create a webhook call by hosting with ngrok or storing the data in GCP Cloud SQL instances to achieve this. But wanted to how to save/fetch data in the local machine's SQL Server from dialogflow.
Thanks in advance.
We are attempting to connect to an Amazon Redshift Instance from Azure Data Factory as a linked service.
Steps Taken:
Provisioned Self Hosted Integration Runtime (Azure)
Created user access to database within Redshift (AWS)
White list IP addresses of SHIR within security group (AWS)
Built linked service to Redshift using log in, server address and database name (Azure)
From testing we know that this user log in works with this database for other sources and in general the process has worked for other technologies.
A screenshot of the error message received can be seen here
Any suggestions would be greatly appreciated :)
To connect to Amazon Redshift from Azure, look at using the Amazon Redshift AWS SDK for .NET. You can use the .NET Service client to write logic that performs CRUD operations on a Redshift cluster.
You can create a service client in .NET with this code:
var dataClient = new AmazonRedshiftDataAPIServiceClient(RegionEndpoint.USWest2);
Ref docs here:
https://docs.aws.amazon.com/sdkfornet/v3/apidocs/items/RedshiftDataAPIService/TRedshiftDataAPIServiceClient.html
I am working on a project where a user clicks a link/button that says Access VM on a webpage, it should internally spin up a Linux based VM (using GCP, AWS or Azure) and provide the VM terminal in a new browser tab for the user to play around in the VM.
How can I achieve this using GCP/AWS/Azure? Which type of VM should I create so that the user can access the VM terminal over a browser without using an SSH client?
I tried creating a VM on Azure and explored the Bastion option. But this Bastion session should always be initiated from within the Azure portal.
Do we have any other option within GCP, AWS or Azure to achieve this?
I am looking for something similar to katacoda website.
There's no built in feature in GCP that will allow such thing possible. There is a button "SSH" in the VM's list but you have to be able to view the list and also have the permission to connect to the instance. But that requires to actually log into GCP which I believe is not what you want.
**You could try and built some solution that after clicking an "Connect" button you your website would send a series of commands to GCP's API to create & connect to the new isntance. It's possible but rather complicated.
Have a look at the documentation how to connect to VM using browser - maybe it will give yolu some ideas.
Ultimately use many other 3rd party tools but you still need to provide an address and credentials - additionally you rely on a service that you don't control so you have to take security (and reliability) into consideration.
At the end you may also consider going through general information how to connect to GCP's instances.
I'm evaluating AWS Data Migration Services. I'm attempting to move data from an Azure SQL database to a SQL Server 2016 database sitting on AWS RDS. I've successfully created the source and was able to connect when I clicked the Run Test button. However, when I entered the Target database connection details information, I'm not able to connect when I click the Run Test button. The information and error message is below.
I am able to connect to this instance using SQL Server Management Studio, with the credentials I'm using in the screen shot.
For timeout concerns, security groups are usually the culprit. Can you verify if the security group of your Target RDS instance allows ingress from the security group that the DMS Replication Instance belongs to?
See the attached screenshot:
See this article for more information: https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Security.Network.html
I have a PowerBI instance and a Postgres database on Heroku. In my search the PostgreSQL connectors for PowerBI all seem to be for desktop:
https://powerbi.microsoft.com/en-us/integrations/postgresql/
https://powerbi.microsoft.com/en-us/documentation/powerbi-desktop-data-sources/
Has anyone found a connector for the cloud PowerBI solution?
Thanks!
The data source "connectors" for Desktop and the service "connectors" for the cloud are different things.
If you build a report in Power BI Desktop that connects to a PostgreSQL database data source, you can publish that report to PowerBI.com (the cloud) and configure a gateway to allow scheduled refreshes directly from the cloud (i.e. once configured, you can pull data from your PostgreSQL database via the gateway into the report in the cloud on a schedule). It sounds like this is what you were looking to do.
More information about the on-premises gateway, including the list that shows that PostgreSQL can be used via the gateway for a scheduled refresh: https://powerbi.microsoft.com/en-us/documentation/powerbi-gateway-onprem/#list-of-available-data-source-types
You cannot, however, start building a blank report in the cloud and connect to an on-premises PostgreSQL database. When you start out from the cloud, you can only connect to a more limited set of cloud-based data sources (Azure and SSAS). In general, you almost always want to start building your Power BI report in Power BI Desktop and then publish it to the cloud.
The service connectors in Power BI are something entirely different. They provide a quick way to connect to other cloud-based services you may subscribe to and bring in your personal data for reporting. E.g. if you use Salesforce, the Power BI connector will connect directly to Salesforce for you. This is different to connecting to your own database directly. It's not your database you're connecting to, but your data that belongs to a 3rd party service that's already in the cloud. More information here: https://powerbi.microsoft.com/en-us/documentation/powerbi-content-packs-services/
I hope this helps clarify things.