I'm looking for a tool that will allow me to query my Redshift cluster via a REST API? I'm building some analytics UI and would rather not have to stand up a separate server in order to query my instance.
Any suggestions would be much appreciated.
You could use boto SDK to describe redshift clusters. (describe_clusters)
http://boto.cloudhackers.com/en/latest/ref/redshift.html
Related
I am at the very beginning of a project where I need to load data from 3 separate SAAS services to a data warehouse on AWS. Each of the 3 SAAS applications offer open RESTful APIs to retrieve data.
The requirement is to extract data from these web services and load to a new data warehouse in AWS. The limitation is that there doesn't appear to be any off-the-shelf API connectors in AWS Glue.
So, in short, are there any GNU ETL tools that run in AWS that can connect to an external API for extraction purposes (GET methods) and then to the transform and load to a cloud-based data warehouse?
As a side note, we're not married to AWS, so if there is a solution in Azure or any other IAAS companies that would be considered.
I've only done diligence at this point. I'm looking into cloud data warehouse options with ETL tools that support REST API extraction.
It sounds like you want something like Amazon AppFlow - https://aws.amazon.com/appflow/ From the docs "Amazon AppFlow can directly extract data from some Services and it is directly integrated with AWS Glue DataBrew for Data Preparation and transformation."
You don't say what SAAS offerings you are using or what the data movement model you are desiring so this may not be the best option. There are many data ingestion options at AWS.
We are attempting to connect to an Amazon Redshift Instance from Azure Data Factory as a linked service.
Steps Taken:
Provisioned Self Hosted Integration Runtime (Azure)
Created user access to database within Redshift (AWS)
White list IP addresses of SHIR within security group (AWS)
Built linked service to Redshift using log in, server address and database name (Azure)
From testing we know that this user log in works with this database for other sources and in general the process has worked for other technologies.
A screenshot of the error message received can be seen here
Any suggestions would be greatly appreciated :)
To connect to Amazon Redshift from Azure, look at using the Amazon Redshift AWS SDK for .NET. You can use the .NET Service client to write logic that performs CRUD operations on a Redshift cluster.
You can create a service client in .NET with this code:
var dataClient = new AmazonRedshiftDataAPIServiceClient(RegionEndpoint.USWest2);
Ref docs here:
https://docs.aws.amazon.com/sdkfornet/v3/apidocs/items/RedshiftDataAPIService/TRedshiftDataAPIServiceClient.html
I am building a serverless application using AWS, with AWS API, AWS Lambda functions, and AWS RDS (database).
I have an existing MySQL schema (basically, a table dump), and I want to create the API automatically from this schema, ideally something that I can easily import into AWS API Gateway (like something from SwaggerHub or similar service).
Then, I want to have the operations for the database (CRUD operations that match the API) also automatically generated for NodeJs or Python, which I can then easily deploy to AWS Lambda, for example using SAM templates, or maybe just uploaded as a package somehow to AWS.
The lambda operations should be able to connect to my AWS RDS database, and perform the CRUD operations described by the API.
The idea is to determine some way to simplify this process. If the database schema changes significantly, for example, I do not want to manually edit a bunch of lambda functions to accommodate the new DB schema every time!
I'm wondering if anyone has any suggestions as to how I could make this work.
AWS Pinpoint Analytics appears to have replaced Amazon Mobile Analytics. In Mobile Analytics, you were able to create custom dashboards.
I'm struggling to find the feature in AWS Pinpoint. I'm assuming it's in there somewhere, but alas, I haven't found it yet.
#D.Patrick, you can create custom dashboards with Pinpoint data but not directly within Pinpoint console i.e You would need first to export your Pinpoint event data to a persistent storage (e.g S3 or Redshift) using Amazon Kinesis. Once in S3, you can use analytics tools to further analyze or visual the data. Such analytic tool offered by AWS include AWS Quicksight or AWS Athena. Other analytics(none-AWS) tools include Splunk
Check out the blog by AWS on this topic:
https://aws.amazon.com/blogs/messaging-and-targeting/creating-custom-pinpoint-dashboards-using-amazon-quicksight-part-1/
The 3 parts of this session describe in detail how to use Python 3, with AWS Lambda to create the custom dashboards.
I currently have a mysql database on AWS RDS.
I am in need of accessing and looking up information on a daily basis of the data in that database via an external CRM.
In an ideal scenario, I could create an API on Lambda that would have access to the RDS instance and have that be my front facing API that my CRM API could hit up and lookup and work with my mysql data.
My question is... is this the best way to do it? Is there some sort of service out there that facilitates me creating an API or do I need to create an API from scratch...
Any suggestions? Thanks a bunch!
You will have to create your own AWS Lambda app, but there is a framework - the Serverless Framework which will make the whole endeavour much easier!
This Tutorial on Accessing RDS from AWS Lambda might help as well.