I am at the very beginning of a project where I need to load data from 3 separate SAAS services to a data warehouse on AWS. Each of the 3 SAAS applications offer open RESTful APIs to retrieve data.
The requirement is to extract data from these web services and load to a new data warehouse in AWS. The limitation is that there doesn't appear to be any off-the-shelf API connectors in AWS Glue.
So, in short, are there any GNU ETL tools that run in AWS that can connect to an external API for extraction purposes (GET methods) and then to the transform and load to a cloud-based data warehouse?
As a side note, we're not married to AWS, so if there is a solution in Azure or any other IAAS companies that would be considered.
I've only done diligence at this point. I'm looking into cloud data warehouse options with ETL tools that support REST API extraction.
It sounds like you want something like Amazon AppFlow - https://aws.amazon.com/appflow/ From the docs "Amazon AppFlow can directly extract data from some Services and it is directly integrated with AWS Glue DataBrew for Data Preparation and transformation."
You don't say what SAAS offerings you are using or what the data movement model you are desiring so this may not be the best option. There are many data ingestion options at AWS.
Related
What is the equivalent Google Cloud Platform product related to AWS Data Exchange
https://aws.amazon.com/data-exchange/
I've looked in the official Google Cloud Platform documentation that compares AWS and GCP products and they don't mention the AWS Data Exchange product at all.
So, most likely there isn't an equivalent GCP product to AWS Data Exchange.
That said, BigQuery supports third party data transfer, so maybe this is closest to what you're looking for:
Third party transfers for BigQuery Data Transfer Service allow you to automatically schedule and manage recurring load jobs for external data sources such as Salesforce CRM, Adobe Analytics, and Facebook Ads.
AWS Pinpoint Analytics appears to have replaced Amazon Mobile Analytics. In Mobile Analytics, you were able to create custom dashboards.
I'm struggling to find the feature in AWS Pinpoint. I'm assuming it's in there somewhere, but alas, I haven't found it yet.
#D.Patrick, you can create custom dashboards with Pinpoint data but not directly within Pinpoint console i.e You would need first to export your Pinpoint event data to a persistent storage (e.g S3 or Redshift) using Amazon Kinesis. Once in S3, you can use analytics tools to further analyze or visual the data. Such analytic tool offered by AWS include AWS Quicksight or AWS Athena. Other analytics(none-AWS) tools include Splunk
Check out the blog by AWS on this topic:
https://aws.amazon.com/blogs/messaging-and-targeting/creating-custom-pinpoint-dashboards-using-amazon-quicksight-part-1/
The 3 parts of this session describe in detail how to use Python 3, with AWS Lambda to create the custom dashboards.
I heard Athena is data analytics service from AWS which provides the same features like big query. Can we use Athena as alternative product for bigquery?
Athena is often used as a SQL layer for structured data in S3 such as formatted logs rather than a production DB like BigQuery which brings built-in multi-region support, etc. The AWS equivalent would still be a dedicated DB, either Postgres for SQL or Dynamo for NoSQL.
I am new to Azure Data Factory (ADF) and would like to know whether it is technically possible to use ADF to copy data from a source in AWS (not Azure) environment and put it to a sink in another AWS environment. I am aware that we need Integration Runtime (IR) to connect to the source. Can we achive copying to AWS as well using IR?
According to this document
Data stores with * can be on-premises or on Azure IaaS, and require you to install Data Management Gateway on an on-premises/Azure IaaS machine.
But this does not say that we can/cannot transfer to AWS environment.
You are referencing ADF V1 doc. You could reference ADF V2 doc for ADF V2 support more data store.
Currently, ADF V2 support Amazon Marketplace Web Service as Source, but not sink. But you could take a look of generic ODBC if you have odbc driver for your aws sql server.
I'm looking for a tool that will allow me to query my Redshift cluster via a REST API? I'm building some analytics UI and would rather not have to stand up a separate server in order to query my instance.
Any suggestions would be much appreciated.
You could use boto SDK to describe redshift clusters. (describe_clusters)
http://boto.cloudhackers.com/en/latest/ref/redshift.html