I heard Athena is data analytics service from AWS which provides the same features like big query. Can we use Athena as alternative product for bigquery?
Athena is often used as a SQL layer for structured data in S3 such as formatted logs rather than a production DB like BigQuery which brings built-in multi-region support, etc. The AWS equivalent would still be a dedicated DB, either Postgres for SQL or Dynamo for NoSQL.
Related
I am at the very beginning of a project where I need to load data from 3 separate SAAS services to a data warehouse on AWS. Each of the 3 SAAS applications offer open RESTful APIs to retrieve data.
The requirement is to extract data from these web services and load to a new data warehouse in AWS. The limitation is that there doesn't appear to be any off-the-shelf API connectors in AWS Glue.
So, in short, are there any GNU ETL tools that run in AWS that can connect to an external API for extraction purposes (GET methods) and then to the transform and load to a cloud-based data warehouse?
As a side note, we're not married to AWS, so if there is a solution in Azure or any other IAAS companies that would be considered.
I've only done diligence at this point. I'm looking into cloud data warehouse options with ETL tools that support REST API extraction.
It sounds like you want something like Amazon AppFlow - https://aws.amazon.com/appflow/ From the docs "Amazon AppFlow can directly extract data from some Services and it is directly integrated with AWS Glue DataBrew for Data Preparation and transformation."
You don't say what SAAS offerings you are using or what the data movement model you are desiring so this may not be the best option. There are many data ingestion options at AWS.
I am trying to migrate data from Azure Dedicated SQL Pool (formerly SQL DW) into S3.
My initial approach was to do so using AWS DMS. However, DMS doesn't have an endpoint for Azure SQL Pool. I am also aware of other ways like sending data from SQL Pool into Azure Blob Storage and then migrating from Blob Storage to S3.
I am however looking for some kind of 'direct transfer' by which I can pull data straight from SQL Pool into S3. Or, some way I can push from SQL Pool into S3.
Is there any way this can be done?
If not, which would be the most efficient way to migrate this data?
Does Google Cloud SQL support column-level encryption?
I know that it is possible for BigQuery tables but not sure about Cloud SQL!
link
It's not a out of the bow feature on Cloud SQL, you need to do it manually when you read and write the data. You can use Cloud KMS for hat.
With BigQuery, keep in mind that you need to keep the key in BigQuery also and only the IAM permission allow to access or not to the keyset.
Eventually, all the data are encrypted at rest, but I'm sure that your use case is for a specific column, not for the whole database.
I need fetch the records using ROW_NUMBER(). Is there any possibility to achieve this using Google Cloud Datastore?
Cloud Datastore is a NoSQL database. On the other hand, ROW_NUMBER is a standard SQL numbering function which is supported by Google’s BigQuery service that supports querying using ANSI SQL.
I am trying to use AWS DMS Database Migration Service along with Snowflake as a source database. Is there any way I can achieve this ?
All I could see options for IBMDB2, MySQL, SQL Server, Amazon Aurora, Oracle, SAP Sybase etc. But not for Snowflake.
Can ODBC string for SnowFlake be put in as a source endpoint ? Or any workaround
Because DMS doesn't support Snowflake as destination yet so I think you could use S3 as target then use
Snowflake bulkload to load data from S3 https://docs.snowflake.com/en/user-guide/data-load-s3-create-stage.html
Snowpipe to do continuous loading.