We want to decommission the azure lake and maintain just aws moving forward. Issue is several of my customers use synapse that sources from azure lake. We are hoping to give them something closer to the synapse tool but the data would come from the s3 bucket. Any ideas?
These clients do not have aws accounts so wondering if we enable something like redshift, would we have to give them access to our account?
Related
Currently, we use AWS IAM User permanent credentials to transfer customers' data from our company's internal AWS S3 buckets to customers' Google BigQuery tables following BigQuery Data Transfer Service documentation.
Using permanent credentials possesses security risks related to the data stored in AWS S3.
We would like to use AWS IAM Role temporary credentials, which require the support of a session token on the BiqQuery side to get authorized on the AWS side.
Is there a way that the BigQuery Data Transfer Servce can use AWS IAM roles or temporary credentials to authorise against AWS and transfer data?
We considered Omni framework (https://cloud.google.com/bigquery/docs/omni-aws-cross-cloud-transfer) to transfer data from S3 to BQ, however, we faced several concerns/limitations:
Omni framework targets data analysis use-case rather than data transfer from external services. This concerns us that the design of Omni framework may have drawbacks in relation to data transfer at high scale
Omni framework currently supports only AWS-US-EAST-1 region (we require support at least in AWS-US-WEST-2 and AWS-EU-CENTRAL-1 and corresponding Google regions). This is not backward compatible with current customers' setup to transfer data from internal S3 to customers' BQ.
Our current customers will need to signup for Omni service to properly migrate from the current transfer solution we use
We considered a workaround with exporting data from S3 through staging in GCS (i.e. S3 -> GCS -> BQ), but this will also require a lot of effort from both customers and our company's sides to migrate to the new solution.
Is there a way that the BigQuery Data Transfer Servce can use AWS IAM roles or temporary credentials to authorise against AWS and transfer data?
No unfortunately.
The official Google BigQuery Data Transfer Service only mentions AWS access keys all throughout the documentation:
The access key ID and secret access key are used to access the Amazon S3 data on your behalf. As a best practice, create a unique access key ID and secret access key specifically for Amazon S3 transfers to give minimal access to the BigQuery Data Transfer Service. For information on managing your access keys, see the AWS general reference documentation.
The irony of the Google documentation is that while it refers to best practices and links to the official AWS docs, it actually doesn't endorse best practices and ignores what AWS mention:
We recommend that you use temporary access keys over long term access keys, as mentioned in the previous section.
Important
Unless there is no other option, we strongly recommend that you don't create long-term access keys for your (root) user. If a malicious user gains access to your (root) user access keys, they can completely take over your account.
You have a few options:
hook into both sides manually (i.e. link up various SDKs and/or APIs)
find an alternative BigQuery-compatible service, which does as such
accept the risk of long-term access keys.
In conclusion, Google is at fault here of not following security best practices and you - as a consumer - will have to bear the risk.
We've got some data in Snowflake that we'd like to pull into our GCP environment, on a periodic basis. One of our engineers has done the equivalent setup on AWS on a previous project, using the documentation here. I find this setup to be a lot simpler than setting up a push data flow, which requires creating an integration and service account from the Snowflake side, then granting the service account some IAM permissions in GCP.
Can anyone tell me if GCP offers a similar pull-based connector API/setup for Snowflake?
We were doing most of our cloud processing (and still do) using AWS. However, we also now have some credits on GCP and would like to use and want to explore interoperability between the cloud providers.
In particular, I was wondering if it is possible to use AWS S3 from within GCP. I am not talking about migrating the data but whether there is some API which will allow AWS S3 to work seamlessly from within GCP. We have a lot of data and databases that are hosted on AWS S3 and would prefer to keep everything there as it still does the bulk of our compute.
I guess one way would be to transfer the AWS keys to the GCP VM and then use the boto3 library to download content from AWS S3 but I was wondering if GCP, by itself, provides some other tools for this.
From an AWS perspective, an application running on GCP should appear logically as an on-premises computing environment. This means that you should be able to leverage the services of AWS that can be invoked from an on-premises solution. The GCP environment will have Internet connectivity to AWS which should be a pretty decent link.
In addition, there is a migration service which will move S3 storage to GCP GCS ... but this is distinct from what you were asking.
See also:
Getting started with Amazon S3
Storage Transfer Service Overview
AWS Pinpoint Analytics appears to have replaced Amazon Mobile Analytics. In Mobile Analytics, you were able to create custom dashboards.
I'm struggling to find the feature in AWS Pinpoint. I'm assuming it's in there somewhere, but alas, I haven't found it yet.
#D.Patrick, you can create custom dashboards with Pinpoint data but not directly within Pinpoint console i.e You would need first to export your Pinpoint event data to a persistent storage (e.g S3 or Redshift) using Amazon Kinesis. Once in S3, you can use analytics tools to further analyze or visual the data. Such analytic tool offered by AWS include AWS Quicksight or AWS Athena. Other analytics(none-AWS) tools include Splunk
Check out the blog by AWS on this topic:
https://aws.amazon.com/blogs/messaging-and-targeting/creating-custom-pinpoint-dashboards-using-amazon-quicksight-part-1/
The 3 parts of this session describe in detail how to use Python 3, with AWS Lambda to create the custom dashboards.
I am developing the application where "form" plays an important role. I use the "form" for data collection from the users which I store it on DynamoDB and then these data should be displayed in my application synchronously or immediately after some trigger functions when the data has been inserted into the dynamoDB.
What is the best way to achieve this? How should I frame my infrastructure on the AWS Cloud? What are the services should I rely on?
For "realtime" applications you should use something like graphQL.
On AWS you can use: AWS App Sync
https://console.aws.amazon.com/appsync/home?region=us-east-1#/
There are multiple ways you can achieve this. However, one of most modernized way of building your applications today is using a Serverless Architecture. You can host your website on S3 and can go serverless with sample architecture as below
(Note: you can just replace the Amazon Aurora with Dynamo DB in the architecture reference)
You can create a server less application on AWS using following AWS services:
AWS Lambda: AWS Lambda is a compute service that lets you run code without provisioning or managing servers.
AWS API Gateway: Amazon API Gateway is a fully managed service that makes it easy for developers to create, publish, maintain, monitor, and secure APIs at any scale.
AWS S3: AWS S3 is Object storage built to store and retrieve any amount of data from anywhere
AWS DynamoDB: Amazon
DynamoDB is a fast and flexible nonrelational database service for
all applications that need consistent, single-digit millisecond latency at any scale.
AWS Route53: For creating and registering a domain name for the web app.
AWS IAM: AWS IAM for creating users,roles and policies.
AWS Cognito: for authentication, access control to your web app.