We were doing most of our cloud processing (and still do) using AWS. However, we also now have some credits on GCP and would like to use and want to explore interoperability between the cloud providers.
In particular, I was wondering if it is possible to use AWS S3 from within GCP. I am not talking about migrating the data but whether there is some API which will allow AWS S3 to work seamlessly from within GCP. We have a lot of data and databases that are hosted on AWS S3 and would prefer to keep everything there as it still does the bulk of our compute.
I guess one way would be to transfer the AWS keys to the GCP VM and then use the boto3 library to download content from AWS S3 but I was wondering if GCP, by itself, provides some other tools for this.
From an AWS perspective, an application running on GCP should appear logically as an on-premises computing environment. This means that you should be able to leverage the services of AWS that can be invoked from an on-premises solution. The GCP environment will have Internet connectivity to AWS which should be a pretty decent link.
In addition, there is a migration service which will move S3 storage to GCP GCS ... but this is distinct from what you were asking.
See also:
Getting started with Amazon S3
Storage Transfer Service Overview
Related
Sorry, I’m sort of a newbie when it comes to Amazon AWS Cloud so sorry if I sound naïve.
For .NET developers, I’ve used Visual Studio 2019’s AWS Lambda project to code Lambda functions and ultimately deploy it to Amazon AWS cloud
However, my concern is that there is No way to version and/or back up the Configurations for the AWS Services ( i.e., S3 bucket, Amazon SNS & SQS, etc.) that are invoked and/or trigger the various AWS Lambda Functions
The problem is that IT developers who configure said AWS Services have to use the ADFS AWS Services console website’s GUI to configure the various AWS Services, and if someone mistakenly deletes an AWS Services then they lose the configuration settings as well?
How do we go about versioning and/or backing up Configurations for the AWS Services?
There are Infrastructure as a Code frameworks like Terraform and Ansible designed to address that.
You can't really delete an AWS service.
it seems like you guys are kind of "new" to AWS so I will recommend using CloudFormation templates as Infrastructure As Code tool. All the configuration of how your AWS resources are supposed to look like can be added to the template and you deploy the template to create your AWS resources. Its AWS-Native and does not cost you anything.
On top of it, you also want to add your CloudFormation templates to version control system.
We've got some data in Snowflake that we'd like to pull into our GCP environment, on a periodic basis. One of our engineers has done the equivalent setup on AWS on a previous project, using the documentation here. I find this setup to be a lot simpler than setting up a push data flow, which requires creating an integration and service account from the Snowflake side, then granting the service account some IAM permissions in GCP.
Can anyone tell me if GCP offers a similar pull-based connector API/setup for Snowflake?
I am still new to the cloud and when I first started I used Clever Cloud.
But now I want to migrate to AWS, and I have data that I want to move from Cellar to Amazon S3.
I am not sure what are the conventions on this or the best practices, and if anyone can help with documentation or explanation on how I can proceed is very much appreciated.
Thank you very much.
Clever Cloud Cellar is an Amazon S3 compatible service. This means it operates pretty much the same as S3.
Clever Cloud is not able to communicate directly to Amazon S3, and Amazon S3 is not able to communicate directly to Cellar. Therefore, you will need to:
Download the files from Cellar using s3cmd or the AWS Command-Line Interface (CLI) (see instructions on CleverCloud Cellar website)
Upload the files to an Amazon S3 bucket using the AWS CLI and your AWS credentials
This activity would be most efficient if performed from an Amazon EC2 instance since it has high bandwidth connectivity to Amazon S3.
Note that there will be Data Transfer costs from Clever Cloud for "Outbound traffic".
I suggest you start by getting s3cmd or the AWS CLI working with Cellar to download a single file, and then get the AWS CLI working with Amazon S3 to upload a single file. You can then use the sync command to copy whole directories of files.
I am working on a pet project based on multi-cloud (AWS and GCP) which is based on serverless architecture.
Now there are files generated by the business logic within GCP (using Cloud Functions and Pub/Sub) and they are stored in GCP Cloud storage. I want to ingest these files dynamically to AWS S3 bucket from the Cloud Storage.
One possible way is by using the gsutil library (Exporting data from Google Cloud Storage to Amazon S3) but this would require a compute instance, and run the gsutil commands manually which I want to avoid.
In answering this I'm reminded a bit of a Rube Goldberg type setup but I don't think this is too bad.
From the Google side you would create a Cloud Function that is notified when a new file is created. You would use the Object Finalize event. This function would get the information about the file and then call an AWS Lambda fronted by AWS API Gateway.
The GCP Function would pass the bucket and file information to the AWS Lambda. On the AWS side you would have your GCP credentials and the GCP API download the file and upload it to S3.
Something like:
All serverless on both GCP and AWS. Testing isn't bad as you can keep them separate - make sure that GCP is sending what you want and make sure that AWS is parsing and doing the correct thing. There is likely some authentication that needs to happen from the GCP cloud function to API gateway. Additionally, the API gateway can be eliminated if you're ok pulling AWS client libraries into the GCP function. Since you've got to pull GCP libraries into the AWS Lambda this shouldn't be much of a problem.
I am developing the application where "form" plays an important role. I use the "form" for data collection from the users which I store it on DynamoDB and then these data should be displayed in my application synchronously or immediately after some trigger functions when the data has been inserted into the dynamoDB.
What is the best way to achieve this? How should I frame my infrastructure on the AWS Cloud? What are the services should I rely on?
For "realtime" applications you should use something like graphQL.
On AWS you can use: AWS App Sync
https://console.aws.amazon.com/appsync/home?region=us-east-1#/
There are multiple ways you can achieve this. However, one of most modernized way of building your applications today is using a Serverless Architecture. You can host your website on S3 and can go serverless with sample architecture as below
(Note: you can just replace the Amazon Aurora with Dynamo DB in the architecture reference)
You can create a server less application on AWS using following AWS services:
AWS Lambda: AWS Lambda is a compute service that lets you run code without provisioning or managing servers.
AWS API Gateway: Amazon API Gateway is a fully managed service that makes it easy for developers to create, publish, maintain, monitor, and secure APIs at any scale.
AWS S3: AWS S3 is Object storage built to store and retrieve any amount of data from anywhere
AWS DynamoDB: Amazon
DynamoDB is a fast and flexible nonrelational database service for
all applications that need consistent, single-digit millisecond latency at any scale.
AWS Route53: For creating and registering a domain name for the web app.
AWS IAM: AWS IAM for creating users,roles and policies.
AWS Cognito: for authentication, access control to your web app.