AWS Transfer Family supports integration with AD Connector (https://docs.aws.amazon.com/directoryservice/latest/admin-guide/ad_connector_app_compatibility.html). As far as I understand, connectors are deployed in vpn-linked subnets that allows them to proxy calls to an on-premise Active Directory.
What exactly happens (what resources are created/updated under the hood) when I select AD connector as the authenticator for AWS Transfer? I'm specifically curious as to what changes are made in VPC to allow this integration.
In relation to AWS Directory Service, AWS Transfer does not seem to mutate your VPC. If you create an AD and then associate it with AWS Transfer, and take a look at your VPC, there is no new networking resources of any kind. Similar to other applications (https://docs.aws.amazon.com/directoryservice/latest/admin-guide/ms_ad_manage_apps_services.html), AWS Directory Services authorizes AWS Transfer to access your AD (in this case, connector) for Transfer logins.
Related
I'm trying to prepare a flow where we can regularly pull the available new files in third parties' on-prem server to our S3 using AWS Transfer family.
I read this documentation https://aws.amazon.com/blogs/storage/how-discover-financial-secures-file-transfers-with-aws-transfer-family/, but it was not clear on setting up and configuring the process.
Can someone share any clear documentation or reference links on using AWS Transfer Family to pull files from external on-prem server to our S3?
#Sampath, I think you misunderstood the available features of the AWS Transfer service. That service is actually acting as a serverless SFTP with AWS S3 as the backend storage to which you can connect via SFTP protocol (now supports FTP and FTPS as well). You can either PUSH data to S3 or PULL data from S3 via AWS Transfer service. You cannot PULL data into S3 from anywhere else via AWS Transfer service alone.
You may have to use any other solution like a Python Script running on AWS EC2 for that purpose.
Another solution would be to connect the external third-party server to the AWS Transfer Service and that server PUSHES files on S3 via AWS Transfer.
As per your use case, I think you need a simple solution that connects to an external third-party server and copies files from it to the AWS S3 bucket. It can be done via a Python script as well and you can run it on either AWS EC2, AWS ECS, AWS Lambda, AWS Batch, etc, depending on the specifications and requirements.
I have used AWS Transfer once I found it to be very expensive and went on with AWS EC2 instead. In the case of AWS EC2, you can even buy reserved instances to further reduce the cost. If the task is just about copying files from an external server to S3 and the copy job will never take more than 10 minutes, then it is better to run it on AWS Lambda.
In short, you cannot PULL data from any server into S3 using the AWS Transfer service. You can only PUSH data to or PULL data from S3 using the AWS Transfer service.
References to some informative blogs:
Centralize data access using AWS Transfer Family and AWS Storage Gateway
How Discover Financial secures file transfers with AWS Transfer Family
Moving external site data to AWS for file transfers with AWS Transfer Family
Easy SFTP Setup with AWS Transfer Family
With the AWS Transfer Family service you can create servers that uses SFTP, FTPS, and FTP protocols for your file transfers, and use the Amazon S3 and EFS as domains to store and access your files.
To connect your on-premise servers with the Transfer Family server you will need to use a service like File Gateway/Storage Gateway and connect via HTTPS to S3 to sync your files.
Your architecture will be something like this:
If you want more details of how to connect with your on-premises servers with the AWS S3/Transfer Family services take a look on this blog post: Centralize data access using AWS Transfer Family and AWS Storage Gateway
We were doing most of our cloud processing (and still do) using AWS. However, we also now have some credits on GCP and would like to use and want to explore interoperability between the cloud providers.
In particular, I was wondering if it is possible to use AWS S3 from within GCP. I am not talking about migrating the data but whether there is some API which will allow AWS S3 to work seamlessly from within GCP. We have a lot of data and databases that are hosted on AWS S3 and would prefer to keep everything there as it still does the bulk of our compute.
I guess one way would be to transfer the AWS keys to the GCP VM and then use the boto3 library to download content from AWS S3 but I was wondering if GCP, by itself, provides some other tools for this.
From an AWS perspective, an application running on GCP should appear logically as an on-premises computing environment. This means that you should be able to leverage the services of AWS that can be invoked from an on-premises solution. The GCP environment will have Internet connectivity to AWS which should be a pretty decent link.
In addition, there is a migration service which will move S3 storage to GCP GCS ... but this is distinct from what you were asking.
See also:
Getting started with Amazon S3
Storage Transfer Service Overview
Can i connect to different account AWS services(s3, dynamoDb) from my account ec2 using VPC Endpoint?
Amazon S3 and Amazon DynamoDB are accessed on the Internet via API calls.
When a call is made to these services, a set of credentials is provided to identify the account and user.
If you wish to access S3 or DynamoDB resources belonging to a different account, you simply need to use credentials that belong to the target account. The actual request can be made from anywhere on the Internet (eg from Amazon EC2 or from a computer under your desk) — the only things that matters is that you have valid credentials linked to the desired AWS account.
There is no need to manipulate VPC configurations to access resources belonging to a different AWS Account. The source of the request is actually irrelevant.
both of them are used to authorize from company network to use AWS resource. I do not know what is the difference btw them.
AWS Directory Service for Microsoft Active Directory (Enterprise Edition), also known as AWS Microsoft AD, enables your directory-aware workloads and AWS resources to use managed Active Directory in the AWS Cloud. The Microsoft AD service is built on actual Microsoft Active Directory and does not require you to synchronize or replicate data from your existing Active Directory to the cloud.
see https://aws.amazon.com/directoryservice/
whereas STS:
The AWS Security Token Service (STS) is a web service that enables you to request temporary, limited-privilege credentials for AWS Identity and Access Management (IAM) users or for users that you authenticate (federated users).
see http://docs.aws.amazon.com/STS/latest/APIReference/Welcome.html
The AWS Directory Service is a full AD implementation which can "join" your in house servers. This would work better if there is a VPN and/or AWS Direct Connect in place. So replication occurs just as it would if you set up another Microsoft AD server
STS for linking your on premise accounts to AWS has some kind of shim software that is custom, running on premise. In the case you are asking about the local on premise environment has AD. The custom software deals with authentication against a local AD, then negotiates temporary creditials with the AWS account. AWS do custom software for this use case see https://aws.amazon.com/blogs/security/how-to-connect-your-on-premises-active-directory-to-aws-using-ad-connector/
So the difference between them is
AWS Directory Service sets up a new server on the cloud. This has to be linked to your on premise equipment to get it to work. It works better with a good link between on premise and AWS. Could be a better fit if your are running Microsoft stack on AWS
STS in this use case requires installing software on your on premise systems. There is also configuration to the AWS account
We're a growing shop that has no real user management. It's come the time to have single-sign-on. We host everything in AWS with the exception of an in-house desktop server that basically just hosts a NAS.
I'm only slightly familiar enough with Active Directory that I could set up an in-house domain and have it federate users. I've read the FAQs on AWS Directory Service and just looking to get clarity at what it can and cannot do.
My goal is to have an AD that can manage users in AWS as well as in-house. I'm unsure if I can do both in AWS DS or if it's just to handle the AWS portion.
Per the FAQ: AWS Directory Service makes it easy for you to setup and run directories in the AWS cloud, or connect your AWS resources with an existing on-premises Microsoft Active Directory
Does this mean that even if I ran AWS DS I'd still have to host an in-house AD and join them to manage users in AWS as well as in-house? I'd like to avoid having to run one on-premesis since my goal is to get everything into AWS.
You can use AWS SSO to manage authentication to the AWS console and AWS CLI and it integrates directly with AWS Managed Directory Services, so you can have all your authentication in one place. Take care to create a "break glass" IAM account in case you have issues in AD, lest you lock yourself out of AWS and therefore can't manage the AD servers.
https://docs.aws.amazon.com/singlesignon/latest/userguide/connectawsad.html