I have an old archive folder that exists on an on premise Windows server that I need to put into an S3 bucket, but having issues, it's more my knowledge of AWS tbh, but I'm trying.
I have created the S3 bucket and I can to attach it to the server using net share (AWS gives you the command via the AWS gateway) and I gave it a drive letter. I then tried to use robocopy to copy the data, but it didn't like the drive letter for some reason.
I then read I can use the AWS CLI so I tried something like:
aws s3 sync z: s3://archives-folder1
I get - fatal error: Unable to locate credentials
I guess I need to put some credentials in somewhere (.aws), but after reading too many documents I'm not sure what to do at this point, could someone advise?
Maybe there is a better way.
Thanks
You do not need to 'attach' the S3 bucket to your system. You can simply use the AWS CLI command to communicate directly with Amazon S3.
First, however, you need to provide the AWS CLI with a set of AWS credentials that can be used to access the bucket. You can do this with:
aws configure
It will ask for an Access Key and Secret Key. You can obtain these from the Security Credentials tab when viewing your IAM User in the IAM management console.
Related
This might be a stupid question, but I've never used AWS before.
So apparently, to create an AWS account I need to give my credit card information, but I don't want to do that.
Is there any other way to access the information from this link?:
https://s3.console.aws.amazon.com/s3/buckets/quizdb-public/?region=us-east-1&tab=overview
The URL https://s3.console.aws.amazon.com/s3/buckets/quizdb-public/?region=us-east-1&tab=overview
is the link that will be shown in the address bar when you log into the AWS console, go to S3 and click on the bucket. If you do not have access to that specific AWS account and the AWS console you will not be able to access the information in the bucket with that URL.
I'm trying to execute Exercise 1.1 in the AWS Certified Solutions Architect Study Guide, and am stymied right away.
It says to "Install (if necessary) and configure te AWS CLI on your local system...."
I don;t know if it is installed, adn have no idea what to do here, but I will press on.
It also says "To get you started, here are some basice CLI commets:"
aws s3 ls
aws s3 mb <bucketname>
aws s3 cp /path/to/file.txt s3://bucketname
OK. I type aws s3 ls and get an error saying that my access keys aren't set up and that I can fix this my running aws configure.
I run aws configure and am asked for: 1) AWS Access Key ID, 2) AWS Secret Access Key, 3) Default region name, and 4) Default output format.
I have no idea whatsoever what any of this is. i recall having set up a key pair at some point in the not to distant past, and manage to find a reference to it. I put in the fingerprint of the public key in the first, and manage to find the file that I was sent for the secret key, and cut and paste it. But it has a bunch of lines, so cut and pasting it doesn't work. I really doubt that this is the right stuff, anyway, but I have no idea what would be the right stuff. Can anyone help?
Access key and secret are your credentials to hit the AWS API. You've to create an IAM user (as best practice), and then created the access key and secret. And then you've to configure the details using aws configure.
https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html
Configuring the AWS CLI
I generate some files(size of the order KBs) periodically on an on-prem server. I want to push them to s3 as soon as they are generated. How do I go about managing the accesses?
I'm using the boto3 package of python to do so. How do I manage access, as in do I create a new IAM role? If so, how to specify the permissions?
There's a few requirements here:
You need something to 'trigger' the upload. This could be a cron job / scheduled task, or it could be specifically triggered by whatever process is generating the files
You could then use either the AWS Command-Line Interface (CLI) command to upload the files (either aws s3 cp or aws s3 sync), or you could write your own program as you suggested.
The AWS CLI or your own program will require AWS credentials. The recommended method would be:
Create a User in IAM and make note of the credentials (Access Key, Secret Key)
Assign minimal permissions to the User so that it has enough permission to perform the function (eg s3:PutObject for the given bucket). Do not assign s3:* on every bucket! (Very bad for security.)
On the on-premises computer, run aws configure (part of the AWS CLI) and enter the credentials that were provided when the IAM User was created
This will place the credentials in a .aws/credentials file on the computer. Never put credentials in your programming code! (Again, bad for security.)
If you are asking how to create an IAM User, please see: Creating an IAM User in Your AWS Account - AWS Identity and Access Management
According to many advices, we should not configure IAM USER but using IAM Role instead to avoid someone managed to grab the user confidential in .aws folder.
Lets say I don't have any EC2 instances. Can I still able to perform S3 operation via AWS CLI? Says aws s3 ls
MacBook-Air:~ user$ aws s3 ls
Unable to locate credentials. You can configure credentials by running "aws configure".
You are correct that, when running applications on Amazon EC2 instances or as AWS Lambda functions, an IAM role should be assigned that will provide credentials via the EC2 metadata service.
If you are not running on EC2/Lambda, then the normal practice is to use IAM User credentials that have been created specifically for your application, with least possible privilege assigned.
You should never store the IAM User credentials in an application -- there have been many cases of people accidentally saving such files into GitHub, and bad actors grab the credentials and have access to your account.
You could store the credentials in a configuration file (eg via aws configure) and keep that file outside your codebase. However, there are still risks associated with storing the credentials in a file.
A safer option is to provide the credentials via environment variables, since they can be defined through a login profile and will never be included in the application code.
I don't think you can use service roles on your personal machine.
You can however use multi-factor authentication for AWS CLI
You can use credentials on any machine not just EC2.
Follow the steps as described by the documentation for your OS.
http://docs.aws.amazon.com/cli/latest/userguide/installing.html
I've been using AWS Codedeploy using github as the revision source. I have couple of configuration files that contains credentials(e.g. NewRelic and other third party license key) which I do not want to add it to my github repository. But, I need them in the EC2 instances.
What is a standard way of managing these configurations. Or, what tools do you guys use for the same purpose?
First, use IAM roles. That removes 90% of your credentials. Once you've done that, you can store (encrypted!) credentials in an S3 bucket and carefully control access. Here's a good primer from AWS:
https://blogs.aws.amazon.com/security/post/Tx1XG3FX6VMU6O5/A-safer-way-to-distribute-AWS-credentials-to-EC2
The previous answers are useful for managing AWS roles/credential specifically. However, your question is more about general non-AWS credentials, and how to manage them securely using AWS.
What works well for us is to secure the credentials in a properties file in a S3 bucket. Using same technique as suggested by tedder42 in A safer way to distribute AWS credentials to EC2, you can upload your credentials in a properties file into a highly secured S3 bucket, only available to your instance, which has been configured with the appropriate IAM role.
Then using CodeDeploy, you can add a BeforeInstall lifecycle hook to download the credential files to a local directory via the AWS CLI. For example:
aws s3 cp s3://credentials-example-com/credentials.properties
c:\credentials
Then when the application starts, it can read those credentials from the local file.
Launch your EC2 instances with an instance profile and then give the associated role access to all the things your service needs access to. That's what the CodeDeploy agent is using to make calls, but it's really there for any service you are running to use.
http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2_instance-profiles.html