I've been using AWS Codedeploy using github as the revision source. I have couple of configuration files that contains credentials(e.g. NewRelic and other third party license key) which I do not want to add it to my github repository. But, I need them in the EC2 instances.
What is a standard way of managing these configurations. Or, what tools do you guys use for the same purpose?
First, use IAM roles. That removes 90% of your credentials. Once you've done that, you can store (encrypted!) credentials in an S3 bucket and carefully control access. Here's a good primer from AWS:
https://blogs.aws.amazon.com/security/post/Tx1XG3FX6VMU6O5/A-safer-way-to-distribute-AWS-credentials-to-EC2
The previous answers are useful for managing AWS roles/credential specifically. However, your question is more about general non-AWS credentials, and how to manage them securely using AWS.
What works well for us is to secure the credentials in a properties file in a S3 bucket. Using same technique as suggested by tedder42 in A safer way to distribute AWS credentials to EC2, you can upload your credentials in a properties file into a highly secured S3 bucket, only available to your instance, which has been configured with the appropriate IAM role.
Then using CodeDeploy, you can add a BeforeInstall lifecycle hook to download the credential files to a local directory via the AWS CLI. For example:
aws s3 cp s3://credentials-example-com/credentials.properties
c:\credentials
Then when the application starts, it can read those credentials from the local file.
Launch your EC2 instances with an instance profile and then give the associated role access to all the things your service needs access to. That's what the CodeDeploy agent is using to make calls, but it's really there for any service you are running to use.
http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2_instance-profiles.html
Related
I have a Java program which needs to access Amazon S3 to put some files there. Please note that this Java program is running in my desktop (not in EC2). What's the best secure way to access Amazon S3 using credentials? Following are the ways I am aware of.
Using access token and secret
a. In sdk properties file
b. As environment variables
c. In command line system properties
d. Directly hard coding in program
Of course I'd prefer options b and c for security reasons.
Is there a role based permissions possible here? My understanding is that it's not possible since my Java program is running in an external machine which AWS doesn't know.
Any other method of access possible?
Thanks in advance.
The best way is to use the default provider chain, which means that the [DefaultCredentialsProvider] (https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/auth/credentials/DefaultCredentialsProvider.html) class will decide from where to take the credentials based on a specific hierarchy:
1. Java System Properties - aws.accessKeyId and aws.secretAccessKey
2. Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
3. Web Identity Token credentials from system properties or environment variables
4. Credential profiles file at the default location (~/.aws/credentials) shared by all AWS SDKs and the AWS CLI
5. Credentials delivered through the Amazon EC2 container service if AWS_CONTAINER_CREDENTIALS_RELATIVE_URI" environment variable is set and security manager has permission to access the variable,
6. Instance profile credentials delivered through the Amazon EC2 metadata service
For local development the recommended way is to set up your credentials using the aws configure command and let the default provider chain take advantage of that.
Although environment variables may be a reasonable choice in some cases (and the default chain will be able to use them), please NEVER ever hardcode any credentials in your code!
Yes it is. We can assume a role using the AWS CLI:
aws sts assume-role --role-arn "arn:aws:iam::123456789012:role/example-role" --role-session-name AWSCLI-Session
This will provide a temporary AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN which can be provided to the application. The application will communicate with AWS services using the permissions provided by the assumed role.
Yes, there is another way if the goal is to access S3. We can use presigned urls.
When working with the AWS SDK for Java V2, refer to the AWS SDK for Java
Developer Guide V2. This developer guide contains a lot of information that answers questions like this.
To learn how credentials work, please refer to this topic:
Using credentials
All AWS Examples in Github assume credentials are loaded from the credential file. As explained in the docs, the credentials file is located in
Windows - C:\Users<yourUserName>.aws\credentials
Linux, macOS, Unix - ~/.aws/credentials
See this topic that will show you how to get up and running using the Amazon S3 API- including setting up your credentials.
Get started with the AWS SDK for Java 2.x
The Amazon S3 Java API has methods like pubObject that lets you place objects into an Amazon S3 bucket.
How to access, upload and delete objects of the S3 bucket from the web URL securely?
We are accessing the objects in S3 from our Application. But that bucket is public which is not secure.
I have tried CloudFront with OAI on the s3 bucket and putting bucket private but access is denied from the application when trying to upload an object to the s3 bucket.
We want to upload and delete objects in s3 bucket. We want that bucket to private only. And we want to do this from web applications only not from CLI, not from any tool. How could we achieve this?
Any suggestion will be appreciated.
Thanks!
Your application can use an AWS SDK to communicate directly with AWS services.
Your application will require a set of credentials to gain access to resources in your AWS Account. This can be done in one of two ways:
If your application is running on an Amazon EC2 instance, assign an IAM Role to the instance
Otherwise, create an IAM User and store the AWS credentials on the application server by using the AWS CLI aws configure command
You can control the exact permissions and access given to the IAM Role / IAM User. This can include granting the application permission to access your Amazon S3 buckets. This way, the buckets can be kept private, but the application will be able to upload/download/delete objects in the bucket.
To add more to the previous answer, you can find many S3 SDK examples in the AWS Github located here:
https://github.com/awsdocs/aws-doc-sdk-examples
If you look under each programming language, you will find Amazon S3 examples. You can use the AWS SDK to perform actions on a bucket when its private. You can take security a step further too and use encryption as shown here:
https://github.com/awsdocs/aws-doc-sdk-examples/blob/master/java/example_code/s3/src/main/java/aws/example/s3/S3EncryptV2.java
Also, you can interact with Amazon S3 bucket from a web app as well by using the AWS SDK. Here is an example of building a web app using Spring Boot that interacts with an Amazon S3 bucket by reading all of the objects in the bucket.
Creating an example AWS photo analyzer application using the AWS SDK for Java
It's a bad practice to use long term credentials. AWS recommends to use short term credentials along with STS. Here is an article using Python/Flask to upload a file into private S3 bucket using STS/short term credentials.
Connect on-premise Python application with AWS services using Roles
I could have listed down all the steps in this post. But, it's a bit too long and so the reference to the above link.
According to many advices, we should not configure IAM USER but using IAM Role instead to avoid someone managed to grab the user confidential in .aws folder.
Lets say I don't have any EC2 instances. Can I still able to perform S3 operation via AWS CLI? Says aws s3 ls
MacBook-Air:~ user$ aws s3 ls
Unable to locate credentials. You can configure credentials by running "aws configure".
You are correct that, when running applications on Amazon EC2 instances or as AWS Lambda functions, an IAM role should be assigned that will provide credentials via the EC2 metadata service.
If you are not running on EC2/Lambda, then the normal practice is to use IAM User credentials that have been created specifically for your application, with least possible privilege assigned.
You should never store the IAM User credentials in an application -- there have been many cases of people accidentally saving such files into GitHub, and bad actors grab the credentials and have access to your account.
You could store the credentials in a configuration file (eg via aws configure) and keep that file outside your codebase. However, there are still risks associated with storing the credentials in a file.
A safer option is to provide the credentials via environment variables, since they can be defined through a login profile and will never be included in the application code.
I don't think you can use service roles on your personal machine.
You can however use multi-factor authentication for AWS CLI
You can use credentials on any machine not just EC2.
Follow the steps as described by the documentation for your OS.
http://docs.aws.amazon.com/cli/latest/userguide/installing.html
I'm currently using AWS S3 credentials inside my application's code in a file ignored by git, but every developer has that file so it can work with images.
I want to hide those credentials from the developers, but still have the ability to use S3 in the development environment.
What are the best practices on that?
Should I replace S3 usage to local files in the development environment?
Give each of your developers IAM accounts with their own API keys, set your application to read the API keys from environment variables rather than from a config file. When the code runs on EC2 instances, use IAM roles so that you don't need to use API keys at all. If you're using AWS SDKs it will use role credentials by default.
I have a bunch of Amazon S3 files that I want to download from within my app. However, I don't want them to be public.
Is there a way to require a key in the URL query in order to download the file? So far, I have not found documentation on this.
What you're looking for is known as "Presigned URLS"
Basically, you use the AWS SDK to generate a temporary URL, which includes some credentials that expire after a duration that you specify, and provide that to your end user.
It depends where your app runs.
If it runs on an ec2 instance, then ec2 instance runs with a certain role specified at creation (see "IAM role" in instance details). You can either configure that role to have full access to S3 or grant that role permissions to particular files in S3.
If your app runs outside of AWS. It's slightly more complicated as you need to configure credentials to be used. Don't know what you use to write your app, so below is a link to configuring go SDK. Others must be very similar. If my memory is correct, I just configured aws cli on my dev machine and that saved credentials in the right place for my app to use. To be fair, you can use this approach on an ec2 instance as well if you want, but I find IAM role approach easier there.
Access Control List (ACL) Overview
Using an IAM Role to Grant Permissions
SDK Configuration It's for go language, but others will be very similar. See Specifying Credentials section.