I work on a multi env AWS project where we use Java for our Lambda functions.
Each developer has one AWS account and multiple profiles (Dev, Stg, Prod...).
Using the Java SDK, We want to be able debug the Lambda functions locally using Intellij IDEA, for that, we added a main that calls the handler.
The question is, how can we globally specify the profile to use when we execute our code ?
For every execution, we are getting a not authorized access to our AWS services like Secrets Manager or S3. We figured out that it always considers the basic account and not the profile even when we specify AWS_PROFILE environment variable.
Thank you very much.
The profile is controlled globally with the AWS_PROFILE environment variable.
On the Java side you should use ProfileCredentialsProvider.
Related
I have a Java program which needs to access Amazon S3 to put some files there. Please note that this Java program is running in my desktop (not in EC2). What's the best secure way to access Amazon S3 using credentials? Following are the ways I am aware of.
Using access token and secret
a. In sdk properties file
b. As environment variables
c. In command line system properties
d. Directly hard coding in program
Of course I'd prefer options b and c for security reasons.
Is there a role based permissions possible here? My understanding is that it's not possible since my Java program is running in an external machine which AWS doesn't know.
Any other method of access possible?
Thanks in advance.
The best way is to use the default provider chain, which means that the [DefaultCredentialsProvider] (https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/auth/credentials/DefaultCredentialsProvider.html) class will decide from where to take the credentials based on a specific hierarchy:
1. Java System Properties - aws.accessKeyId and aws.secretAccessKey
2. Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
3. Web Identity Token credentials from system properties or environment variables
4. Credential profiles file at the default location (~/.aws/credentials) shared by all AWS SDKs and the AWS CLI
5. Credentials delivered through the Amazon EC2 container service if AWS_CONTAINER_CREDENTIALS_RELATIVE_URI" environment variable is set and security manager has permission to access the variable,
6. Instance profile credentials delivered through the Amazon EC2 metadata service
For local development the recommended way is to set up your credentials using the aws configure command and let the default provider chain take advantage of that.
Although environment variables may be a reasonable choice in some cases (and the default chain will be able to use them), please NEVER ever hardcode any credentials in your code!
Yes it is. We can assume a role using the AWS CLI:
aws sts assume-role --role-arn "arn:aws:iam::123456789012:role/example-role" --role-session-name AWSCLI-Session
This will provide a temporary AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN which can be provided to the application. The application will communicate with AWS services using the permissions provided by the assumed role.
Yes, there is another way if the goal is to access S3. We can use presigned urls.
When working with the AWS SDK for Java V2, refer to the AWS SDK for Java
Developer Guide V2. This developer guide contains a lot of information that answers questions like this.
To learn how credentials work, please refer to this topic:
Using credentials
All AWS Examples in Github assume credentials are loaded from the credential file. As explained in the docs, the credentials file is located in
Windows - C:\Users<yourUserName>.aws\credentials
Linux, macOS, Unix - ~/.aws/credentials
See this topic that will show you how to get up and running using the Amazon S3 API- including setting up your credentials.
Get started with the AWS SDK for Java 2.x
The Amazon S3 Java API has methods like pubObject that lets you place objects into an Amazon S3 bucket.
I have as php library I wrote to help with working along side Amazon Web Services. It was built to either look for the default $HOME/.aws/credentials (or be pointed to a similar format file) or to look for the key and secret in the environment before proceeding.
We are now going to be running it on an EC2 and I was shown how you can use roles in conjunction with the EC2 to get and keep much better control on what the server code can and can't do. But I need to modify my code to be able to know when it has proper permissions before proceeding and I don't see anywhere in the docs on assigning an EC2 instance a given role how you know in the SDK that it has the permissions of that role.
Is there some way once I instantiate the SDK to ask something akin to 'hasRole' or 'getRoleArn' or something like that?
SDKs are mapped directly to API calls. So if you know what cli command to call, it makes it much easier to google. So you want the aws sts get-caller-identity most likely.
Doing a google for "PHP sts sdk aws" is then the search you would do. And then you would wind up on this page.
So that way is using the SDK. There are a couple of other ways as well. As you are using ec2 you can use instance meta-data as well.
On another note I do think you should be careful though with leaking the AWS role into your application code. It probably makes more sense to use user identity context, such as with Cogito, and then use different groups with different permission sets. The role on the actual ec2 instance shouldn't be changing (unless you do a re-deploy), so there is no need for your code to check something that won't change during the normal running of the application. You could simply use an environment variable to convey whatever configuration you want to your application.
aws sts get-caller-identity --query 'Arn'
arn:aws:iam::1232412321:role/YourRole
I'm writing a lambda function in Python 3.8. The function connects with a dynamodb using boto3:
db = boto3.resource('dynamodb', region_name='foo', aws_access_key_id='foo', aws_secret_access_key='foo')
That is what I have while I am developing on my local machine and need to test the function. But, when I deploy this to lambda, I can just remove the credentials and my function will connect to the dynamodb if I have the proper IAM roles and policies setup in place. For example, this code would work fine when deployed to lambda:
db = boto3.resource('dynamodb', region_name='foo')
The question is, how can I manage this in terms of pushing code to lambda? I am using AWS SAM to deploy to AWS. Right now what I do is once I'm done developing my function, I remove the aws_access_key_id='foo' and aws_secret_access_key='foo' parts manually and then deploy the functions using SAM.
There must be a better way to do this? Could I embed these into my IDE instead? I'm using PyCharm. Would that be a better way? If not, what else?
You should never put credentials in the code like that.
When running the code locally, use the AWS CLI aws configure command to store local credentials in a ~/.aws/config file. The AWS SDKs will automatically look in that file to obtain credentials.
In sam you can invoke your function locally using sam local invoke or sam local start-lambda.
Both of them take --profile parameter:
The AWS credentials profile to use.
This will ensure that your local lambda environment executes with correct credentials without needing to hard code them in your code. Subsequently, you can test your code without modifications which would otherwise be needed when hard coding the key id and secret key.
You can use environment variables.
Environment variables can be configured both in pycharm, as well as in AWS Lambda and AWS SAM.
As stated in the Lambda best practices: "Use environment variables to pass operational parameters to your function. For example, if you are writing to an Amazon S3 bucket, instead of hard-coding the bucket name you are writing to, configure the bucket name as an environment variable."
You can also use an environment variable to specify which environment is being used, which can then be used to explicitly determine whether credentials are necessary.
So it says on the github documentation here that
AWS Vault is a tool to securely store and access AWS credentials in a
development environment.
AWS Vault stores IAM credentials in your operating system's secure
keystore and then generates temporary credentials from those to expose
to your shell and applications. It's designed to be complementary to
the AWS CLI tools, and is aware of your
But what does this actually mean? As a developer does this mean to create a kind of lock to prevent anyone from using my code without the aws-vault profile? When should I use this technology? I want to know a bit more about it before I use it.
It actually doesn't have anything related to development.
While working with Amazon managed services we can take advantage of IAM roles but that doesn't work when you're doing it from our local environment or from some other Cloud VM like accessing a S3 bucket. It comes handy when you're doing a lot of work with AWS CLI or even writing terraform for your environment. It is just for a precaution so we don't expose or IAM credentials to external world (you will receive an abuse notification from Amazon whenever your keys are compromised). There are many other ways to make sure your keys don't get compromised like before pushing your code to a version control use git-secrets to make sure you don't push any sensitive information.
AWS .Net SDK specifies several ways to specify credentials. One of them is SDK store - http://docs.aws.amazon.com/AWSSdkDocsNET/V3/DeveloperGuide/net-dg-config-creds.html#net-dg-config-creds-sdk-store
In that same document two seemingly contradictory things are written:
"SDK Store profiles are specific to a particular user on a particular host. They cannot be copied to other hosts or other users. For this reason, SDK Store profiles cannot be used in production applications."
"The associated credentials are incorporated into the application during the build process."
IF the credentials are incorporated in the program at the build time, then why can't I use the SDK store method in production environment?
That leaves me with either storing credentials in .config files, or credentials files, or using roles.
To clarify, the documentation is stating that if you are only using the SDK store, you can't simply pick up your application and move it to any other machine while expecting it to work as-is. If you are using profile names, your application is referencing the profile name in the local SDK store (or, failing that, the credentials file used by other AWS resources like the AWS CLI).
For example, if you created a profile named dev-Amit on your personal computer and then moved to another machine in production, dev-Amit wouldn't exist there unless you had already configured the SDK store on that machine with the same profile name.
This behavior is contrary to the recommended practice of passing credentials in EC2, using temporary credentials via IAM Roles. Using this method, you can reliably pass credentials to your applications on EC2 instances by simply attaching an IAM role to the instance that has sufficient permissions to do whatever your application needs to do. This method is preferred because you don't have to login to the instance to configure anything; you simply attach the IAM role at creation time.
Further Resources:
AWS Documentation - Tutorial: Grant Access Using an IAM Role and the AWS SDK for .NET
AWS Documentation - Order in which credentials are searched for