I'm writing a lambda function in Python 3.8. The function connects with a dynamodb using boto3:
db = boto3.resource('dynamodb', region_name='foo', aws_access_key_id='foo', aws_secret_access_key='foo')
That is what I have while I am developing on my local machine and need to test the function. But, when I deploy this to lambda, I can just remove the credentials and my function will connect to the dynamodb if I have the proper IAM roles and policies setup in place. For example, this code would work fine when deployed to lambda:
db = boto3.resource('dynamodb', region_name='foo')
The question is, how can I manage this in terms of pushing code to lambda? I am using AWS SAM to deploy to AWS. Right now what I do is once I'm done developing my function, I remove the aws_access_key_id='foo' and aws_secret_access_key='foo' parts manually and then deploy the functions using SAM.
There must be a better way to do this? Could I embed these into my IDE instead? I'm using PyCharm. Would that be a better way? If not, what else?
You should never put credentials in the code like that.
When running the code locally, use the AWS CLI aws configure command to store local credentials in a ~/.aws/config file. The AWS SDKs will automatically look in that file to obtain credentials.
In sam you can invoke your function locally using sam local invoke or sam local start-lambda.
Both of them take --profile parameter:
The AWS credentials profile to use.
This will ensure that your local lambda environment executes with correct credentials without needing to hard code them in your code. Subsequently, you can test your code without modifications which would otherwise be needed when hard coding the key id and secret key.
You can use environment variables.
Environment variables can be configured both in pycharm, as well as in AWS Lambda and AWS SAM.
As stated in the Lambda best practices: "Use environment variables to pass operational parameters to your function. For example, if you are writing to an Amazon S3 bucket, instead of hard-coding the bucket name you are writing to, configure the bucket name as an environment variable."
You can also use an environment variable to specify which environment is being used, which can then be used to explicitly determine whether credentials are necessary.
Related
I work on a multi env AWS project where we use Java for our Lambda functions.
Each developer has one AWS account and multiple profiles (Dev, Stg, Prod...).
Using the Java SDK, We want to be able debug the Lambda functions locally using Intellij IDEA, for that, we added a main that calls the handler.
The question is, how can we globally specify the profile to use when we execute our code ?
For every execution, we are getting a not authorized access to our AWS services like Secrets Manager or S3. We figured out that it always considers the basic account and not the profile even when we specify AWS_PROFILE environment variable.
Thank you very much.
The profile is controlled globally with the AWS_PROFILE environment variable.
On the Java side you should use ProfileCredentialsProvider.
I have a cross account IAM question about running in a Lambda function. (I know people may use STS assume, but this package really isn't worth protecting, and I don't want to go through linking the accounts)
Account “A”.
S3 – package “s3://foo/foo”
IAM credentials “pkg_creds” for bucket "s3://foo"
Account “B”
Lamba function “gogo” runs
In this Lambda function, it attempts to use boto3 and the pkg_creds to
download package “s3://foo/foo”, but if fails with this error:
**The provided token is malformed or otherwise invalid.**
Lambda is read only, but I believe boto3 will not write credentials to ~/.aws if I'm using boto3.client (not session). However, I also set the AWS_CONFIG_FILE to /tmp just in case. It still fails. I suspect what I'm proposing isn't possible because LAMBDA has immutable AWS credentials, where you can't change scopes, even one that is explicitly given to boto3.
Let me know your thoughts. I may try do the job with Faragate, but Lambda function is easier to maintain and deploy.
Thanks in advance!
Lambda isn't using a ~/.aws config file at all, it is using environment variables by default. There are many ways to configure AWS credentials in boto3. You should be able to create a new boto3 client in your Lambda function with explicit AWS credentials like so:
client = boto3.client(
's3',
aws_access_key_id=ACCOUNT_A_ACCESS_KEY,
aws_secret_access_key=ACCOUNT_A_SECRET_KEY
)
And pass ACCOUNT_A_ACCESS_KEY and ACCOUNT_A_SECRET_KEY as environment variables to the function.
User error. I can verify boto3 in a lambda function can use credentials outside of its scope.
after troubleshooting more. the issue was that i was took in the "environmental variable" SESSION which is set on the lambda function, but not on my ec2 instance. so i was always using the lambda session key that seems to overide the explicit key and secret.
We have an API app that uses Firebase Admin to send messages to devices.
Earlier, we used to specify the service account key using environment variable like GOOGLE_APPLICATION_CREDENTIALS="path_to_file.json".
But now, since we are shifting to AWS Elastic Container Service on Fargate, I am unable to figure out how to put this file in the container for AWS ECS.
Any advice highly appreciated.
Thanks
Solved it by storing the service key as a JSON Stringified environment variable & using admin.credential.cert() instead of defaultAppCredentials.
Refer: https://firebase.google.com/docs/reference/admin/node/admin.credential#cert
I would suggest instead AWS Secrets Manager that is purpose-built for storing secrets. Take a look to his blog post:
https://aws.amazon.com/blogs/compute/securing-credentials-using-aws-secrets-manager-with-aws-fargate/
Even better than using environment variables which have their own downsides, you can leverage AWS Parameter Store which is a secure way to manage secrets in the AWS environment (where secrets are encrypted both in transit and at rest).
You'd need to create an IAM role for Amazon ECS for your code to have access to the Parameter Store.
You may want to check this article: https://aws.amazon.com/blogs/compute/managing-secrets-for-amazon-ecs-applications-using-parameter-store-and-iam-roles-for-tasks/
Use the specific method from_service_account_info as described here. You then pass the content of the credentials json file as a dictionary.
My Aws Lambda function is written in Java. I am getting data from DynamoDb by giving some static credentials like below;
new BasicAWSCredentials(ACCESSKEY, SECRETKEY)
However, when i try to define my services in Aws Cloudformation. I could not find any way, how can i change these accesskey and secretkey credentials. What is the best way for managing these credentials?, Because they are special keys for each account and embedded in Java code.
Although you could use the context or download a file to pass credentials in runtime, one should not use explicit hard-coded credentials, as that is harder to acquire and rotate securely.
It is easier and safer to use roles, as described in the lambda permission model: http://docs.aws.amazon.com/lambda/latest/dg/intro-permission-model.html
Use explicit credentials only outside AWS (your dev machine for example), and even so do not hard-code them, use environment variables or CLI profiles.
I have a lambda function configured through the API Gateway that is supposed to hit an external API via Node (ex: Twilio). I don't want to store the credentials for the functions right in the lambda function though. Is there a better place to set them?
The functionality to do this was probably added to Lambda after this question was posted.
AWS documentation recommends using the environment variables to store sensitive information. They are encrypted (by default) using the AWS determined key (aws/lambda) when you create a Lambda function using the AWS Lambda console.
It leverages AWS KMS and allows you to either: use the key determined by AWS, or to select your own KMS key (by selecting Enable encryption helpers); you need to have created the key in advance.
From AWS DOC 1...
"When you create or update Lambda functions that use environment variables, AWS Lambda encrypts them using the AWS Key Management Service. When your Lambda function is invoked, those values are decrypted and made available to the Lambda code.
The first time you create or update Lambda functions that use environment variables in a region, a default service key is created for you automatically within AWS KMS. This key is used to encrypt environment variables. However, should you wish to use encryption helpers and use KMS to encrypt environment variables after your Lambda function is created, then you must create your own AWS KMS key and choose it instead of the default key. The default key will give errors when chosen."
The default key certainly does 'give errors when chosen' - which makes me wonder why they put it into the dropdown at all.
Sources:
AWS Doc 1: Introduction: Building Lambda Functions » Environment Variables
AWS Doc 2: Create a Lambda Function Using Environment Variables To Store Sensitive Information
While I haven't done it myself yet, you should be able to leverage AWS KMS to encrypt/decrypt API keys from within the function, granting the Lambda role access to the KMS keys.
Any storage service or database service on AWS will be able to solve your problem here. The question is what are you already using in your current AWS Lambda function? Based on that, and the following considerations:
If you need it fast and cost is not an issue, use Amazon DynamoDB
If you need it fast and mind the cost, use Amazon ElastiCache (Redis or Memcache)
If you are already using some relational database, use Amazon RDS
If you are not using anything and don't need it fast, use Amazon S3
In any case, you need to create some security policy (either IAM role or S3 bucket policy) to allow exclusive access between Lambda and your choice of storage / database.
Note: Amazon VPC support for AWS Lambda is around the corner, therefore any solution you choose, make sure it's in the same VPC with your Lambda function (learn more at https://connect.awswebcasts.com/vpclambdafeb2016/event/event_info.html)
I assume you're not referring to AWS credentials, but rather the external API credentials?
I don't know that it's a great place, but I have found posts on the AWS forums where people are putting credentials on S3.
It's not your specific use-case, but check out this forum thread.
https://forums.aws.amazon.com/thread.jspa?messageID=686261
If you put the credentials on S3, just make sure that you secure it properly. Consider making it available only to a specific IAM role that is only assigned to that Lambda function.
For 2022 we have AWS Secrets Manager for storing sensitive data like Database Credentials, API Tokens, Auth keys, etc.