I am using amazon web services cli. I use a makefile to to build my lambda project and upload it to aws lambda. I am on a windows machine and using powershell to call make.
I try to delete my lambda function with the following lines
AWS_PATH = /cygdrive/c/Users/TestBox/AppData/Roaming/Python/Scripts/aws
AWS_WIN_PATH = $(shell cygpath -aw ${AWS_PATH})
AWS_REGION = eu-west-2
lambda_delete:
$(AWS_WIN_PATH) lambda delete-function --function-name LambdaTest --region $(AWS_REGION) --debug
I get this error..
NoCredentialsError: Unable to locate credentials
Unable to locate credentials. You can configure credentials by running "aws configure".
Running aws configure list prints out a valid default profile.
I think the problem is because i am using gnu make installed by cygwin on a windows machine. Using powershell to call make.
So the path to credentials looks like this "cygdrive/c/users/testbox/.aws/credentials" instead "c:\users\testbox.aws\credentials", when ~/.aws/credentials is evaluated by aws. I think :)
I had the same problem with the path to aws itself and had to use $(shell cygpath -aw ${AWS_PATH}) to convert it to a path windows python could use.
Is there any way to pass the credentials directly to the lambda delete-function or indirectly through a path to a file? I cant seem to think of a way because the code that searches for the credentials is internal to botocore.
Is there a way around this that you know off?
Alternative solution, consider using AWS SAM templates
Use AWS SAM templates to deploy your Lambda functions and AWS resources using CloudFormation.
Edit your SAM template and define your AWS resources. For example, define Lambda functions/path to your code.
aws cloudformation package to package and upload your local code to S3.
aws cloudformation deploy to provision and update AWS resources with the updated code on S3.
This would work in CMD/Powershell without the make hassle. You will also have the benefit of having your resources versioned as code and you won't need to worry about tracking or adding new AWS APIs in your make file.
More complex serverless frameworks for reference:
AWS Chalice https://github.com/aws/chalice
Django/Flask + Lambda https://github.com/Miserlou/Zappa
Cross cloud serverless solution https://github.com/serverless/serverless
Related
I am trying to setup some build and deployment servers based on EC2 instances to deploy software to AWS via CloudFormation.
The current setup uses the AWS CLI to deploy CloudFormation templates, and authentication is handled using a credentials profile where the ~/.aws/config file has a profile with:
[profile x]
role_arn = x
credential_source = Ec2InstanceMetadata
region = x
The setup using the AWS CLI appears to be working fine, and can deploy CloudFormation templates, upload files to S3 etc.
I wanted to automate this further and use a configuration-based approach to allow for more flexibility in our deployments. To achieve this, I have written some Python code to parse a config file and use the Boto3 library (which the AWS CLI also uses) to replicate the functionality. However when I am trying to do similar things in Boto3 (like deploy CloudFormation and upload files to S3), I get the following error: Connection to sts.amazonaws.com timed out. Unfortunately I can't provide the full stack trace since it's on a separate network. I am running Python 3.7 and boto3-1.21-13, botocore-1.24.13.
I assume it might be because I need to setup a VPC endpoint for STS? However, I can't work out why and how the AWS CLI works fine, but Boto3 doesn't. Especially since AWS CLI uses Boto3 under the hood.
In addition, I have confirmed that I can retrieve instance metadata using curl from the EC2 instances.
To reproduce the error, this command fails for me:
python -c "import boto3;print(boto3.Session(profile_name='x').client('s3').list_objects('bucket')"
However this AWS cli command works:
aws --profile x s3 ls bucket
I guess I don't understand why the AWS CLI command works, when the boto3 command fails. Why does boto3 needs to call the sts.amazonaws.com endpoint, when the AWS CLI seemingly doesn't? What am I missing?
The aws cli and boto3 both use botocore, which is only a minor detail. Nevertheless, both the cli and boto3, when run in the same environment with the same access to the credentials, should indeed be able to reach the same endpoint.
This:
aws sts get-caller-identity --profile x
and:
python -c "import boto3;print(boto3.Session(profile_name='x').client('sts').get_caller_identity())"
are equivalent and should make the same api calls to the same endpoint.
As an aside, I find it is often best not to have your code concerned with session handling at all. It seems most simple to me for the code to expect the environment to handle that. So just export AWS_PROFILE and run the code. This prevents other user of the script from having to have the same profile and name it the same.
Yeah so it turns out I just needed to set/export AWS_STS_REGIONAL_ENDPOINTS='regional'.
After many hours of trawling the botocore and awscli source and logs, I found out that botocore sets it by default to 'legacy'.
Where as in v2 of the AWS CLI, they set it to 'regional'.
I am trying to convert a SAM project to a cloudformation template in order to call
cloudformation.createStack()
to create multiple stacks when a lambda is invoked. So far I can upload the SAM project with
sam build
sam package
But the size of the S3 is to big and I am getting errors. What are the steps to correctly upload the cloudformation template?
These pre-reqs need to be met before continuing:
Install the SAM CLI.
Create an Amazon S3 bucket to store the serverless code artifacts that the SAM template generates. At a minimum, you will need permission to put objects into the bucket.
The permissions applied to your IAM identity must include iam:ListPolicies.
4.You must have AWS credentials configured either via the AWS CLI or in your shell's environment via the AWS_* environment variables.
5.Git installed.
6.Python 3.x installed.
(Optional) Install Python's virtualenvwrapper.
Source Link:- https://www.packetmischief.ca/2020/12/30/converting-from-aws-sam-to-cloudformation/
I have a ecs task running with aws fargate. I generate some files on the container and need to upload these files to an s3 bucket.
Can I do this by installing the aws cli to the container?
I'm not sure about the following stuff:
Do I need to use some rest api (like python boto3 library) or can I use the aws console?
How should I authenticate the requests (iam and aws secrets manager?)
Do I need to use some rest api (like python boto3 library) or can I
use the aws console?
Are you asking how to install the AWS CLI into the Docker container running in ECS? You would need to update your Docker image to include the AWS CLI and then redeploy the container to ECS. The AWS API, Boto3, or the AWS console are not going to help with that task.
How should I authenticate the requests (iam and aws secrets manager?)
By assigning an IAM role to the ECS task.
We are using Jenkins and have got AWS Credentials stored in Jenkins Credential store.
I am now configuring a Build job to get the list of APIs from AWS to allow users to select the API (from parameter dropdown). To do this, I am writing a groovy script to get the list of AWS APIs. E.g.
aws apigateway get-rest-apis
But in order to run the above command, I need to first get the AWS credentials from the Jenkins credentials store. How can I do this?
(Correct me if I am wrong, the script which is going to be part of Extended parameter is going to run on Jenkins master node (and not on the Slave node) and not sure how do I get the AWS credentials)
Adding below in groovy script:
env.AWS_CREDENTIAL_ID="user_id"
user_id is id that is set in credentials for AWS account
for everyone's benefit, I used command like
env AWS_ACCESS_KEY_ID=${YourIDVariable} AWS_SECRET_ACCESS_KEY=${YourKeyVariable} aws apigateway get-rest-apis
and it worked
For a month or so, I've been studying AWS services and now I have to accomplish some basic stuff on AWS elastic beanstalk via command line. As far as I understand there are the aws elasticbeanstalk [command] and the eb [command] CLI installed on the build instance.
When I run eb status inside application folder, I get response in the form:
Environment details for: app-name
Application name: app-name
Region: us-east-1
Deployed Version: app-version
Environment ID: env-name
Platform: 64bit Amazon Linux ........
Tier: WebServer-Standard
CNAME: app-name.elasticbeanstalk.com
Updated: 2016-07-14 .......
Status: Ready
Health: Green
That tells me eb init has been run for the application.
On the other hand if I run:
aws elasticbeanstalk describe-application-versions --application-name app-name --region us-east-1
I get the error:
Unable to locate credentials. You can configure credentials by running "aws configure".
In home folder of current user there is a .aws directory with a credential file containing a [profile] line and aws_access_key_id and
aws_secret_access_key lines all set up.
Beside the obvious problem with the credentials, what I really lack is understanding of the two cli. Why is EB cli not asking for credentials and AWS cli is? When do I use one or the other? Can I use only aws cli? Any clarification on the matter will be highly appreciated.
EDIT:
For anyone ending up here, having the same problem with "Unable to locate credentials". Adding --profile profile-name option solved the problem for me. profile-name can be found in ~/.aws/config (or credentials) file on [profile profile-name] line.
In order to verify that the AWS CLI is configured on your system run aws configure and provide it with all the details it requires. That should fix your credentials problem and checking the change in configuration will allow you to understand what's wrong with your current conf.
the eb cli and the aws cli have very similar capabilities, and I too am a bit confused as to why they both should exist. From my experience the main differences are that the cli is used to interact with your AWS account using simple requests while the eb cli creates connections between you and the eb envs and so allows for finer control over them.
For instance - I've just developed a CI/CD pipeline for our beanstalk apps. When I use the eb cli I can monitor the deployment of our apps and notify the developers when it's finished. aws cli does not offer that functionality, and the only to achieve that is to repeatedly query the service until you receive the desired result.
The AWS CLI is a general tool that works on all AWS resources. It's not tied to a specific software project, the type of machine you're on, the directory you're in, or anything like that. It only needs credentials, whether they've been put there manually if it's your own machine, or generated by AWS if it's an EC2 instance.
The EB CLI is a high level tool to wrangle your software project into place. It's tied to the directory you're in, it assumes that the stuff in your directory is your project, and it has short commands that do a lot of background work to magically put everything in the right place.