I am trying to create my Gitlab CI/CD pipeline with AWS. The goal is to Terminate the Existing EC2 Instance, Run a new instance from a template, then Associate an Elastic IP to the new EC2. The runner I'm using is a Docker runner using the python:latest image. When I run my CI/CD pipeline I get
An error occurred (AuthFailure) when calling the DescribeInstances operation: AWS was not able to validate the provided access credentials
My .gitlab-ci.yml is as follows:
stages:
- build
AWS_Install:
image: python:latest
stage: build
tags:
- Docker
script:
- pip install awscli
- export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
- export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
- export AWS_DEFAULT_REGION=$AWS_DEFAULT_REGION
- echo "running script :)"
- OLDEC2=$(aws ec2 describe-instances --filters "Name=instance-state-name,Values=running" --query "Reservations[*].Instances[*].[InstanceId]" --output text)
- aws ec2 terminate-instances --instance-ids "$OLDEC2"
- sleep 200.0
- aws ec2 run-instances --launch-template LaunchTemplateId=[launch-template-id],Version=12
- sleep 120.0
- NEWEC2=$(aws ec2 describe-instances --filters "Name=instance-state-name,Values=running" --query "Reservations[*].Instances[*].[InstanceId]" --output text)
- aws ec2 associate-address --allocation-id [allocation-id] --instance-id "$NEWEC2" --allow-reassociation
What I've checked/tried:
- AWS credentials: They are correct and valid
- aws configure: Everything sets correctly (checked using aws configure get)
- Ensured UNIX line endings were being used
- Adding a variable section to the YAML file to include environment variables
- Hardcoding credential values
- New user on AWS with all necessary credentials
- Using export to get the environment variables
- Running everything in one script rather than having a before script
- Having multiple stages/Jobs
Turns out the solution was to use a public runner on GitLab rather than a customer one.
Related
I'm trying to run the following command from gitlab CI:
$ aws ecr get-login-password --region eu-west-3
Here's how the job in the .gitlab-ci.yml looks like this
publish-job:
stage: publish
image:
name: amazon/aws-cli:latest
entrypoint: [""]
script:
- aws configure set aws_access_key_id MY_ACCESS_KEY_ID
- aws configure set aws_secret_access_key MY_SECRET_ACCESS_KEY
- aws configure set default.region eu-west-3
- aws ecr get-login-password --region eu-west-3
And at the last step I get the following error:
$ aws ecr get-login-password --region eu-west-3
An error occurred (UnrecognizedClientException) when calling the GetAuthorizationToken operation: The security token included in the request is invalid.
I know there's a similar question on stack overflow but I think it's not the same problem. In that question it's an issue that has to do with permissions. In my case I'm pretty sure it isn't for 2 reasons:
I gave the user associated with the access key AdministratorAccess (temporarily in order to rule out the possibility that I'm dealing with an permissions issue)
I performed the exact same steps (by copying and pasting) with docker and it works, so it's not the credentials. Here's the Dockerfile:
FROM amazon/aws-cli:latest
RUN aws configure set aws_access_key_id THE_SAME_ACCESS_KEY_ID
RUN aws configure set aws_secret_access_key THE_SAME_SECRET_ACCESS_KEY
RUN aws configure set default.region eu-west-3
RUN aws ecr get-login-password --region eu-west-3
Then I ran $ docker build --progress=plain . and the last step returned a hash
Any Idea why those steps give inconsistent results? And how to fix the CI?
I declared an AWS_DEFAULT_REGION environment variable that was preventing the cli from executing the command (even though I hardcoded the credentials at this stage). When I removed the environment variable, everything started working properly.
I am looking to run aws rds describe-db-clusters --db-cluster-identifier CLI command with wildcard. Something like:
aws rds describe-db-clusters --db-cluster-identifier prod% --region us-east-1
I want to retrieve info about all the RDS clusters whose name start with prod. When I run the above cli command, I get an error
An error occurred (InvalidParameterValue) when calling the DescribeDBClusters operation: Invalid database cluster identifier: prod%
Is there a way (via CLI or Py Code) to get the list of all RDS Clusters whose name start with prod?
Thanks
I am running jenkins off an ec2 image and building a docker image to push into ecr
I keep getting this error
Running shell script
+ aws ecr get-login --no-include-email --region us-east-2
Unable to locate credentials. You can configure credentials by running "aws configure".
I have tried to create the credentials file
ie
touch ~/.aws/credentials and echo >> to the file
I have tried with
--build-arg AWS_ACCESS_KEY_ID=xxxxxxx
and I have also added my credentials into Jenkins
Please any help would be appreciated
You can fix this by configuring aws cli by jenkins user. Just fire this command as jenkins user and mention your access key and secret key along with the AWS region.
First change to Jenkins user
[root#symphony ~]# su -s /bin/bash jenkins
Then configure AWS CLI
[jenkins#symphony ~]$ aws configure
AWS Access Key ID [None]: xxxxxxxxxxxxxxxx
AWS Secret Access Key [None]: yyyyyyyyyyyy
Default region name [None]: us-east-2
Default output format [None]:
Verify the aws cli with this command
[jenkins#symphony ~]$ aws ecr get-login --no-include-email --region us-east-2
NOTE: Make sure your user has access to ECR and ECS.
I am very new to AWS. I have a Windows Server EC2 instance. I installed AWS CLI on my laptop. Then I opened a CMD window, typed in "aws configure", put in the access key credentials, and was able to connect to the EC2.
From here, how do I get the http://169.254.169.254/latest/meta-data working? How do I retrieve some meta data?
On your Laptop
On your local machine you only can use the cli to retrieve metadata about your instance. Simply use this aws cli command:
aws ec2 describe-instance-attribute --instance-id <your-ec_instance_id e.g. i-ab12345> --attribute instanceType --region <your_region e.g. eu-west-1>
Documentation: http://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instance-attribute.html
On your EC2-Instance only:
On your instance you can use the cli (like above) and the following:
PowerShell >3.0:
Invoke-RestMethod -uri http://169.254.169.254/latest/meta-data/instance-type
Documentation: http://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/ec2-instance-metadata.html
Or you can install "curl for windows" and run:
curl http://169.254.169.254/latest/meta-data/instance-type
When running on an EC2 instance, you can query the metadata service, like so:
curl http://169.254.169.254/latest/meta-data/public-ipv4
You can also use:
curl http://instance-data/latest/meta-data/public-ipv4
From outside the EC2 instance, you can use the awscli, like so:
aws ec2 describe-instances
--instance-ids i-01234567890123456
--query "Reservations[0].Instances[0].PublicIpAddress"
--output text
You cannot use http://169.254.169.254/latest/meta-data from AWS cli on your laptop
Use the ec2 describe-instances command instead for getting instance details
More details here
I have found a script for starting/stopping a dynamically created ec2 instance, but how do I start any instances in my inventory?
Seems you are talking about scripting, not SDK. So there are two tools to do the job.
1 AWS CLI tools
download aws cli tool and set the API Key in $HOME/.aws/credentials
list all instances on region us-east-1
Confirm which instances you are targeting.
aws ec2 describe-instances --query 'Reservations[].Instances[].InstanceId' --region us-east-1 --output text
2 Amazon EC2 Command Line Interface Tools
download and setup instruction
list all instances on region us-east-1
You should get same output as WAY #1.
ec2-describe-instances --region us-west-2 |awk '/INSTANCE/{print $2}'
With the instance ID list, you can use your command to start them one by one.
for example, the instance name are saved in file instance.list
while read instance
do
echo "Starting instance $instance ..."
ec2-start-instances "$linstance"
done < instance.list
BMW, give you an excellent startup, but you can even summarise the thing like this:
1) First get the id of all the instances and save them into a file
aws ec2 describe-instances --query 'Reservations[].Instances[].InstanceId' --region us-east-1 --output text >> id.txt
2) Then simply run this command to start all the instances
for id in $(awk '{print $1}' id.txt); do echo "starting the following instance $id"; aws ec2 start-instances --instance-ids --region us-east-1 $id; done
Please change the region, I am considering that you have installed and setup the AWS CLI tools properly. Thanks