How do I get the latest Amazon LInux 2 ami using Ansible? - amazon-web-services

I'm using Ansible to provision AWS EC2 instances with the Amazon.Aws collection. On the cli, if I want to get the latest Amazon Linux 2 AMI I may run something like this:
$ aws ec2 describe-images \
--region us-east-1 \
--owners amazon \
--query 'reverse(sort_by(Images, &CreationDate))[:1]' \
--filters 'Name=name,Values=amzn2-ami-hvm-2.0.*-x86_64-gp2'
How do I make an equivalent query using Ansible's amazon.aws collection?
The amazon.aws.ec2_ami_info module seems to do what I want, but the provided example in the docs makes it seem like the lookup requires an OwnerId.
- name: Gather a list of all Amazon Linux \2 AMIs
amazon.aws.ec2_ami_info:
owners: 137112412989
filters:
name: amzn2-ami-hvm-*-x86_64-gp2
The OwnerId has to be retrieved with a cli command, like this:
$ aws ec2 describe-images \
--region us-east-1 \
--owners amazon \
--filters 'Name=name,Values=amzn2-ami-hvm-2.0.*-x86_64-gp2' \
--query 'reverse(sort_by(Images, &CreationDate))[:1].OwnerId' \
--output text
137112412989
The problem is, I want to run that playbook on a node that doesn't have the awscli binary. I also don't want to hardcode the OwnerId.

There were two missing peices - extracting the element with the most recent creation time, and specifying the owner using a name rather than the OwnerId. Here's a working solution.
- name: 'Find the latest Amazon Linux 2 AMI'
block:
- name: 'Get all Amazon Linux 2 AMIs'
amazon.aws.ec2_ami_info:
owners:
- amazon
filters:
name: amzn2-ami-hvm-*-x86_64-gp2
architecture: x86_64
block-device-mapping.volume-type: gp2
virtualization-type: hvm
register: amis
- name: Extract the most recently created AMI from the list
debug:
msg: "{{ amis.images[-1].image_id }}"
register: latest_amzl2_ami

Related

Falco output aws instance metadata

I run falco and falcosidekick with docker compose, without k8s.
I need to retrive aws instance metadata to falco rules output.
I've found the jevt field class but I encountered an error on falco container start
Invalid output format 'command=%jevt.value[/awsRegion': 'invalid formatting token jevt.value[/awsRegion']
Here my rules:
- rule: Terminal shell in container
desc: A shell was used as the entrypoint/exec point into a container with an attached terminal.
condition: >
spawned_process and container
and shell_procs and proc.tty != 0
and container_entrypoint
and not user_expected_terminal_shell_in_container_conditions
output: >
command=%jevt.value["/awsRegion"]
priority: NOTICE
tags: [ container, shell, mitre_execution ]
How can I do?
Thank you
several things to know:
the syntax for jevt.value is jevt.value[/awsRegion] (no quotes)
these kind fields are for events in json format, it works for kubernetes audit logs but in your case where the rule is based on syscalls
falco will not query aws metadata either, you will not have this information in your output like this
Regards,
Falco doesn't query AWS metadata, so I retrieved the metadata with an aws cli describe-instances and passed the metadata to falcosidekick container.
#loading EC2 metadata
INSTANCE_ID=$(curl http://169.254.169.254/latest/meta-data/instance-id)
INSTANCE_IP=$(aws ec2 describe-instances --instance-id "$INSTANCE_ID" --region eu-west-1 --query 'Reservations[*].Instances[*].{InstanceIp:PublicIpAddress}' --output text)
CLUSTER_NAME=$(aws ec2 describe-instances --instance-id "$INSTANCE_ID" --region eu-west-1 --query 'Reservations[*].Instances[*].{ClusterName:Tags[?Key==`Name`]|[0].Value}' --output text)
docker run -d -p 2801:2801 -d \
-e CUSTOMFIELDS=INSTANCE_ID:"$INSTANCE_ID",INSTANCE_IP:"$INSTANCE_IP",CLUSTER_NAME:"$CLUSTER_NAME" \
--name falcosidekick \
falcosecurity/falcosidekick

Authfailure: AWS was not able to validate the provided access credentials

I am trying to create my Gitlab CI/CD pipeline with AWS. The goal is to Terminate the Existing EC2 Instance, Run a new instance from a template, then Associate an Elastic IP to the new EC2. The runner I'm using is a Docker runner using the python:latest image. When I run my CI/CD pipeline I get
An error occurred (AuthFailure) when calling the DescribeInstances operation: AWS was not able to validate the provided access credentials
My .gitlab-ci.yml is as follows:
stages:
- build
AWS_Install:
image: python:latest
stage: build
tags:
- Docker
script:
- pip install awscli
- export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
- export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
- export AWS_DEFAULT_REGION=$AWS_DEFAULT_REGION
- echo "running script :)"
- OLDEC2=$(aws ec2 describe-instances --filters "Name=instance-state-name,Values=running" --query "Reservations[*].Instances[*].[InstanceId]" --output text)
- aws ec2 terminate-instances --instance-ids "$OLDEC2"
- sleep 200.0
- aws ec2 run-instances --launch-template LaunchTemplateId=[launch-template-id],Version=12
- sleep 120.0
- NEWEC2=$(aws ec2 describe-instances --filters "Name=instance-state-name,Values=running" --query "Reservations[*].Instances[*].[InstanceId]" --output text)
- aws ec2 associate-address --allocation-id [allocation-id] --instance-id "$NEWEC2" --allow-reassociation
What I've checked/tried:
- AWS credentials: They are correct and valid
- aws configure: Everything sets correctly (checked using aws configure get)
- Ensured UNIX line endings were being used
- Adding a variable section to the YAML file to include environment variables
- Hardcoding credential values
- New user on AWS with all necessary credentials
- Using export to get the environment variables
- Running everything in one script rather than having a before script
- Having multiple stages/Jobs
Turns out the solution was to use a public runner on GitLab rather than a customer one.

Finding the full name of windows server

This command lists hundreds of windows servers. How do I select the most popular ones those are displayed on web console while I create a new instance?
# aws ec2 describe-images --owners amazon --filters "Name=name,Values=Windows_Server*" --query 'sort_by(Images, &CreationDate)[].Name'
[
"Windows_Server-2016-English-Full-ECS_Optimized-2017.11.24",
"Windows_Server-2016-English-Full-ECS_Optimized-2018.01.10",
"Windows_Server-2016-English-Full-ECS_Optimized-2018.02.21",
"Windows_Server-2016-English-Full-ECS_Optimized-2018.03.26",
"Windows_Server-2016-English-Nano-Base-2018.04.11",
...
...
]
I am looking for the full name and not just the ami-id.
For e.g. which one of the above is "ami-04ca2d0801450d495"?
The DescribeImages API call returns the name of the AMI along with the rest of the info. To extract just the name of the AMI, you can run the following command:
aws ec2 describe-images --image-ids $IMAGE_ID \
--output text --query 'Images[*].Name'
Details about the describe-images command can be found here.
This command will return the full name of the given ami ID
aws ssm get-parameters-by-path --path "/aws/service/ami-windows-latest" --region us-east-1 | grep -C3 '04ca2d0801450d495'

Unable to tag EBS Volumes using UserData bash script

I have been trying to tag EBS Volumes attached to EC2 instances in the CloudFormation UserData section. Here was my first attempt:
Example 1:
AWS_INSTANCE_ID=`curl -s http://169.254.169.254/latest/meta-data/instance-id`
ROOT_DISK_ID=`aws ec2 describe-volumes \
--filter Name=attachment.instance-id,Values="${AWS_INSTANCE_ID}" \
--query "Volumes[].VolumeId" --region us-east-1 --out text`
aws ec2 create-tags --resources "${ROOT_DISK_ID}" \
--tags 'Key=VolumeTagName,Value=VolumeTagValue' --region us-east-1
This resulted in a Template format error: Unresolved resource dependencies [AWS_INSTANCE_ID, ROOT_DISK_ID] in the Resources block of the template error.
A post I came across mentioned that using the ! when calling the variable in the Cloudformation UserData script will get around this, so it now looks like this:
Example 2:
AWS_INSTANCE_ID=`curl -s http://169.254.169.254/latest/meta-data/instance-id`
ROOT_DISK_ID=`aws ec2 describe-volumes \
--filter Name=attachment.instance-id,Values="${!AWS_INSTANCE_ID}" \
--query "Volumes[].VolumeId" --region us-east-1 --out text`
aws ec2 create-tags --resources "${!ROOT_DISK_ID}" \
--tags 'Key=VolumeTagName,Value=VolumeTagValue' --region us-east-1
This gets around that error, yet still, no tags appear on the Volume attached to an instance launched with this template. If I ssh into the instance and run Example 1, it works just fine. Example 2 does not give me any errors to work with.
What am I doing wrong in bash, that is specific to Cloudformation?
If I understand correctly you're trying to create your script using cloudformation, and then executing it on the ec2-instance on startup. Using yaml, this is my userdata section:
UserData: !Base64
Fn::Join:
- ''
- - "#!/bin/bash -xe \n"
- "cat << 'EOF' > /home/ec2-user/script.sh \n"
- "AWS_INSTANCE_ID=`curl -s http://169.254.169.254/latest/meta-data/instance-id`\n"
- "ROOT_DISK_ID=`aws ec2 describe-volumes "
- "--filter Name=attachment.instance-id,Values=\"${AWS_INSTANCE_ID}\" "
- "--query \"Volumes[*].[VolumeId]\" --region eu-west-1 --out text`\n"
- "aws ec2 create-tags --resources \"${ROOT_DISK_ID}\" "
- "--tags 'Key=MyAutoTagName,Value=MyAutoTagValue' --region eu-west-1\n"
- "EOF\n"
- "chmod +x /home/ec2-user/script.sh\n"
- "/home/ec2-user/script.sh\n"
I changed the region due to the region I'm using.
If I view the contents of my script.sh file I get the below:
AWS_INSTANCE_ID=`curl -s http://169.254.169.254/latest/meta-data/instance-id`
ROOT_DISK_ID=`aws ec2 describe-volumes --filter Name=attachment.instance-id,Values="${AWS_INSTANCE_ID}" --query "Volumes[*].[VolumeId]" --region eu-west-1 --out text`
aws ec2 create-tags --resources "${ROOT_DISK_ID}" --tags 'Key=MyAutoTagName,Value=MyAutoTagValue' --region eu-west-1
The only difference I can see is your "Volumes[].[VolumeId]*", I'm not sure what your userdata section looks like, so it may be issues with escaping.
Using my UserData section above the tag was created as soon as the instance was spun up and userdata section ran.

How to create EC2 instance without given Image-Id in AWS template?

I want to create a RHEL OS EC2 Instance using AWS template. I don't have any RHEL instance currently. So don't have any Image-ID.
Red Hat maintains RHEL AMIs. We can use the CLI describe-images to query their public AMIs based on:
Account ID: 309956199498
A known string pattern matching their AMI names: RHEL-*_HVM_GA-*-Hourly2-GP
For the sake of this example, we will sort the images by CreationDate, request only the last element in the collection (via -1) and filter the results down to Name, ImageId, and CreationDate.
Example:
aws ec2 describe-images \
--owners 309956199498 \
--filters "Name=name,Values=RHEL-*_HVM_GA-*-Hourly2-GP2" \
--query 'sort_by(Images, &CreationDate)[-1].[Name, ImageId, CreationDate]' \
--output text
Output:
RHEL-7.3_HVM_GA-20161026-x86_64-1-Hourly2-GP2 ami-b63769a1 2016-10-26T22:32:29.000Z
To verify this is correct, you can double-check by visiting the 'Quick Start' section of the AWS Console's EC2 'Launch Instances' Wizard and checking the most recent RHEL AMI that is sorted near the top of this quick start listing. At the time of this posting, the most recent RHEL AMI was ami-b63769a1.
After this, you would take that resulting ImageId and use it as part of your request to launch a new instance.
Further Reading:
AWS Documentation - Finding a Linux AMI