#!/bin/bash
SECURITY_GROUP_ID="$(aws ec2 describe-security-groups | jq -r ' .SecurityGroups[] | select(.IpPermissions[] | .FromPort == 22 and .IpRanges[].CidrIp == "0.0.0.0/0") | .GroupId')"
aws ec2 describe-instances \
--filters "Name=network-interface.group-id,Values=${SECURITY_GROUP_ID}" \
| jq -r ".Reservations | .[] | .Instances | .[] | .InstanceId"
this is giving empty output with job showing as succeeded. The actual output should be list of all ec2 instances with security groups meeting the IpPermissions criteria. Can anyone correct this script?
Thanks
#!/bin/bash
SECURITY_GROUP_ID="$(aws ec2 describe-security-groups | jq -r ' .SecurityGroups[] | select(.IpPermissions[] | .FromPort == 22 and .IpRanges[].CidrIp == "0.0.0.0/0") | .GroupId')"
aws ec2 describe-instances \
--filters "Name=instance.group-id,Values=`echo -n $SECURITY_GROUP_ID | tr '\n' ','`" \
| jq -r ".Reservations | .[] | .Instances | .[] | .InstanceId"
I tried using ‘for’ loop in the script and describe the instances using filter and query option inside ‘for’ loop. So it will take the output from describe security groups and used it as variable in ‘for’ loop to query instances as given below.
aws ec2 describe-instances --filters Name=network-interface.group-id,Values=$sg --query …………
this worked well if you want to list instances with particular security group ids
Related
I need to get a list of running ECS tasks with their image names/tags.
Trying in 2 steps:
Extracting task ARNs
ARNS=$(aws ecs list-tasks --cluster $CLUSTER_NAME \
--desired-status 'RUNNING' --query 'taskArns' \
--output json --profile $PROFILE)
Describing tasks
aws ecs describe-tasks --cluster $CLUSTER_NAME --tasks $ARNS \
--profile $PROFILE --output table \
--query "tasks[] | [].[startedAt,containers[0].image]"
The problem is I have multiple sidecar containers in each task, and their order is inconsistent, so containers[0] returns a random container every time.
Example output
-----------------------------------------------------------------------------------------------
| DescribeTasks |
+-----------------------------------+---------------------------------------------------------+
| 2022-08-15T21:01:22.513000-07:00 | lacework/datacollector:latest-sidecar |
| 2022-08-15T21:01:21.511000-07:00 | lacework/datacollector:latest-sidecar |
| 2022-08-15T21:01:22.102000-07:00 | lacework/datacollector:latest-sidecar |
| 2022-08-15T21:01:21.743000-07:00 | 999999999999.dkr.ecr.us-east-1.amazonaws.com/bar:prod |
| 2022-08-15T21:02:02.298000-07:00 | 999999999999.dkr.ecr.us-east-1.amazonaws.com/bar:prod |
| 2022-08-15T21:02:31.743000-07:00 | 999999999999.dkr.ecr.us-east-1.amazonaws.com/bar:prod |
+-----------------------------------+---------------------------------------------------------+
Can I filter the list to keep the primary containers only, or at least sort containers in some consistent way?
A possible solution to that is to query the list by images starting with your ECR account ID.
I made it work like this:
aws ecs describe-tasks \
--cluster yourClusterName \
--output table \
--query 'tasks[] | [].[startedAt,containers[?starts_with(image, to_string(`999999999999`))].image]' \
--tasks `aws ecs list-tasks --desired-status RUNNING --query taskArns --cluster yourClusterName --output text`
Which produces an output like this one:
------------------------------------------------------------------------------
| DescribeTasks |
+----------------------------------------------------------------------------+
| 2022-10-24T17:29:16.003000+02:00 |
| 999999999999.dkr.ecr.us-east-2.amazonaws.com/business:v0.9.1 |
| 2022-10-19T17:53:46.015000+02:00 |
| 999999999999.dkr.ecr.us-east-2.amazonaws.com/datacore:v0.5.1 |
| 2022-10-24T17:30:05.670000+02:00 |
| 999999999999.dkr.ecr.us-east-2.amazonaws.com/application:v0.16.2 |
| 2022-10-24T18:53:31.795000+02:00 |
| 999999999999.dkr.ecr.us-east-2.amazonaws.com/frontend:development-v1.9.7 |
+----------------------------------------------------------------------------+
I wasn't able to fix the format of the output. JMESPath is not really my thing.
Lets say "foo" is the repository name and I want to call the image which has two tags "boo, boo-0011"
This command displays all the images in the repository:
aws ecr describe-images --repository-name foo --query "sort_by(imageDetails,& imagePushedAt)[ * ].imageTags[ * ]"
From this how do I grep only the one which has a tag "boo"
You can use --filter tagStatus=xxx but that only allows you to filter on TAGGED or UNTAGGED images, not images with a specific tag.
To find images with a specific tag, say boo, you should be able to use the somewhat inscrutable, but very helpful, jq utility. For example:
aws ecr describe-images \
--region us-east-1 \
--repository-name foo \
--filter tagStatus=TAGGED \
| jq -c '.imageDetails[] | select([.imageTags[] == "boo"] | any)'
Personally I use grep for this
aws ecr describe-images --repository-name foo --query "sort_by(imageDetails,& imagePushedAt)[ * ].imageTags[ * ]" | grep -w 'boo'
-w is the grep command for the whole word matching.
As far as I Know Amazon RDS Supports Stopping and Starting of Database Instances.
I am running the instance from a Mac OS Sierra
I want to start a DB instance using the AWS Command Line Interface (following this tutorial: http://docs.aws.amazon.com/cli/latest/reference/rds/start-db-instance.html)
But somehow I got an error:
MacBook-Pro-de-lopes:~ lopes$ aws rds start-db-instance lopesdbtest
usage: aws [options] <command> <subcommand> [<subcommand> ...] [parameters]
To see help text, you can run:
aws help
aws <command> help
aws <command> <subcommand> help
aws: error: argument operation: Invalid choice, valid choices are:
add-source-identifier-to-subscription | add-tags-to-resource
apply-pending-maintenance-action | authorize-db-security-group-ingress
copy-db-cluster-snapshot | copy-db-parameter-group
copy-db-snapshot | copy-option-group
create-db-cluster | create-db-cluster-parameter-group
create-db-cluster-snapshot | create-db-instance
create-db-instance-read-replica | create-db-parameter-group
create-db-security-group | create-db-snapshot
create-db-subnet-group | create-event-subscription
create-option-group | delete-db-cluster
delete-db-cluster-parameter-group | delete-db-cluster-snapshot
delete-db-instance | delete-db-parameter-group
delete-db-security-group | delete-db-snapshot
delete-db-subnet-group | delete-event-subscription
delete-option-group | describe-account-attributes
describe-certificates | describe-db-cluster-parameter-groups
describe-db-cluster-parameters | describe-db-cluster-snapshots
describe-db-clusters | describe-db-engine-versions
describe-db-instances | describe-db-log-files
describe-db-parameter-groups | describe-db-parameters
describe-db-security-groups | describe-db-snapshot-attributes
describe-db-snapshots | describe-db-subnet-groups
describe-engine-default-cluster-parameters | describe-engine-default-parameters
describe-event-categories | describe-event-subscriptions
describe-events | describe-option-group-options
describe-option-groups | describe-orderable-db-instance-options
describe-pending-maintenance-actions | describe-reserved-db-instances
describe-reserved-db-instances-offerings | download-db-log-file-portion
failover-db-cluster | list-tags-for-resource
modify-db-cluster | modify-db-cluster-parameter-group
modify-db-instance | modify-db-parameter-group
modify-db-snapshot-attribute | modify-db-subnet-group
modify-event-subscription | promote-read-replica
purchase-reserved-db-instances-offering | reboot-db-instance
remove-source-identifier-from-subscription | remove-tags-from-resource
reset-db-cluster-parameter-group | reset-db-parameter-group
restore-db-cluster-from-snapshot | restore-db-cluster-to-point-in-time
restore-db-instance-from-db-snapshot | restore-db-instance-to-point-in-time
revoke-db-security-group-ingress | add-option-to-option-group
remove-option-from-option-group | wait
help
Invalid choice: 'start-db-instance', maybe you meant:
* reboot-db-instance
* create-db-instance
You need to update to the latest version of the AWS CLI tool. The version you currently have installed was released before the RDS start/stop feature was available.
It is a new feature (Announced on Jun 1, 2017). You have to upgrade your AWS CLI.
Amazon RDS Supports Stopping and Starting of Database Instances
I have been launching EC2 instances by logging in to the AWS site, hitting the "Launch" button and following the proscribed steps. Now I'd like to launch instance from an Ansible script, and to do this I (think I) need the AMI ID of the image I wish to launch.
The problem is that I am launching an image from the "Marketplace", and I cannot find the AMI ID. In particular I'm using the Centos 7 image. This is easy to find in the web interface, just go to the marketplace and search for "centos", the image I want is the first one found, but the information provided about the image doesn't seem to include the AMI ID that I need to launch it from a script. The workaround is to manually launch an image, and then when inspecting the running image, the AMI ID is given. But is there an easier way to find it?
CentOS publishes their AMI product codes to their wiki. The wiki provides the following information for the latest CentOS 7 AMI:
Owner: aws-marketplace
Product Code: aw0evgkw8e5c1q413zgy5pjce
Using this information, we can query describe-images with the AWS CLI:
Example:
aws ec2 describe-images \
--owners 'aws-marketplace' \
--filters 'Name=product-code,Values=aw0evgkw8e5c1q413zgy5pjce' \
--query 'sort_by(Images, &CreationDate)[-1].[ImageId]' \
--output 'text'
Output:
ami-6d1c2007
This query returns a single AMI ID, selected by sorting the collection by creation date and then selecting the last (most recent) element in the collection.
Per the CentOS wiki, multiple AMI ids may be associated with a product key, so while this query would currently only return a single AMI because only one matching this product currently exists... in the future if a new AMI is created for this product code for any reason this query will return it instead.
Check in this page
or
AWS has been changing the “aws marketplace” UI. Now (2022) We get the ami id in the “configure this software” Page in the AWS Marketplace itself.
Steps:
Search and locate the image. Click on the image.
On the top right, click “continue to subscribe”
On the next window, click on “Continue to configuration”
On the next window, Find the Ami ID located. We can change software versions to get different ami Ids.
If you are looking for all Centos7 images
$ aws ec2 describe-images \
--owners aws-marketplace \
--filters Name=product-code,Values=aw0evgkw8e5c1q413zgy5pjce \
--query 'Images[*].[CreationDate,Name,ImageId]' \
--filters "Name=name,Values=CentOS Linux 7*" \
--region us-west-2 \
--output table \
| sort -r
| 2018-06-13T15:58:14.000Z| CentOS Linux 7 x86_64 HVM EBS ENA 1805_01-b7ee8a69-ee97-4a49-9e68-afaee216db2e-ami-77ec9308.4 | ami-3ecc8f46 |
| 2018-05-17T09:30:44.000Z| CentOS Linux 7 x86_64 HVM EBS ENA 1804_2-b7ee8a69-ee97-4a49-9e68-afaee216db2e-ami-55a2322a.4 | ami-5490ed2c |
| 2018-04-04T00:11:39.000Z| CentOS Linux 7 x86_64 HVM EBS ENA 1803_01-b7ee8a69-ee97-4a49-9e68-afaee216db2e-ami-8274d6ff.4 | ami-0ebdd976 |
| 2017-12-05T14:49:18.000Z| CentOS Linux 7 x86_64 HVM EBS 1708_11.01-b7ee8a69-ee97-4a49-9e68-afaee216db2e-ami-95096eef.4 | ami-b63ae0ce |
add to it similar query available for RedHat
aws ec2 describe-images \
--owners 309956199498 \
--query 'Images[*].[CreationDate,Name,ImageId]' \
--filters "Name=name,Values=RHEL-7.?*GA*" \
--region us-east-1 \
--output table \
| sort -r
| 2018-03-23T20:42:08.000Z | RHEL-7.5_HVM_GA-20180322-x86_64-1-Hourly2-GP2 | ami-6871a115 |
| 2017-08-08T15:37:31.000Z | RHEL-7.4_HVM_GA-20170808-x86_64-2-Hourly2-GP2 | ami-c998b6b2 |
| 2017-07-24T15:44:39.000Z | RHEL-7.4_HVM_GA-20170724-x86_64-1-Hourly2-GP2 | ami-cdc999b6 |
| 2016-10-26T22:32:29.000Z | RHEL-7.3_HVM_GA-20161026-x86_64-1-Hourly2-GP2 | ami-b63769a1 |
| 2015-11-12T21:06:58.000Z | RHEL-7.2_HVM_GA-20151112-x86_64-1-Hourly2-GP2 | ami-2051294a |
| 2015-02-25T20:24:23.000Z | RHEL-7.1_HVM_GA-20150225-x86_64-1-Hourly2-GP2 | ami-12663b7a |
| 2015-02-09T22:54:40.000Z | RHEL-7.0_HVM_GA-20150209-x86_64-1-Hourly2-GP2 | ami-60a1e808 |
| 2014-10-17T20:29:24.000Z | RHEL-7.0_HVM_GA-20141017-x86_64-1-Hourly2-GP2 | ami-a8d369c0 |
| 2014-05-28T19:17:11.000Z | RHEL-7.0_GA_HVM-x86_64-3-Hourly2
| ami-785bae10 |
I used the other answers here for a good way to test AMI lookups in Terraform.
Using...
aws ec2 describe-images \
--owners aws-marketplace \
--filters '[
{"Name": "name", "Values": ["CentOS Linux 7*"]},
{"Name": "virtualization-type", "Values": ["hvm"]},
{"Name": "architecture", "Values": ["x86_64"]},
{"Name": "image-type", "Values": ["machine"]}
]' \
--query 'sort_by(Images, &CreationDate)[-1]' \
--region us-east-1 \
--output json
...gave me the chance to trial and error my lookup of
data "aws_ami" "centos" {
most_recent = true
owners = ["aws-marketplace"]
filter {
name = "virtualization-type"
values = ["hvm"]
}
filter {
name = "architecture"
values = ["x86_64"]
}
filter {
name = "image-type"
values = ["machine"]
}
filter {
name = "name"
values = ["CentOS Linux 7*"]
}
}
resource "aws_launch_configuration" "launch_configuration" {
name_prefix = "${var.name}-"
image_id = "${data.aws_ami.centos.image_id}"
instance_type = "t2.nano"
iam_instance_profile = "${aws_iam_instance_profile.instance_profile.name}"
security_groups = ["${aws_security_group.lc_security_group.id}"]
user_data = "${data.template_file.user_data.rendered}"
lifecycle {
create_before_destroy = true
}
}
I'm trying to remove all my AWS EC2 snapshots except the last 6 with this script:
#!/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
# Backup script
Volume="{VOL-DATA}"
Owner="{OWNER}"
Description="{DESCRIPTION}"
Local_numbackups=6
Local_region="us-west-1"
# Remove old snapshots associated to a description, keep the last $Local_numbackups
aws ec2 describe-snapshots --filters Name=description,Values=$Description | grep "SnapshotId" | head -n -$Local_numbackups | awk '{print $2}' | sed -e 's/,//g' | xargs -n 1 -t aws ec2 delete-snapshot --snapshot-id
However it doesn't work. It deletes instances, but not the oldest ones. Why?
You're trying to do something too complex to be handled (gracefully) in one line, so we'll need to break it down a bit. First, let's get the snapshots sorted by age, oldest to newest:
aws ec2 describe-snapshots --filters Name=description,Values=$Description --query 'Snapshots[*].[StartTime,SnapshotId]' --output text | sort -n
Then we can drop the StartTime field to get the snapshot ID alone:
aws ec2 describe-snapshots --filters Name=description,Values=$Description --query 'Snapshots[*].[StartTime,SnapshotId]' --output text | sort -n | sed -e 's/^.*\t//'
head (or tail) aren't really suitable for discarding the fixed number of snapshots we want to keep. We need to filter those out another way. So, putting it altogether:
# Get array of snapshot IDs sorted by age (oldest to newest)
snapshots=($(aws ec2 describe-snapshots --filters Name=description,Values=$Description --query 'Snapshots[*].[StartTime,SnapshotId]' --output text | sort -n | sed -e 's/^.*\t//'))
# Get number of snapshots
count=${#snapshots[#]}
if [ "$count" -lt "$Local_numbackups" ]; then
echo "We already have less than $Local_numbackups snapshots"
exit 0
else
# Drop the last (newest) $Local_numbackups IDs from the array
snapshots=(${snapshots[#]:0:$((count - Local_numbackups))})
# Loop through the remaining snapshots and delete
for snapshot in ${snapshots[#]}; do
aws ec2 delete-snapshot --snapshot-id $snapshot
done
fi
(While it's obviously possible to do this in bash with the AWS CLI, it's complex enough that I'd personally rather use a more robust language and the AWS SDK.)
Here is a sample.
days2keep="30"
region="us-west-2"
name="jdoe"
#date - -v is for Osx
cutoffdate=`date -j -v-${days2keep}d '+%Y-%m-%d'`
echo "Finding list of snapshots before $cutoffdate "
oldsnapids=$(aws ec2 describe-snapshots --region $region --filters Name=tag:Name,Values=$name --query Snapshots[?StartTime\<=\`$cutoffdate\`].SnapshotId --output text)
for snapid in $oldsnapids
do
echo Deleting snapshot $snapid
aws ec2 delete-snapshot --snapshot-id $snapid --region $region
done
We can delete all old snapshots using below steps:-
List out all snapshots ID's they are old and put in one file like:- /opt/snapshot.txt
And then use "aws configure" command for setup access AWS account from command line, at this time we need to provide credentials:-
Such as:
AWS Access Key ID [None]: XXXXXXXXXXXXXXXXXX
AWS Secret Access Key [None]: XXXXXXXXXXXXXXXXXXXXX
Default region name [None]: XXXXXXXXXXXXXXXX
After that we can use below shell script, we need to give snapshots ID's file name
Codes:
#!/bin/bash
list=$(cat /opt/snapshot.txt)
for i in $list
do
aws ec2 delete-snapshot --snapshot-id $i
if [ $? -eq 0 ]; then
echo Going Good
else
echo FAIL
fi
done
Thanks