I am trying to find out all EC2 instances in 10 different accounts which are running non-amazon AMI images. Following CLI command gives me the list of all AMI's:
aws ec2 describe-instances --output text --query 'Reservations[*].Instances[*].[ImageId]' | sort | uniq -c
I think I can modify this further to get all non-amazon AMI's but is there a way to run this across 10 different accounts in one call?
is there a way to run this across 10 different accounts in one call?
No, that's not possible. You need to write a loop that iterates over each account, calling ec2 describe-instances once for each account.
Here's a script that can find instances using AMIs where the Owner is not amazon:
import boto3
ec2_client = boto3.client('ec2', region_name='ap-southeast-2')
instances = ec2_client.describe_instances()
# Get a set of AMIs used on all the instances
images = set(i['ImageId'] for r in instances['Reservations'] for i in r['Instances'])
# Find which of these are owned by Amazon
amis = ec2_client.describe_images(ImageIds=list(images), Owners=['amazon'])
amazon_amis = [i['ImageId'] for i in amis['Images']]
# Which instances are not using Amazon images?
non_amazon_instances = [(i['InstanceId'], i['ImageId']) for r in instances['Reservations'] for i in r['Instances'] if i['ImageId'] not in amazon_amis]
for i in non_amazon_instances:
print(f"{i[0]} uses {i[1]}")
A few things to note:
Deprecated AMIs might not have accessible information, so might be marked a non-Amazon.
This script, as written, only works on one region. You could change it to loop through regions.
This script, as written, only works on one account. You would need a way to loop through credentials for other accounts.
Use AWS config
Create an agregator in root or delegated account(wait for the agregator to load)
Create query
SELECT
accountId,
resourceId,
configuration.keyName,
availabilityZone
WHERE
resourceType = 'AWS::EC2::Instance'
AND configuration.state.name = 'running'
more details
https://aws.amazon.com/blogs/mt/org-aggregator-delegated-admin/
Related
Our previous Devops created several Key Pairs in EC2. It looks like some of them are not used anymore. So I would like to delete them. How can I find when the Keypairs were created and if they are currently used (preferably in the Console)?
This will provide a list of AWS KeyPairs that are in use.
aws ec2 --profile default describe-key-pairs --query KeyPairs[].[KeyName] --output text |xargs -I {} aws ec2 --profile default describe-instances --filters Name=key-name,Values={} --query Reservations[].Instances[].[KeyName,InstanceId] --output text| uniq
It gets a list of KeyPairs and uses that output to match servers using those Keys. If a KeyPair is not used it will not appear in the list.
Output:
fake_key
second-fake-key
This will match servers that are shut off too.
And I wanted to see what this would look like in Python so here you go. This will get a list of keys and output keys that are not used.
#! /usr/bin/env python
import boto3
region = 'us-east-1'
session = boto3.Session(profile_name='default')
ec2 = session.client('ec2')
response = ec2.describe_key_pairs()['KeyPairs']
for key in response:
found_instance = ec2.describe_instances(
Filters=[
{
'Name': 'key-name',
'Values': [key['KeyName']]
}
]
)['Reservations']
if len(found_instance) == 0:
print (key['KeyName'] + " is unused")
Output:
fake-key is unused
Keypairs are a feature of Linux, not AWS.
When an Amazon EC2 instance is launched from an Amazon Linux AMI, there is some code on the instance that copies the nominated keypair into the /users/ec2-user/.ssh/authorized_keys file. After this, it is just normal Linux.
So, the only way to know if an instance permits login via a particular keypair is to look on every instance, in every user's .ssh/authorized_keys file to look for that keypair.
It should be mentioned that AWS-generated keypairs are not a recommended way to manage logins to instances on an on-going basis. Your organization would probably have an Active Directory or LDAP server, so instances should be configured to use these authentication services rather than AWS keypairs. This way, it would be very simply to deactivate users in a central location rather than having to visit each instance individually.
Bottom line: Follow your normal security procedures on Amazon EC2 as well as on-premises.
Currently, AWS does not provide any way to retrieve the date and time as to when the Key Pair was created. In your case, what you can do is, check the key pairs created for the instances in the EC2 console and delete the rest of the key pairs which are not being used.
I have multi-account AWS environment (set up using AWS Landing Zone) and I need to copy a specific security group to all the accounts. I do have a CFT written, but it's too much of a repetitive task to do this one by one.
The security group is in the central (shared-services) account, which has access to all the other accounts. It's better if there's a way to integrate this to Account Vending Machine (AVM) in order to avoid future tasks of exporting the SG to newly spawned accounts.
You should use CloudFormation Stacksets. StackSets is a feature of cloudformation in which you have a master account in which you create/update/remove the stackset, and you have children accounts. In the stackset, you configure your children aws accounts you want to deploy the CF template and the region as well.
From your comment, your master account is going to be the shared-services and the rest of your accounts, the children ones. You will need to deploy a couple of IAM roles to allow cross-account access, but after that, you will be able to deploy all your templates in up to 500 aws accounts automatically and update them as well.
More information here: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/what-is-cfnstacksets.html
You can export Security Group and other configuration with CloudFormation using CloudFormer, which creates a template from the existing account configuration. Check the steps in this guide https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-using-cloudformer.html It will upload the template on S3 and you can reuse it or some of its parts.
Since you are using AWS Landing Zone, you can add the security group to the aws_baseline templates, either as a new template or added to one of the existing files. When submitted, AWS Landing Zone uses Step Functions and AWS Stack Sets to deploy your SG to all existing and future accounts. If you choose to create the security group in a new file, you must add a reference to it in the manifest.yaml file (compare with how the other templates are referenced).
I was able to do this via the Account Vending Machine. But AGL's Stacksets should be a good alternative too.
To copy AWS Security Gp from one account of any region to other AWS account to any region is required lots of scripting(coding) in aws cli or boto3.
But one thing i done which is feasible to my usecase(Whitelist 14 IPs for HTTPS) is write a bash script
Here prior i create a blank SG on other AWS account(or u may use aws cli to create that too),
`
Region1=
SGFromCopy=
profilefromcopy=
Region2=
SGToCopy=
profiletocopy=
for IP in $(aws ec2 describe-security-groups --region $Region1 --group-id=$SGFromCopy --profile $profilefromcopy --query SecurityGroups[].IpPermissions[].IpRanges[].CidrIp --output text) ;
do
aws ec2 authorize-security-group-ingress --group-id=$SGToCopy --ip-permissions IpProtocol=TCP,FromPort=443,ToPort=443,IpRanges=[{CidrIp=$IP}] --profile $profiletocopy --region $Region2;
done ;
`
U may modify script if u have csv formated of SG and then just had iterated in while loop
BONUS
To get desire output you have to alter output in file or some where else
> aws ec2 describe-security-groups --region $Region1 --group-id $SGFromCopy --profile $profilefromcopy --query SecurityGroups[].IpPermissions[] --output text
I would like to auto-tag certain AWS resources defined in a CloudFormation template with the user who uses the template to create a stack. Is it possible to access any sort of user id in the template?
I don't think this is possible.
AWS services are tied to AWS Accounts. Once IAM confirms that a particular user has permission to make an API call, resources that are generated become associated with an Account rather than a particular User. For example, it is not possible to look at an EC2 instance and determine who launched the instance.
This information is, however, available in AWS CloudTrail, but that is more of an audit log — it does not provide the user information back to the service.
So, I suspect that a stack is not provided with information about the User that launched it.
There is one way of doing it, but it isn't pretty and there is a caveat.
You can run a bash script like the below on an ec2 instance:
AWS_INSTANCE_ID=`curl -s http://169.254.169.254/latest/meta-data/instance-id`
USER_ID=`aws sts get-caller-identity --output text --query 'Arn'`
aws ec2 create-tags --resources "${AWS_INSTANCE_ID}" --tags 'Key=CreatedBy,Value="${USER_ID}"' --region eu-west-1
That will tag the current instance with the name of the user that run the CLI on that instance.
The caveat is however that the CLI needs to be run by the user - and not a role, so your users keys will have to be copied to the server - and then removed again at the end of the script.
Not ideal, but it gives you an option.
Looking for a quick way to pull my account number, I had originally thought of using aws iam get-account-authorization-details --max-items 1 but there are several issues with doing it this way. Is there a way to do this that might not cross account origins?
You can get the account number from the Secure Token Service subcommand get-caller-identity using the following:
aws sts get-caller-identity --query Account --output text
From my related answer for the AWS PowerShell CLI, your Account ID is a part of the Arn of resources that you create... and those that are automatically created for you. Some resources will also list you as an OwnerId.
The Default Security Group is automatically created for you in each region's default VPC as a reserved security group. From the documentation:
You can't delete a default security group. If you try to delete the EC2-Classic default security group, you'll get the following error: Client.InvalidGroup.Reserved: The security group 'default' is reserved. If you try to delete a VPC default security group, you'll get the following error: Client.CannotDelete: the specified group: "sg-51530134" name: "default" cannot be deleted by a user.
This makes it a reliable candidate for retrieving our account Id, as long as you are in EC2 classic or have a default VPC (*see edge cases if you don't).
Example:
aws ec2 describe-security-groups \
--group-names 'Default' \
--query 'SecurityGroups[0].OwnerId' \
--output text
This uses --query to filter the output down to the "owner ID" for the first result from this request, and then uses --output to output your account ID as plaintext:
123456781234
Edge cases:
(Thanks #kenchew) Note that if you've deleted your default VPC in a given region, this security group no longer exists and you should use one of these alternative solutions:
query STS get-caller-identity, per #Taras
use the first security group returned, per #Phillip
Further reading:
AWS EC2 Documentation: Default Security Groups
AWS CLI Documentation: aws ec2 describe-security-groups
Controlling Command Output from the AWS Command Line Interface
If you are running on a server that is running with an assumed role you can't call aws sts get-caller-identity. Also, with describe-security-groups you can't always use the --group-names filter (it doesn't work if you don't have a default VPC), so just pick the first security group. I've found this to be the most reliable regardless of what sort of authentication you use or what sort of VPC you have.
aws ec2 describe-security-groups --query 'SecurityGroups[0].OwnerId' --output text
My favorite method is to use aws iam get-user [--profile <profile>] since you only need IAM self service role for this to work.
The problem I am trying to solve is how to make my code running within an EC2 instance which is part of a load balanced AWS cluster aware of how many other EC2 instances are withing the same cluster/loadbalancer.
I have the following code which when given the name of a LoadBalancer can tell me how many EC2 instances are associated with that Loadbalancer.
DescribeLoadBalancersResult dlbr = loadBalancingClient.describeLoadBalancers();
List<LoadBalancerDescription> lbds = dlbr.getLoadBalancerDescriptions();
for( LoadBalancerDescription lbd : lbds )
{
if( lbd.getDNSName().equalsIgnoreCase("MyLoadBalancer"))
{
System.out.println(lbd.getDNSName() + " has " + lbd.getInstances().size() + " instances") ;
}
}
which works fine and prints out the loadbalancer name and number of instances is has associated with it.
However I want to see if I can get this info without having to provide the Loadbalancer name. In our setup an EC2 instance will only ever be associated with one Loadbalancer so is there any way to go back the way from EC2 instance to Loadbalancer?
I figure I can go down the route of getting all loadbalancers from All regions, iterating through them until I find the one that contains my EC2 instance but I figured there might be an easier way?
An interesting challenge -- I would have to wrangle with the code myself to think this through, but my gut first response would be to use the AWS CLI here, and to just invoke it from within your Java/C#.
You can make this call:
aws elb describe-load-balancers
And get all manner of information about any and all ELBs, and could simply --query filter that by the instance ID of the instance making the call anyway -- in order to find out what other friends the instance has joined to its same ELB. Just call the internal instance metadata to get that ID:
http://169.254.169.254/latest/meta-data/instance-id
Or another fun way to go would be to bootstrap your instance AMIs so that when they are spawned and joined to an ELB, they register themselves in a SimpleDB or DynamoDB table. We do this all the time as a way of keeping current inventories of websites, or software installed, etc. So this way you would have a list, which you could then keep trimmed by checking for "running" status.
EDIT - 4/13/2015
#MayoMan I have hadto make use of this as well in some current work -- to identify healthy instances attached to an ELB in an auto-scaling group and then act upon them. I've found 'jq' to be a really helpful command-line tool. You could also make these calls directly to an ELB, but here it's describing an ASG:
aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names <ASG Name> | jq -r .AutoScalingGroups[0].Instances[0].HealthStatus
Or to list the InstanceIds themselves:
aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names <ASG Name> | jq -r .AutoScalingGroups[0].Instances[0-3].InstanceId