The problem I am trying to solve is how to make my code running within an EC2 instance which is part of a load balanced AWS cluster aware of how many other EC2 instances are withing the same cluster/loadbalancer.
I have the following code which when given the name of a LoadBalancer can tell me how many EC2 instances are associated with that Loadbalancer.
DescribeLoadBalancersResult dlbr = loadBalancingClient.describeLoadBalancers();
List<LoadBalancerDescription> lbds = dlbr.getLoadBalancerDescriptions();
for( LoadBalancerDescription lbd : lbds )
{
if( lbd.getDNSName().equalsIgnoreCase("MyLoadBalancer"))
{
System.out.println(lbd.getDNSName() + " has " + lbd.getInstances().size() + " instances") ;
}
}
which works fine and prints out the loadbalancer name and number of instances is has associated with it.
However I want to see if I can get this info without having to provide the Loadbalancer name. In our setup an EC2 instance will only ever be associated with one Loadbalancer so is there any way to go back the way from EC2 instance to Loadbalancer?
I figure I can go down the route of getting all loadbalancers from All regions, iterating through them until I find the one that contains my EC2 instance but I figured there might be an easier way?
An interesting challenge -- I would have to wrangle with the code myself to think this through, but my gut first response would be to use the AWS CLI here, and to just invoke it from within your Java/C#.
You can make this call:
aws elb describe-load-balancers
And get all manner of information about any and all ELBs, and could simply --query filter that by the instance ID of the instance making the call anyway -- in order to find out what other friends the instance has joined to its same ELB. Just call the internal instance metadata to get that ID:
http://169.254.169.254/latest/meta-data/instance-id
Or another fun way to go would be to bootstrap your instance AMIs so that when they are spawned and joined to an ELB, they register themselves in a SimpleDB or DynamoDB table. We do this all the time as a way of keeping current inventories of websites, or software installed, etc. So this way you would have a list, which you could then keep trimmed by checking for "running" status.
EDIT - 4/13/2015
#MayoMan I have hadto make use of this as well in some current work -- to identify healthy instances attached to an ELB in an auto-scaling group and then act upon them. I've found 'jq' to be a really helpful command-line tool. You could also make these calls directly to an ELB, but here it's describing an ASG:
aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names <ASG Name> | jq -r .AutoScalingGroups[0].Instances[0].HealthStatus
Or to list the InstanceIds themselves:
aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names <ASG Name> | jq -r .AutoScalingGroups[0].Instances[0-3].InstanceId
Related
For a AWS EC2 Instance, Is it possible to unassign or remove a security group from a instance using the cmd line. It is easy to do via the UI, but I can't seem to do it easily via the API. I am restricted to using AWS cli version 1 for now. https://docs.aws.amazon.com/cli/latest/reference/ec2/index.html
I have looked at using the modify instance attribute but it seems like a really round about way to unassign a security group from a instance. I have to reassign all other groups except for the security group I do not want assigned
aws ec2 modify-instance-attribute --instance-id [Instance ID] --groups [groups]
Deleting the security group, will not work as it is assigned to a instance.
aws ec2 delete-security-group --group-id [grou id]
I get the correct error
<br>
An error occurred (DependencyViolation) when calling the DeleteSecurityGroup operation
Is there a easier way to remove or unassign a security group from a instance other than using the modify-instance-attribute using the cmd line?
The AWS CLI command to attach a security group to running EC2 instance is as below.
aws ec2 modify-instance-attribute --instance-id i-12345 --groups sg-12345 sg-67890
But the above command will remove the currently attached security groups and attach the new one.
I have a use case where there are 100+ servers and I have to attach a new security group to all those servers without detaching the current security groups.
How can I achieve this using the AWS CLI?
The --groups does a complete replacement based on the arguments passed & there's no way to bypass this behaviour so you'll need to implement the logic of getting the existing security groups (SGs), appending the new SG on & then passing that as an input to --groups.
Confirmed by aws ec2 modify-instance-attribute documentation:
--groups (list)
[EC2-VPC] Replaces the security groups of the instance with the specified security groups. You must specify at least one security group, even if it’s just the default security group for the VPC. You must specify the security group ID, not the security group name.
(string)
This command should store all of the security groups for an instance with ID i-12345 in $securitygroups:
securitygroups=$(aws ec2 describe-instances --instance-ids i-12345 --query "Reservations[].Instances[].SecurityGroups[].GroupId[]" --output text)
The output of echo $securitygroups will look something similar to this:
sg-074bb9206bd7edaf2 sg-07cd92995b937cbd2 sg-05414d9cef32901be
Given that your new security group ID is sg-67890, execute the below command to append the new SG ID onto the list of security groups that we want to set (the space is important & needed):
securitygroups+=" sg-67890"
The output of echo $securitygroups should now have the new SG ID appended:
sg-074bb9206bd7edaf2 sg-07cd92995b937cbd2 sg-05414d9cef32901be sg-67890
Finally, pass $securitygroups to the --groups option of aws ec2 modify-instance-attribute.
This variable will contain the existing assigned SG IDs as well as the new SG ID to be assigned so it'll be an assigning the new SG without unassigning any current SGs:
aws ec2 modify-instance-attribute --instance-id i-067a3aae02b8239e6 --groups $securitygroups
Put this in a loop for however many instances you have, problem solved.
This is a strange one... If I click on the instance id, and then navigate to security, it tells me the instance has role X. Then I back out to view all instances, mark the checkbox for the instance in question, go to Actions -> Security -> Modify IAM Role, and it shows me a different role, role Y. I then try to set it to No IAM Role (or any various role), and I get this error:
"Multiple roles associated to instance
The selected instance has more than one IAM role associated. This usually occurs when the instance is in the process of replacing an existing instance profile association. "
I have no idea what to do because I didn't think an EC2 instance was supposed to be able to have two roles... nothing can assume two roles at once, anyway. So this feels like a bug... can anyone help me solve this?
I had the same issue and it seems that when replacing the instance profile it somehow stays in a state that's not completely associated.
Using the CLI we can see the status of the profile association:
aws ec2 describe-iam-instance-profile-associations
In my case the problematic profile was showing as "associating", while all the others show "associated".
Get the AssociationID for the problematic association and disassociate it with the command
aws ec2 disassociate-iam-instance-profile --association-id iip-assoc-xxxxxx
After that you should see the previous profile you had originally and everything should be consistent.
Hope it helps solving the problem.
I wanted to expound on Nelson Brito's answer since I found a way to return your instance to a normal state. I ran into this situation yesterday when helping a user, and I observed my instance with two profile associations -- one in a state of associating and one in a state of disassociating. The command to find this was:
aws ec2 describe-iam-instance-profile-associations --filters Name=instance-id,Values=i-xxxxxx
To fix the issue, I first removed the associating profile using the command:
aws ec2 disassociate-iam-instance-profile --association-id iip-assoc-xxxxxx
Next, I went to the console and detached the instance from all profiles (there is probably a CLI invocation, but I didn't figure it out). When done, you should have a clean instance:
aws ec2 describe-iam-instance-profile-associations --filters Name=instance-id,Values=i-xxxxxx
{
"IamInstanceProfileAssociations": []
}
Here's where we get to the root cause. When I re-assigned the role that was previously stuck in associating, well, it remained stuck in associating. The root cause of my problem was that the user had created the role without setting a trust relationship with EC2. The fix was two parts:
Update the instance trust relationship to add "Service": "ec2.amazonaws.com" as an allowed principal
Use the CLI to disassociate the role and add it again
tl;dr - If you ever run into this, ensure that the role you're trying to assign to your ec2 instance can be assumed by your ec2 instance.
We had the issue Multiple roles associated to instance after bringing an existing EC2 into CloudFormation. Nelson Brito's answer helped to resolve the issue.
To avoid the issue altogether when bringing an existing EC2 into CloudFormation, I now recommend
first disassociate the IAM role (for example via the AWS Console)
import the EC2 only, i.e. without InstanceProfile and Role in the import template
add InstanceProfile and Role to the template and update the stack.
I am trying to find out all EC2 instances in 10 different accounts which are running non-amazon AMI images. Following CLI command gives me the list of all AMI's:
aws ec2 describe-instances --output text --query 'Reservations[*].Instances[*].[ImageId]' | sort | uniq -c
I think I can modify this further to get all non-amazon AMI's but is there a way to run this across 10 different accounts in one call?
is there a way to run this across 10 different accounts in one call?
No, that's not possible. You need to write a loop that iterates over each account, calling ec2 describe-instances once for each account.
Here's a script that can find instances using AMIs where the Owner is not amazon:
import boto3
ec2_client = boto3.client('ec2', region_name='ap-southeast-2')
instances = ec2_client.describe_instances()
# Get a set of AMIs used on all the instances
images = set(i['ImageId'] for r in instances['Reservations'] for i in r['Instances'])
# Find which of these are owned by Amazon
amis = ec2_client.describe_images(ImageIds=list(images), Owners=['amazon'])
amazon_amis = [i['ImageId'] for i in amis['Images']]
# Which instances are not using Amazon images?
non_amazon_instances = [(i['InstanceId'], i['ImageId']) for r in instances['Reservations'] for i in r['Instances'] if i['ImageId'] not in amazon_amis]
for i in non_amazon_instances:
print(f"{i[0]} uses {i[1]}")
A few things to note:
Deprecated AMIs might not have accessible information, so might be marked a non-Amazon.
This script, as written, only works on one region. You could change it to loop through regions.
This script, as written, only works on one account. You would need a way to loop through credentials for other accounts.
Use AWS config
Create an agregator in root or delegated account(wait for the agregator to load)
Create query
SELECT
accountId,
resourceId,
configuration.keyName,
availabilityZone
WHERE
resourceType = 'AWS::EC2::Instance'
AND configuration.state.name = 'running'
more details
https://aws.amazon.com/blogs/mt/org-aggregator-delegated-admin/
I have multi-account AWS environment (set up using AWS Landing Zone) and I need to copy a specific security group to all the accounts. I do have a CFT written, but it's too much of a repetitive task to do this one by one.
The security group is in the central (shared-services) account, which has access to all the other accounts. It's better if there's a way to integrate this to Account Vending Machine (AVM) in order to avoid future tasks of exporting the SG to newly spawned accounts.
You should use CloudFormation Stacksets. StackSets is a feature of cloudformation in which you have a master account in which you create/update/remove the stackset, and you have children accounts. In the stackset, you configure your children aws accounts you want to deploy the CF template and the region as well.
From your comment, your master account is going to be the shared-services and the rest of your accounts, the children ones. You will need to deploy a couple of IAM roles to allow cross-account access, but after that, you will be able to deploy all your templates in up to 500 aws accounts automatically and update them as well.
More information here: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/what-is-cfnstacksets.html
You can export Security Group and other configuration with CloudFormation using CloudFormer, which creates a template from the existing account configuration. Check the steps in this guide https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-using-cloudformer.html It will upload the template on S3 and you can reuse it or some of its parts.
Since you are using AWS Landing Zone, you can add the security group to the aws_baseline templates, either as a new template or added to one of the existing files. When submitted, AWS Landing Zone uses Step Functions and AWS Stack Sets to deploy your SG to all existing and future accounts. If you choose to create the security group in a new file, you must add a reference to it in the manifest.yaml file (compare with how the other templates are referenced).
I was able to do this via the Account Vending Machine. But AGL's Stacksets should be a good alternative too.
To copy AWS Security Gp from one account of any region to other AWS account to any region is required lots of scripting(coding) in aws cli or boto3.
But one thing i done which is feasible to my usecase(Whitelist 14 IPs for HTTPS) is write a bash script
Here prior i create a blank SG on other AWS account(or u may use aws cli to create that too),
`
Region1=
SGFromCopy=
profilefromcopy=
Region2=
SGToCopy=
profiletocopy=
for IP in $(aws ec2 describe-security-groups --region $Region1 --group-id=$SGFromCopy --profile $profilefromcopy --query SecurityGroups[].IpPermissions[].IpRanges[].CidrIp --output text) ;
do
aws ec2 authorize-security-group-ingress --group-id=$SGToCopy --ip-permissions IpProtocol=TCP,FromPort=443,ToPort=443,IpRanges=[{CidrIp=$IP}] --profile $profiletocopy --region $Region2;
done ;
`
U may modify script if u have csv formated of SG and then just had iterated in while loop
BONUS
To get desire output you have to alter output in file or some where else
> aws ec2 describe-security-groups --region $Region1 --group-id $SGFromCopy --profile $profilefromcopy --query SecurityGroups[].IpPermissions[] --output text