I want to query all VPCs belonging to a particular region in my Go-based microservice.
https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeVpcs.html indicates that there exists no filter option by region or any other request parameter.
Golang SDK reference document:
https://docs.aws.amazon.com/sdk-for-go/api/service/ec2/#EC2.DescribeVpcs
Command line SDK reference document: https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-vpcs.html
Here's an example.
{
"Vpcs": [
{
"CidrBlock": "30.1.0.0/16",
"DhcpOptionsId": "dopt-19edf471",
"State": "available",
"VpcId": "vpc-0e9801d129EXAMPLE",
"OwnerId": "111122223333",
"InstanceTenancy": "default",
"CidrBlockAssociationSet": [
{
"AssociationId": "vpc-cidr-assoc-062c64cfafEXAMPLE",
"CidrBlock": "30.1.0.0/16",
"CidrBlockState": {
"State": "associated"
}
}
],
"IsDefault": false,
"Tags": [
{
"Key": "Name",
"Value": "Not Shared"
}
]
}
]
}
However, if I use the command
$aws ec2 describe-vpcs --region us-west-1
then I can query all vpcs in region us-west-1.
Question 1. Why is the --region option not mentioned in the CLI SDK document?
Question 2. How can I incorporate the same in DescribeVpcsInput while using GO SDK?
The --region flag on the CLI is not a filter, it is a required setting that tells the AWS CLI what region to connect to. The ec2 describe-vpcs command is always limited to a single region (most AWS commands are).
You would configure your AWS SDK client with the region you want it to connect to as well. See "Specifying the AWS Region" here.
Related
I have private snapshots in one account (source) that I have shared with another account (target). I am able to see the snapshots themselves from the target account, but the tags are not available, neither on the console nor via the cli. This makes it impossible to filter for a desired snapshot from the target account. For background, the user in the target account has the following policy in effect:
"Effect": "Allow",
"Action": "ec2:*",
"Resource": "*"
Here's an example of what I'm seeing; from the source account:
$ aws --region us-east-2 ec2 describe-snapshots --snapshot-ids snap-XXXXX
{
"Snapshots": [
{
"Description": "snapshot for testing",
"VolumeSize": 50,
"Tags": [
{
"Value": "test-snapshot",
"Key": "Name"
}
],
"Encrypted": true,
"VolumeId": "vol-XXXXX",
"State": "completed",
"KmsKeyId": "arn:aws:kms:us-east-2:XXXXX:key/mrk-XXXXX",
"StartTime": "2022-04-19T18:29:36.069Z",
"Progress": "100%",
"OwnerId": "XXXXX",
"SnapshotId": "snap-XXXXX"
}
]
}
but from the target account
$ aws --region us-east-2 ec2 describe-snapshots --owner-ids 012345678900 --snapshot-ids snap-11111111111111111
{
"Snapshots": [
{
"Description": "snapshot for testing",
"VolumeSize": 50,
"Encrypted": true,
"VolumeId": "vol-22222222222222222",
"State": "completed",
"KmsKeyId": "arn:aws:kms:us-east-2:012345678900:key/mrk-00000000000000000000000000000000",
"StartTime": "2022-04-19T18:29:36.069Z",
"Progress": "100%",
"OwnerId": "012345678900",
"SnapshotId": "snap-11111111111111111"
}
]
}
Any ideas on what's going on here?
Cheers!
From https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html#tag-restrictions
When you tag public or shared resources, the tags you assign are
available only to your AWS account; no other AWS account will have
access to those tags. For tag-based access control to shared
resources, each AWS account must assign its own set of tags to control
access to the resource.
I'm a beginner in using AWS.
I just want to stop and start several EC2 instances automatically and periodically(Not reboot).
Is there any recommended way to do this?
Amazon recently (Feb 2018) released the EC2 instance scheduler tool:
The AWS Instance Scheduler is a simple AWS-provided solution that
enables customers to easily configure custom start and stop schedules
for their Amazon Elastic Compute Cloud (Amazon EC2) and Amazon
Relational Database Service (Amazon RDS) instances. The solution is
easy to deploy and can help reduce operational costs for both
development and production environments. Customers who use this
solution to run instances during regular business hours can save up to
70% compared to running those instances 24 hours a day.
I had this up and running in my account in 15 minutes; very simple to use, and practically free.
https://aws.amazon.com/answers/infrastructure-management/instance-scheduler/
AWS has a good doc explaining how you can achieve this using Lambda and Cloudwatch events. You can refer it - https://aws.amazon.com/premiumsupport/knowledge-center/start-stop-lambda-cloudwatch/
This solution can be modified to get the EC2 list dynamically, or operate on a set of instances which can be identified based on a certain tag.
Yes, you can do that using AWS Lambda. You can select the trigger in Cloudwatch which runs on Cron expressions on UTC.
Here is a related link https://aws.amazon.com/premiumsupport/knowledge-center/start-stop-lambda-cloudwatch/
Another alternative is to use awscli which is available from pip, apt-get, yum or brew, and then running aws configure with your credentials from IAM and executing the following bash script, to stop an EC2 that has been tagged with Name: Appname and Value: Appname Prod. You can use awscli to tag your instances or tag it manually from the AWS console. aws ec2 stop-instances will stop the instance and jq is used to filter the json query and fetch the correct instance id using the tags from aws ec2 describe-instances.
To verify that aws configure was successful and returns json output run aws ec2 describe-instances and your running instance id should be there in the output. Here is a sample output
{
"Reservations": [
{
"Instances": [
{
"Monitoring": {
"State": "disabled"
},
"PublicDnsName": "ec2-xxx.ap-south-1.compute.amazonaws.com",
"State": {
"Code": xx,
"Name": "running"
},
"EbsOptimized": false,
"LaunchTime": "20xx-xx-xxTxx:16:xx.000Z",
"PublicIpAddress": "xx.127.24.xxx",
"PrivateIpAddress": "xxx.31.3.xxx",
"ProductCodes": [],
"VpcId": "vpc-aaxxxxx",
"StateTransitionReason": "",
"InstanceId": "i-xxxxxxxx",
"ImageId": "ami-xxxxxxx",
"PrivateDnsName": "ip-xxxx.ap-south-1.compute.internal",
"KeyName": "node",
"SecurityGroups": [
{
"GroupName": "xxxxxx",
"GroupId": "sg-xxxx"
}
],
"ClientToken": "",
"SubnetId": "subnet-xxxx",
"InstanceType": "t2.xxxxx",
"NetworkInterfaces": [
{
"Status": "in-use",
"MacAddress": "0x:xx:xx:xx:xx:xx",
"SourceDestCheck": true,
"VpcId": "vpc-xxxxxx",
"Description": "",
"NetworkInterfaceId": "eni-xxxx",
"PrivateIpAddresses": [
{
"PrivateDnsName": "ip-xx.ap-south-1.compute.internal",
"PrivateIpAddress": "xx.31.3.xxx",
"Primary": true,
"Association": {
"PublicIp": "xx.127.24.xxx",
"PublicDnsName": "ec2-xx.ap-south-1.compute.amazonaws.com",
"IpOwnerId": "xxxxx"
}
}
],
"PrivateDnsName": "ip-xxx-31-3-xxx.ap-south-1.compute.internal",
"Attachment": {
"Status": "attached",
"DeviceIndex": 0,
"DeleteOnTermination": true,
"AttachmentId": "xxx",
"AttachTime": "20xx-xx-30Txx:16:xx.000Z"
},
"Groups": [
{
"GroupName": "xxxx",
"GroupId": "sg-xxxxx"
}
],
"Ipv6Addresses": [],
"OwnerId": "xxxx",
"PrivateIpAddress": "xx.xx.xx.xxx",
"SubnetId": "subnet-xx",
"Association": {
"PublicIp": "xx.xx.xx.xxx",
"PublicDnsName": "ec2-xx.ap-south-1.compute.amazonaws.com",
"IpOwnerId": "xxxx"
}
}
],
"SourceDestCheck": true,
"Placement": {
"Tenancy": "default",
"GroupName": "",
"AvailabilityZone": "xx"
},
"Hypervisor": "xxx",
"BlockDeviceMappings": [
{
"DeviceName": "/dev/xxx",
"Ebs": {
"Status": "attached",
"DeleteOnTermination": true,
"VolumeId": "vol-xxx",
"AttachTime": "20xxx-xx-xxTxx:16:xx.000Z"
}
}
],
"Architecture": "x86_64",
"RootDeviceType": "ebs",
"RootDeviceName": "/dev/xxx",
"VirtualizationType": "xxx",
"Tags": [
{
"Value": "xxxx centxx",
"Key": "Name"
}
],
"AmiLaunchIndex": 0
}
],
"ReservationId": "r-xxxx",
"Groups": [],
"OwnerId": "xxxxx"
}
]
}
The following bash script is stop-ec2.sh in /home/centos/cron-scripts/
(instance=$(aws ec2 describe-instances | jq '.Reservations[].Instances | select(.[].Tags[].Value | startswith("Appname Prod") ) | select(.[].Tags[].Key == "Appname") | {InstanceId: .[].InstanceId, PublicDnsName: .[].PublicDnsName, State: .[].State, LaunchTime: .[].LaunchTime, Tags: .[].Tags} | [.]' | jq -r .[].InstanceId) && aws ec2 stop-instances --instance-ids ${instance} )
Run the file using sh /home/centos/cron-scripts/stop-ec2.sh and verify that the EC2 instance gets stopped. To debug run aws ec2 describe-instances | jq '.Reservations[].Instances | select(.[].Tags[].Value | startswith("Appname Prod") ) | select(.[].Tags[].Key == "Appname") | {InstanceId: .[].InstanceId, PublicDnsName: .[].PublicDnsName, State: .[].State, LaunchTime: .[].LaunchTime, Tags: .[].Tags} | [.]' | jq -r .[].InstanceId and see that it returns the correct instance ID which has been tagged.
Then in crontab -e the following line can be added
30 14 * * * sh /home/centos/cron-scripts/stop-ec2.sh >> /tmp/stop
which will log the output to /tmp/stop. The 30 14 * * * is the UTC cron expression that you can check in https://crontab.guru/
Lambda script to stop instance:
import json
import boto3
# Enter the region your instances are in. Include only the region without specifying Availability Zone; e.g., 'us-east-1'
region = 'us-east-1'
def lambda_handler(event, context):
ec2 = boto3.client('ec2', region_name=region)
filter = [{'Name': 'tag:Name', 'Values': ['****-env']}] //give instance name here in place of ****-env
instances = ec2.describe_instances(Filters=filter)
#ec2.stop_instances(InstanceIds=instances)
stop_instance = instances.get('Reservations')[0].get('Instances')[0].get('InstanceId')
stop_instances = []
stop_instances.append(stop_instance)
ec2.stop_instances(InstanceIds=stop_instances)
Lambda script to start instance:
import json
import boto3
# Enter the region your instances are in. Include only the region without specifying Availability Zone; e.g., 'us-east-1'
region = 'us-east-1'
def lambda_handler(event, context):
ec2 = boto3.client('ec2', region_name=region)
filter = [{'Name': 'tag:Name', 'Values': ['****-env']}]
instances = ec2.describe_instances(Filters=filter)
#ec2.stop_instances(InstanceIds=instances)
start_instance = instances.get('Reservations')[0].get('Instances')[0].get('InstanceId')
start_instances = []
start_instances.append(start_instance)
ec2.start_instances(InstanceIds=start_instances)
ASG scheduler is the best and easiest option to mange the EC2 instance if you are using ASG. If not using ASG, then you can use either AWS instance scheduler CF solution or lambda with cloudwatch Cron event.
I have existing infrastructure in us-east-1 region which needed to be cloned exactly to us-east-2 region. Used AWS CloudFormer to generate the JSON template from existing us-east-1 region, replaced all the us-east-1 with us-east-2 and started creating the stack but getting errors saying "Resource creation cancelled", specifically for all the EC2 instances
A snapshot of the template (only EC2 instance):
"instancei071dd59b": {
"Type": "AWS::EC2::Instance",
"Properties": {
"DisableApiTermination": "false",
"InstanceInitiatedShutdownBehavior": "stop",
"ImageId": "ami-1a41b377",
"InstanceType": "t2.medium",
"KeyName": "MyServer",
"Monitoring": "false",
"Tags": [
{
"Key": "MyServer OS",
"Value": "Windows Server"
},
{
"Key": "Name",
"Value": "MyServer_WEB_TEST_2"
}
],
"Volumes": [
{
"Device": "xvdb",
"VolumeId": {
"Ref": "volumevol9124b841"
}
}
],
"NetworkInterfaces": [
{
"DeleteOnTermination": "true",
"DeviceIndex": 0,
"SubnetId": {
"Ref": "subnet24031c0f"
},
"PrivateIpAddresses": [
{
"PrivateIpAddress": "172.31.53.184",
"Primary": "true"
}
],
"GroupSet": [
{
"Ref": "sgMyServerWEB"
}
],
"AssociatePublicIpAddress": "true"
}
]
}
},
"volumevol9124b841": {
"Type": "AWS::EC2::Volume",
"Properties": {
"AvailabilityZone": "us-east-2b",
"Size": "30",
"SnapshotId": "snap-95288b92",
"VolumeType": "gp2"
}
}
Before going with cloudformation template you will need to make sure you have following things in place :
Move your instance AMI to us-east-2 region then replace the snapshot id and AMI id in your template
Create a security group replace the security group id in your template
Replace subnet ID in your CF template with the one in us-east-2 region
The reason you will have to do this is every resource on AWS has unique IDs which cannot be replicated, if you want to replicate same you will need different Ids for that you need to create seperate resources and use them in your template.
If your doing this for a single instance only then you might do it manually by exporting AMI to us-east-2 region.
For collecting AMI ID in different region, I'd recommend to use the image name instead the AMI ID as the key.
To build resources to be placed in different regions, definitely is better to use CloudFormation. In this case you can use the lambda cli2cloudformation (https://github.com/lucioveloso/cli2cloudformation).
Using it, you can get the AMI ID across all regions and whatever other information that you are able to get using CLI.
To collect the AMI ID, create a lambda with cli2cloudformation and inside your template, create a custom resource as bellow:
"imageIdNameBased": {
"Type": "Custom::cli2cfnLambda",
"Properties": {
"ServiceToken": "arn:aws:lambda:eu-west-1:123456789012:function:cli2cfnLambda",
"CliCommandCreate": "ec2 describe-images --filters 'Name=name,Values=amzn-ami-hvm-2017.03.0.20170417-x86_64-gp2' --query 'Images[0]'"
}
}
In this case, I'm getting the AMI ID to the Image named 'amzn-ami-hvm-2017.03.0.20170417-x86_64-gp2'. You can change to your image name.
After that, you can retrieve it in any point of your CloudFormation stack.
"Fn::GetAtt" : ["imageIdNameBased", "ImageId"]
I'm scripting some stuff with aws ec2 describe-addresses, but, for some reason, some instances won't be returned by it.
Example:
$ aws ec2 describe-addresses --filter=Name=instance-id,Values=i-xxxxx
{
"Addresses": []
}
The given instanceId is valid and has addresses, but it just won't be shown by aws cli.
However, for another instances it seems to work just fine:
$ aws ec2 describe-addresses --filter='Name=instance-id,Values=i-yyyyyy'
{
"Addresses": [
{
"PrivateIpAddress": "X.X.X.X",
"InstanceId": "i-yyyyyy",
"NetworkInterfaceOwnerId": "XXXXXXXXXX",
"Domain": "vpc",
"AllocationId": "eipalloc-xxxxxx",
"PublicIp": "Y.Y.Y.Y",
"NetworkInterfaceId": "eni-xxxxxx",
"AssociationId": "eipassoc-xxxxx"
}
]
}
The keys I'm using have EC2FullAccess policy, so, it doesn't seem to be related to security...
What am I doing wrong? Any tips? Is there any limitations of aws cli that I'm not aware of?
I think the reason is you are not listing the instances, actually listing the Elastic IP Addresses. Probably the missing ones are the ones without Elastic IPs.
So for example the command without the filter will list the EIP list
aws ec2 describe-addresses
Output:
{
"Addresses": [
{
"InstanceId": null,
"PublicIp": "198.51.100.0",
"Domain": "standard"
},
{
"PublicIp": "203.0.113.0",
"Domain": "vpc",
"AllocationId": "eipalloc-64d5890a"
}
]
}
Source: http://docs.aws.amazon.com/cli/latest/reference/ec2/describe-addresses.html
I have a cloudformation script that attempts to create a VPC, with one subnet per AZ.
When I run:
aws ec2 describe-availablity-zones
I get 4 zones returned:
"AvailabilityZones": [
{
"State": "available",
"RegionName": "us-east-1",
"Messages": [],
"ZoneName": "us-east-1a"
},
{
"State": "available",
"RegionName": "us-east-1",
"Messages": [],
"ZoneName": "us-east-1b"
},
{
"State": "available",
"RegionName": "us-east-1",
"Messages": [],
"ZoneName": "us-east-1c"
},
{
"State": "available",
"RegionName": "us-east-1",
"Messages": [],
"ZoneName": "us-east-1d"
}
]
However, when I try to create my stack, I get an error:
"ResourceStatusReason": "Value (us-east-1a) for parameter availabilityZone
is invalid. Subnets can currently only be created in the following
availability zones: us-east-1c, us-east-1b, us-east-1d.",
I am specifying the AZ with
"AvailabilityZone" : {
"Fn::Select" : [ "0", { "Fn::GetAZs" : "" } ]
},
Is there a way to check to see if the AZ is really available for the creation of a subnet?
This may not be helpful for the CLI Approach or your exact scenario - but with AWS Management Console this works smooth.
With the recent updates with the CloudFormation Parameters, you would be able pin-point the AZs pertaining to the specified AZs.
This would be much convenient during the DR / DR Drills and making the CFN template Region Independent.
"Parameters": {
"SubnetAZ": {
"Description": "Availability Zone of the Subnet",
"Type": "AWS::EC2::AvailabilityZone::Name"
}
}
More Information About the CloudFormation Parameters
Unfortunately I had the same problem. There is no method in CloudFormation to do this and the zones can be different per AWS account. This is a limitation of VPC infrastructure and it is likely not going to change. Your only option will be to hardcode the zones that you have found in your CloudFOrmation template instead of Fn::Select, for example:
"AvailabilityZone" : "us-east-1b"
Alternatively if you leave AvailabilityZone blank, the default behavior would be AWS will automatically pick one for you.
Fn::GetAZs will provide the available and usable availability zones as long as you have a default vpc with a subnet in each AZ! which by the way all new aws accounts have as long as you don't delete them manually.
I get around the limitation by avoiding json as the medium of expression. I use troposphere to compose my cloudformation templates.(https://github.com/cloudtools/troposphere)
You would however would have to deploy some sort of tooling around the deployment of cloudformation templates to individual regions.