Am trying to copy the custom built ami using packer from one AWS account to another AWS account; however, i was able to copy ami across regions within one account.
"builders": [{
"account_id": "12345678910",
"s3_bucket": "xyz/xqas/asd",
"x509_cert_path": "/Users/txyz/packer/certificate.pem",
"x509_key_path": "/Users/txyz/packer/private-key.pem",
"type": "amazon-instance",
"access_key": "{{useraccess_key}}",
"secret_key": "{{usersecret_key}}",
"region": "us-east-1",
"source_ami": "ami-452bd728",
"instance_type": "r3.xlarge",
"ssh_username": "ubuntu",
"ami_name": "packer-test-hvm {{timestamp}}",
"ami_virtualization_type": "hvm",
"force_deregister": true,
"ami_regions": ["us-east-1", "us-west-2"]
}],
Use AWS ClI and run the following command in the AMI that needs to be transferred
ec2-modify-image-attribute ami-2bb65342 -l -a 111122223333
For other options that are available please refer this guide
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/sharingamis-explicit.html
The following awscli command will add launch permissions to a specified account id. This achieves a similar effect although does not technically copy the ami over.
aws ec2 modify-image-attribute --image-id <image id> --launch-permission "Add=[{UserId=<account-id>}]"
Using Packer this is the way you can share the AMIs from one account to the other.
Always refer the Packer docs:-https://www.packer.io/docs/builders/amazon/ebs here you can find all the information.
in the "ami_users" section you can mention the AWS accounts you need to share with.
"ami_users": ["{{user `REMOTE_AWS_ACCOUNT_ID`}}"]
full code
"builders": [
{
"type": "amazon-ebs",
"access_key": "{{ user `aws_access_key` }}",
"secret_key": "{{ user `aws_secret_key` }}",
"region": "{{ user `region` }}",
"launch_block_device_mappings" : [
{
"device_name": "/dev/sda1",
"volume_size": 60
}
],
"instance_type": "t2.large",
"ami_users": "{{ user `REMOTE_AWS_ACCOUNT_ID` }}",
"source_ami": "{{ user `source_ami` }}",
"ami_name": "xyz-ami",
"user_data_file": "./bootstrap_win.txt",
"communicator": "winrm",
"winrm_username": "Administrator",
"winrm_password": "XXXXXXXXX",
"tags": [{"Name":"testing","release":"packer"}],
"ami_regions": [
"ap-southeast-2",
"us-east-2"
]
}
],
Let's say you want to move the AMI from Account A to Account B, then you can use AWS CLI to do it.
Assume you already set the 2 AWS account credential.
# cat ~/.aws/credentials
[account_a]
aws_access_key_id = aaaaaaaaaaaaaaaaaaa
aws_secret_access_key = yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
[account_b]
aws_access_key_id = bbbbbbbbbbbbbbbbbb
aws_secret_access_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Then using the account_a run the command below to build an image (AMI) from a running instance: (you can skip this step if you already have an AMI)
# aws ec2 create-image --profile account_a --region $region --instance-id i-0aaaaaaaaaaaaa1 --name "image_name_here" --no-reboot
Share the AMI from account_a to account_b
# aws ec2 --profile account_a --region $region modify-image-attribute --image-id ami-aaaaaaaaaaa --launch-permission "Add=[{UserId=$account_b_id}]"
Use account_b to see if you get the shared AMI from account_a
# aws ec2 describe-images --profile account_b --region $region --executable-users self
The output should show if you get the shared AMI, or you also can go to AWS EC2 console, click Images --> AMIs, then change the Owned by me to Prviate images, you should be able to see the shared image as well.
extra:
if you want to launch an instance from the shared AMI on account_b
# aws ec2 run-instances --profile account_b --region $region --image-id ami-aaaaaaaaaaaaa --instance-type t2.micro --key-name $key_pair_for_access_ec2
Don't forget to edit the security group inbound rule for the SSH port open
Related
At my AWS account, I have few VPC. I'm trying to find a way to list all resources that located under a certain VPC.
Thanks!!
You can do it in three ways: AWS CLI, AWS console or code.
AWS CLI
You can use AWS CLI to list all ENIs associated with the VPC and prettify the output using the --query parameter to get a resource list with the desired fields (AZ, instance-id, etc.).
aws ec2 describe-network-interfaces --filters Name=vpc-id,Values=<vpc-id> --query 'NetworkInterfaces[*].[AvailabilityZone, OwnerId, Attachment.InstanceId, PrivateIpAddresses[*].Association.PublicIp]'
aws ec2 describe-network-interfaces --filters Name=vpc-id,Values=<vpc-id> --query 'NetworkInterfaces[*].[RequesterId,Description]'
A sample of the raw output (only one instance on the VPC):
"NetworkInterfaces": [
{
"Association": {
"IpOwnerId": "amazon",
"PublicDnsName": "ec2-54-196-57-169.compute-1.amazonaws.com",
"PublicIp": "54.196.57.169"
},
"Attachment": {
"AttachTime": "2020-08-24T10:59:16+00:00",
"AttachmentId": "eni-attach-047e562690aabbffd",
"DeleteOnTermination": true,
"DeviceIndex": 0,
"InstanceId": "i-0fe495a6c17bd0f82",
"InstanceOwnerId": "570398916848",
"Status": "attached"
},
"AvailabilityZone": "us-east-1d",
"Description": "",
"Groups": [
{
"GroupName": "launch-wizard-1",
"GroupId": "sg-0aa7d8257bb487e1b"
}
],
"InterfaceType": "interface",
"Ipv6Addresses": [],
"MacAddress": "0e:58:38:33:9a:31",
"NetworkInterfaceId": "eni-0b20855178d276783",
"OwnerId": "570398916848",
"PrivateDnsName": "ip-172-31-34-30.ec2.internal",
"PrivateIpAddress": "172.31.34.30",
"PrivateIpAddresses": [
{
"Association": {
"IpOwnerId": "amazon",
"PublicDnsName": "ec2-54-196-57-169.compute-1.amazonaws.com",
"PublicIp": "54.196.57.169"
},
"Primary": true,
"PrivateDnsName": "ip-172-31-34-30.ec2.internal",
"PrivateIpAddress": "172.31.34.30"
}
],
"RequesterManaged": false,
"SourceDestCheck": true,
"Status": "in-use",
"SubnetId": "subnet-e2bc5fbd",
"TagSet": [],
"VpcId": "vpc-6ad2e110"
}
]
And now filtered:
For the first --query
[
"us-east-1d",
"57039816848",
"i-0fe495a6c17bd0f82",
[
"44.196.57.169"
]
]
And for the second --query (another VPC):
[
"amazon-elasticache",
"ElastiCache alon-001"
],
[
"amazon-elasticache",
"ElastiCache alon-002"
],
[
"975289786086",
"arn:aws:ecs:us-east-2:57039916848:attachment/22a90802-fae7-4afb-9a7e-43e6f4be8ca4"
],
[
"074689309192",
"Interface for NAT Gateway nat-069344579d8bda20"
],
[
"amazon-elb",
"ELB app/EC2Co-EcsEl-YX74WCWEGOK/0b6d7bc60b540b1"
],
[
"amazon-elb",
"ELB app/EC2Co-EcsEl-YX74WCWGGOK/0b6bd7c60b540b1"
],
[
"amazon-elasticache",
"ElastiCache alon-003"
]
AWS Console
You can do the same using the AWS console.
Under EC2->Network Interfaces, search for the desired vpc-id in the search bar.
Code
Using a python script called vpc-inside.py you can describe all of your VPC resources.
usage: vpc-inside.py [-h] -v VPC [-r REGION] [-p PROFILE]
optional arguments:
-h, --help show this help message and exit
-v VPC, --vpc VPC The VPC to annihilate
-r REGION, --region REGION AWS region that the VPC resides in
-p PROFILE, --profile PROFILE AWS profile
And the output will look like this:
EKSs in VPC vpc-07ef7f777429cfd82:
Omikron
--------------------------------------------
ASGs in VPC vpc-07ef7f777429cfd82:
eks-pooks-9ebf225b-70a9-a026-034f-c7431df9b7ba resides in vpc-07ef7f777429cfd82
eks-pooks-9ebf225b-70a9-a026-034f-c7431df9b7ba
--------------------------------------------
RDSs in VPC vpc-07ef7f777429cfd82:
--------------------------------------------
EC2s in VPC vpc-07ef7f777429cfd82:
i-0c63874d77ea2ba78
i-043740f224015e69e
--------------------------------------------
Lambdas in VPC vpc-07ef7f777429cfd82:
--------------------------------------------
Classic ELBs in VPC vpc-07ef7f777429cfd82:
--------------------------------------------
ELBs V2 in VPC vpc-07ef7f777429cfd82:
--------------------------------------------
NAT GWs in VPC vpc-07ef7f777429cfd82:
--------------------------------------------
VPC EndPoints in VPC vpc-07ef7f777429cfd82:
--------------------------------------------
IGWs in VPC vpc-07ef7f777429cfd82:
--------------------------------------------
ENIs in VPC vpc-07ef7f777429cfd82:
eni-079231232dc136305
eni-05ff227eca8341a08
eni-0c01b2871887ac3f7
eni-00e11d4f9590161b4
--------------------------------------------
Security Groups in VPC vpc-07ef7f777429cfd82:
sg-0b4554a65e1560745
sg-0f93574d6b180b263
--------------------------------------------
Routing tables in VPC vpc-07ef7f777429cfd82:
rtb-0694bdbdd696b2bed
rtb-072ec82a18d8a04ba
--------------------------------------------
ACLs in VPC vpc-07ef7f777429cfd82:
acl-0c0087eabf9335940
--------------------------------------------
Subnets in VPC vpc-07ef7f777429cfd82:
subnet-0b8cc1132727e5b5d
subnet-0e47ee92a9ca80280
subnet-0c25990d9a138616b
--------------------------------------------
You can try in AWS Config > Advanced queries and run below query :
All resources:
SELECT
resourceId,
resourceName,
resourceType
Resources directly associated to VPC:
SELECT
resourceId,
resourceName,
resourceType
WHERE
relationships.resourceId = 'vpc-02368dae78f1387e5'
Query can be further enhanced, see some example of preconfigured query.
VPCs mostly contain EC2 instances, RDS instances, Load Balancers and Lambda functions. Plus, things that use EC2 underneath, like Elasticache. These are the types of resources that connect into a VPC.
Some people suggest using the Tag Editor to find resources: Is there a way to list all resources in AWS.
I also like aws inventory, which simply runs in your browser and does a great job of showing resources. Just give it an Access Key and Secret Key to run.
There's no built in service to easily do this.
The best hope you'd have of find all resources is programatically looping over resources that support:
SubnetId
VpcId
is there a way to list/describe all the tags in all the services in an aws environment with one command?
E.g. If I wanted to list the tags in ec2 instances, I can do this aws ec2 describe-tags and can see all the tags. Now if I wanted to view, say, eks tags I'd have to do something like aws eks list-tags-for-resource ... or something similar. So my question is if there is a "magic" where I could do aws * describe-tags and see all the tags in all services, instead of going one by one for each service like so: aws <service name> <command to describe/list tags>
You can use : aws resourcegroupstaggingapi get-resources
From the docs:
Returns all the tagged or previously tagged resources that are located
in the specified Region for the AWS account.
Usage:
aws resourcegroupstaggingapi get-resources
Output:
{
"ResourceTagMappingList": [
{
"ResourceARN": " arn:aws:inspector:us-west-2:123456789012:target/0-nvgVhaxX/template/0-7sbz2Kz0",
"Tags": [
{
"Key": "Environment",
"Value": "Production"
}
]
}
]
}
after the VPC is created, how I can grep only 1 VPC-ID from specific VPC by aws ec2 describe-vpcs, so that VPC ID can be passed inside the script for the next step, I know I can see it manually from that command or from AWS console,
for example:
aws ec2 describe-vpcs --vpc-ids |grep VpcId
"VpcId": "vpc-00a0338c2f671a77c",
"VpcId": "vpc-0b3697513d5987516",
"VpcId": "vpc-061e25f5f78877798",
it gives me all of them, or:
aws ec2 describe-vpcs --vpc-ids |grep -i ansible
"Value": "ANSIBLE_VPC",
but I need only to get the VPC-ID for that specific VPC from command.
If you just issued a create-vpc command, then the VPC ID of that VPC would have been returned in response to that command:
Output:
{
"Vpc": {
"CidrBlock": "10.0.0.0/16",
"DhcpOptionsId": "dopt-5EXAMPLE",
"State": "pending",
"VpcId": "vpc-0a60eb65b4EXAMPLE", <-- This is the VPC ID
"OwnerId": "123456789012",
"InstanceTenancy": "default",
"Ipv6CidrBlockAssociationSet": [],
"CidrBlockAssociationSet": [
{
"AssociationId": "vpc-cidr-assoc-07501b79ecEXAMPLE",
"CidrBlock": "10.0.0.0/16",
"CidrBlockState": {
"State": "associated"
}
}
],
"IsDefault": false,
"Tags": []
}
}
Thus, you could create the VPC and store its ID like this:
$ ID=`aws ec2 create-vpc --cidr-block 10.0.0.0/16 --query Vpc.VpcId --output text`
$ echo $ID
vpc-0fb4d08f9d6501e94
If, instead, you are seeking the VPC ID for a VPC given its Name tag, you could use:
$ ID=`aws ec2 describe-vpcs --filter Name=tag:Name,Values=ANSIBLE_VPC --query Vpcs[].VpcId --output text`
$ echo $ID
vpc-0fb4d08f9d6501e94
You can use just the aws cli for this, with filters and query:
aws ec2 describe-vpcs --filters Name=tag:Name,Values=ANSIBLE_VPC --query "Vpcs[].VpcId" --output text"
Or you can use a mix of the --filters command with grep to accomplish your task:
aws ec2 describe-vpcs --filters Name=tag:Name,Values=ANSIBLE_VPC | grep VpcId | grep -oh "vpc-\w*"
I want to tag an interface when I create an instance with multiple interfaces with AWS-CLI.
For example
aws ec2 run-instances --image-id {$byolimageid} --count 1 --instance-type c5.18xlarge --key-name thomaskeypair --network-interfaces '[ { "DeviceIndex": 0, "Groups": ["sg-06ec2ed1714b4b548"], "SubnetId": "subnet-0c36ed5c120 21bbb3", "DeleteOnTermination": true}, {"DeviceIndex": 1, "Groups": ["sg-06ec2ed1714b4b548"], "SubnetId": "subnet-011f43adb412986e0", "DeleteOnTermination": true}]
In this case, I want to tag interface with deviceindex 0 with a key,value.
Can I do it within the aws ec2 run-instances command?
Can't find information on documentation.
I want to automate the running of a cluster and can use tags to get attributes of an EC2 instance like its instance-id.
The documentation on https://docs.aws.amazon.com/cli/latest/reference/emr/create-cluster.html states that
--tags (list)
A list of tags to associate with a cluster, which apply to each Amazon
EC2 instance in the cluster. Tags are key-value pairs that consist of
a required key string with a maximum of 128 characters, and an
optional value string with a maximum of 256 characters.
You can specify tags in key=value format or you can add a tag without
a value using only the key name, for example key . Use a space to
separate multiple tags.
So this applies tags to every EC2 instance including the master and slaves. How do I discern which instance is the master node?
Additional Info:
I am using the following command to get attributes from aws cli based on tags where you can replace the "Name" and "Prod" with your tags key-value pairs respectively.
aws ec2 describe-instances | jq '.Reservations[].Instances | select(.[].Tags[].Value | startswith("Prod") ) | select(.[].Tags[].Key == "Name") | {InstanceId: .[].InstanceId, PublicDnsName: .[].PublicDnsName, State: .[].State, LaunchTime: .[].LaunchTime, Tags: .[].Tags} | [.]' | jq .[].InstanceId
As you noted when you create an EMR cluster, the tags are the same for all nodes (Master, Slave, Task).
You will find that this process using the AWS CLI to be complicated. My recomendation is to review the examples below and then write a Python program to do this.
Process to add your own tags to the EC2 instances.
STEP 1: List your EMR Clusters:
aws emr list-clusters
This will output JSON:
{
"Clusters": [
{
"Id": "j-ABCDEFGHIJKLM",
"Name": "'MyCluster'",
"Status": {
"State": "WAITING",
"StateChangeReason": {
"Message": "Cluster ready after last step completed."
},
"Timeline": {
"CreationDateTime": 1536626095.303,
"ReadyDateTime": 1536626568.482
}
},
"NormalizedInstanceHours": 0
}
]
}
STEP 2: Make a note of the Cluster ID from the JSON:
"Id": "j-ABCDEFGHIJKLM",
STEP 3: Describe your EMR Cluster:
aws emr describe-cluster --cluster-id j-ABCDEFGHIJKLM
This will output JSON (I have truncated this output to just the MASTER section):
{
"Cluster": {
"Id": "j-ABCDEFGHIJKLM",
"Name": "'Test01'",
....
"InstanceGroups": [
{
"Id": "ig-2EHOYXFABCDEF",
"Name": "Master Instance Group",
"Market": "ON_DEMAND",
"InstanceGroupType": "MASTER",
"InstanceType": "m3.xlarge",
"RequestedInstanceCount": 1,
"RunningInstanceCount": 1,
"Status": {
"State": "RUNNING",
"StateChangeReason": {
"Message": ""
},
"Timeline": {
"CreationDateTime": 1536626095.316,
"ReadyDateTime": 1536626533.886
}
},
"Configurations": [],
"EbsBlockDevices": [],
"ShrinkPolicy": {}
},
....
]
}
}
STEP 4: InstanceGroups is an array. Find the entry where InstanceGroupType is MASTER. Make note of the Id.
"Id": "ig-2EHOYXFABCDEF",
STEP 5: List your cluster instances:
aws emr list-instances --cluster-id j-ABCDEFGHIJKLM
This will output JSON (I have truncated the output):
{
"Instances": [
....
{
"Id": "ci-31LGK4KIECHNY",
"Ec2InstanceId": "i-0524ec45912345678",
"PublicDnsName": "ec2-52-123-201-221.us-west-2.compute.amazonaws.com",
"PublicIpAddress": "52.123.201.221",
"PrivateDnsName": "ip-172-31-41-111.us-west-2.compute.internal",
"PrivateIpAddress": "172.31.41.111",
"Status": {
"State": "RUNNING",
"StateChangeReason": {},
"Timeline": {
"CreationDateTime": 1536626164.073,
"ReadyDateTime": 1536626533.886
}
},
"InstanceGroupId": "ig-2EHOYXFABCDEF",
"Market": "ON_DEMAND",
"InstanceType": "m3.xlarge",
"EbsVolumes": []
}
]
}
STEP 6: Find the matching InstanceGroupId ig-2EHOYXFABCDEF. This will give you the EC2 Instance ID for the MASTER: "Ec2InstanceId": "i-0524ec45912345678"
Step 7: Tag your EC2 instance:
aws ec2 create-tags --resources i-0524ec45912345678 --tags Key=EMR,Value=MASTER
The above steps might be simpler with CLI Filters and / or jq, but this should be enough information so that you know how to find and tag the EMR Master Instance.
Below can be used to directly get instance Id
aws emr list-instances --cluster-id ${aws_emr_cluster.cluster.id} --instance-
group-id ${aws_emr_cluster.cluster.master_instance_group.0.id} --query
'Instances[*].Ec2InstanceId' --output text
In an enviroinment where you does not have the aws cli, you can cat the following file:
cat /mnt/var/lib/info/job-flow.json
An example of the content is the following one:
{
"jobFlowId": "j-0000X0X0X00XX",
"jobFlowCreationInstant": 1579512208006,
"instanceCount": 2,
"masterInstanceId": "i-00x0xx0000xxx0x00",
"masterPrivateDnsName": "localhost",
"masterInstanceType": "m5.xlarge",
"slaveInstanceType": "m5.xlarge",
"hadoopVersion": "2.8.5",
"instanceGroups": [
{
"instanceGroupId": "ig-0XX00XX0X0XXX",
"instanceGroupName": "Master - 1",
"instanceRole": "Master",
"marketType": "OnDemand",
"instanceType": "m5.xlarge",
"requestedInstanceCount": 1
},
{
"instanceGroupId": "ig-000X0XXXXXXX",
"instanceGroupName": "Core - 2",
"instanceRole": "Core",
"marketType": "OnDemand",
"instanceType": "m5.xlarge",
"requestedInstanceCount": 1
}
]
NOTE: i've omitted the ID of the jobs using 0 where a number is expected and X where a ltter is expected.
You can do this programmatically in 3 lines of code, without having to copy-paste any of the specific information:
# get cluster id
CLUSTER_ID=$(aws emr list-clusters --active --query "Clusters[0].Id" --output text)
# get instance id
INSTANCE_ID=$(aws emr list-instances --cluster-id $CLUSTER_ID --instance-group-types MASTER --query "Instances[0].Ec2InstanceId" --output text)
# tag
aws ec2 create-tags --resources $INSTANCE_ID --tags Key=EMR,Value=MASTER
Below example is for Instance Fleet, it saves Cluster ID, Instance Fleet ID and Master IP as environment variables.
Replace cluster name "My-Cluster" to the actual cluster name.
export CLUSTER_ID=$(aws emr list-clusters --active --query 'Clusters[?Name==`My-Cluster`].Id' --output text)
export INSTANCE_FLEET=$(aws emr describe-cluster --cluster-id $CLUSTER_ID | jq -r '.[].InstanceFleets | .[] | select(.InstanceFleetType=="MASTER") | .Id')
export PRIVATE_IP=aws emr list-instances --cluster-id $CLUSTER_ID --instance-fleet-id $INSTANCE_FLEET --query 'Instances[*].PrivateIpAddress' --output text
"Cleanest" way:
aws emr list-clusters --active
Search for Master cluster ID (j-xxxxxxxxxxx), then
aws emr list-instances --region {your_region} --instance-group-types MASTER --cluster-id j-xxxxxxxxxxxxx
Immediately filters out the master instance(s) with it's information using --instance-group-types MASTER flag.
For tagging refers to the other answers with aws {resource} create-tags and --tag flag.