Retrieving Id elasticbeanstalk (EBS) environment in the terminal? - amazon-web-services

How do I get the Id of my elasticbeanstalk (EBS) environment in the terminal?
This command returns an object in the terminal with some properties for the environment:
aws elasticbeanstalk describe-environments --environment-names my-env
Is it possible to get only the EnvironmentId from that object in the terminal?
{
"Environments": [
{
"ApplicationName": "xxxx-xxxx-xxxx-xxxxx",
"EnvironmentName": "my-env",
"VersionLabel": "Initial Version",
"Status": "Ready",
"EnvironmentArn": "arn:aws:elasticbeanstalk:eu-central-1:xxxxxxx:environment/xxxx-xxxxx-xxxx-xxxx/my-env",
"EnvironmentLinks": [],
"PlatformArn": "arn:aws:elasticbeanstalk:eu-central-1::platform/Multi-container Docker running on 64bit Amazon Linux/2.11.0",
"EndpointURL": "awseb-e-2-xxxxx-xxxxxx-xxxxx.eu-central-1.elb.amazonaws.com",
"SolutionStackName": "64bit Amazon Linux 2018.03 v2.11.0 running Multi-container Docker 18.03.1-ce (Generic)",
"EnvironmentId": "e-1234567",
"CNAME": "my-env.elasticbeanstalk.com",
"AbortableOperationInProgress": false,
"Tier": {
"Version": "1.0",
"Type": "Standard",
"Name": "WebServer"
},
"Health": "Green",
"DateUpdated": "2018-07-12T06:10:17.056Z",
"DateCreated": "2018-07-11T20:03:26.970Z"
}
]
}
In this case, the result that I'm expecting in my terminal to appear is e-1234567

If you want to use the AWS CLI for this, you would need to filter the output of aws elasticbeanstalk describe-environments --environment-names my-env using a tool such as grep. One possible (by no means optimal/concise) solution :
aws elasticbeanstalk describe-environments --environment-names my-env | grep EnvironmentId | grep -Eo "e-[A-Za-z0-9_]+"
The better solution is to use an AWS SDK such as boto3 (the Python AWS SDK).
import boto3
elasticbeanstalk = boto3.client(
'elasticbeanstalk',
region_name='us-west-2'
)
response = elasticbeanstalk.describe_environments(
EnvironmentNames=['my-env']
)
if response['Environments']:
print(response['Environments'][0]['EnvironmentId'])
AWS SDKs are available in other popular languages such as Go, Java, Ruby, JavaScript, PHP as well.

Related

awscli list tags in all services

is there a way to list/describe all the tags in all the services in an aws environment with one command?
E.g. If I wanted to list the tags in ec2 instances, I can do this aws ec2 describe-tags and can see all the tags. Now if I wanted to view, say, eks tags I'd have to do something like aws eks list-tags-for-resource ... or something similar. So my question is if there is a "magic" where I could do aws * describe-tags and see all the tags in all services, instead of going one by one for each service like so: aws <service name> <command to describe/list tags>
You can use : aws resourcegroupstaggingapi get-resources
From the docs:
Returns all the tagged or previously tagged resources that are located
in the specified Region for the AWS account.
Usage:
aws resourcegroupstaggingapi get-resources
Output:
{
"ResourceTagMappingList": [
{
"ResourceARN": " arn:aws:inspector:us-west-2:123456789012:target/0-nvgVhaxX/template/0-7sbz2Kz0",
"Tags": [
{
"Key": "Environment",
"Value": "Production"
}
]
}
]
}

AWS CLI EMR get Master node Instance ID and tag it

I want to automate the running of a cluster and can use tags to get attributes of an EC2 instance like its instance-id.
The documentation on https://docs.aws.amazon.com/cli/latest/reference/emr/create-cluster.html states that
--tags (list)
A list of tags to associate with a cluster, which apply to each Amazon
EC2 instance in the cluster. Tags are key-value pairs that consist of
a required key string with a maximum of 128 characters, and an
optional value string with a maximum of 256 characters.
You can specify tags in key=value format or you can add a tag without
a value using only the key name, for example key . Use a space to
separate multiple tags.
So this applies tags to every EC2 instance including the master and slaves. How do I discern which instance is the master node?
Additional Info:
I am using the following command to get attributes from aws cli based on tags where you can replace the "Name" and "Prod" with your tags key-value pairs respectively.
aws ec2 describe-instances | jq '.Reservations[].Instances | select(.[].Tags[].Value | startswith("Prod") ) | select(.[].Tags[].Key == "Name") | {InstanceId: .[].InstanceId, PublicDnsName: .[].PublicDnsName, State: .[].State, LaunchTime: .[].LaunchTime, Tags: .[].Tags} | [.]' | jq .[].InstanceId
As you noted when you create an EMR cluster, the tags are the same for all nodes (Master, Slave, Task).
You will find that this process using the AWS CLI to be complicated. My recomendation is to review the examples below and then write a Python program to do this.
Process to add your own tags to the EC2 instances.
STEP 1: List your EMR Clusters:
aws emr list-clusters
This will output JSON:
{
"Clusters": [
{
"Id": "j-ABCDEFGHIJKLM",
"Name": "'MyCluster'",
"Status": {
"State": "WAITING",
"StateChangeReason": {
"Message": "Cluster ready after last step completed."
},
"Timeline": {
"CreationDateTime": 1536626095.303,
"ReadyDateTime": 1536626568.482
}
},
"NormalizedInstanceHours": 0
}
]
}
STEP 2: Make a note of the Cluster ID from the JSON:
"Id": "j-ABCDEFGHIJKLM",
STEP 3: Describe your EMR Cluster:
aws emr describe-cluster --cluster-id j-ABCDEFGHIJKLM
This will output JSON (I have truncated this output to just the MASTER section):
{
"Cluster": {
"Id": "j-ABCDEFGHIJKLM",
"Name": "'Test01'",
....
"InstanceGroups": [
{
"Id": "ig-2EHOYXFABCDEF",
"Name": "Master Instance Group",
"Market": "ON_DEMAND",
"InstanceGroupType": "MASTER",
"InstanceType": "m3.xlarge",
"RequestedInstanceCount": 1,
"RunningInstanceCount": 1,
"Status": {
"State": "RUNNING",
"StateChangeReason": {
"Message": ""
},
"Timeline": {
"CreationDateTime": 1536626095.316,
"ReadyDateTime": 1536626533.886
}
},
"Configurations": [],
"EbsBlockDevices": [],
"ShrinkPolicy": {}
},
....
]
}
}
STEP 4: InstanceGroups is an array. Find the entry where InstanceGroupType is MASTER. Make note of the Id.
"Id": "ig-2EHOYXFABCDEF",
STEP 5: List your cluster instances:
aws emr list-instances --cluster-id j-ABCDEFGHIJKLM
This will output JSON (I have truncated the output):
{
"Instances": [
....
{
"Id": "ci-31LGK4KIECHNY",
"Ec2InstanceId": "i-0524ec45912345678",
"PublicDnsName": "ec2-52-123-201-221.us-west-2.compute.amazonaws.com",
"PublicIpAddress": "52.123.201.221",
"PrivateDnsName": "ip-172-31-41-111.us-west-2.compute.internal",
"PrivateIpAddress": "172.31.41.111",
"Status": {
"State": "RUNNING",
"StateChangeReason": {},
"Timeline": {
"CreationDateTime": 1536626164.073,
"ReadyDateTime": 1536626533.886
}
},
"InstanceGroupId": "ig-2EHOYXFABCDEF",
"Market": "ON_DEMAND",
"InstanceType": "m3.xlarge",
"EbsVolumes": []
}
]
}
STEP 6: Find the matching InstanceGroupId ig-2EHOYXFABCDEF. This will give you the EC2 Instance ID for the MASTER: "Ec2InstanceId": "i-0524ec45912345678"
Step 7: Tag your EC2 instance:
aws ec2 create-tags --resources i-0524ec45912345678 --tags Key=EMR,Value=MASTER
The above steps might be simpler with CLI Filters and / or jq, but this should be enough information so that you know how to find and tag the EMR Master Instance.
Below can be used to directly get instance Id
aws emr list-instances --cluster-id ${aws_emr_cluster.cluster.id} --instance-
group-id ${aws_emr_cluster.cluster.master_instance_group.0.id} --query
'Instances[*].Ec2InstanceId' --output text
In an enviroinment where you does not have the aws cli, you can cat the following file:
cat /mnt/var/lib/info/job-flow.json
An example of the content is the following one:
{
"jobFlowId": "j-0000X0X0X00XX",
"jobFlowCreationInstant": 1579512208006,
"instanceCount": 2,
"masterInstanceId": "i-00x0xx0000xxx0x00",
"masterPrivateDnsName": "localhost",
"masterInstanceType": "m5.xlarge",
"slaveInstanceType": "m5.xlarge",
"hadoopVersion": "2.8.5",
"instanceGroups": [
{
"instanceGroupId": "ig-0XX00XX0X0XXX",
"instanceGroupName": "Master - 1",
"instanceRole": "Master",
"marketType": "OnDemand",
"instanceType": "m5.xlarge",
"requestedInstanceCount": 1
},
{
"instanceGroupId": "ig-000X0XXXXXXX",
"instanceGroupName": "Core - 2",
"instanceRole": "Core",
"marketType": "OnDemand",
"instanceType": "m5.xlarge",
"requestedInstanceCount": 1
}
]
NOTE: i've omitted the ID of the jobs using 0 where a number is expected and X where a ltter is expected.
You can do this programmatically in 3 lines of code, without having to copy-paste any of the specific information:
# get cluster id
CLUSTER_ID=$(aws emr list-clusters --active --query "Clusters[0].Id" --output text)
# get instance id
INSTANCE_ID=$(aws emr list-instances --cluster-id $CLUSTER_ID --instance-group-types MASTER --query "Instances[0].Ec2InstanceId" --output text)
# tag
aws ec2 create-tags --resources $INSTANCE_ID --tags Key=EMR,Value=MASTER
Below example is for Instance Fleet, it saves Cluster ID, Instance Fleet ID and Master IP as environment variables.
Replace cluster name "My-Cluster" to the actual cluster name.
export CLUSTER_ID=$(aws emr list-clusters --active --query 'Clusters[?Name==`My-Cluster`].Id' --output text)
export INSTANCE_FLEET=$(aws emr describe-cluster --cluster-id $CLUSTER_ID | jq -r '.[].InstanceFleets | .[] | select(.InstanceFleetType=="MASTER") | .Id')
export PRIVATE_IP=aws emr list-instances --cluster-id $CLUSTER_ID --instance-fleet-id $INSTANCE_FLEET --query 'Instances[*].PrivateIpAddress' --output text
"Cleanest" way:
aws emr list-clusters --active
Search for Master cluster ID (j-xxxxxxxxxxx), then
aws emr list-instances --region {your_region} --instance-group-types MASTER --cluster-id j-xxxxxxxxxxxxx
Immediately filters out the master instance(s) with it's information using --instance-group-types MASTER flag.
For tagging refers to the other answers with aws {resource} create-tags and --tag flag.

Setting Environment Variables per step in AWS EMR

I am unable to set environment variables for my spark application. I am using AWS EMR to run a spark application. Which is more like a framework I wrote in python on top of spark, to run multiple spark jobs according to environment variables present. So in order for me to start the exact job, I need to pass the environment variable into the spark-submit. I tried several methods to do this. But none of them works. As I try to print the value of the environment variable inside the application it returns empty.
To run the cluster in the EMR I am using following AWS CLI command
aws emr create-cluster --applications Name=Hadoop Name=Hive Name=Spark --ec2-attributes '{"KeyName":"<Key>","InstanceProfile":"<Profile>","SubnetId":"<Subnet-Id>","EmrManagedSlaveSecurityGroup":"<Group-Id>","EmrManagedMasterSecurityGroup":"<Group-Id>"}' --release-label emr-5.13.0 --log-uri 's3n://<bucket>/elasticmapreduce/' --bootstrap-action 'Path="s3://<bucket>/bootstrap.sh"' --steps file://./.envs/steps.json --instance-groups '[{"InstanceCount":1,"InstanceGroupType":"MASTER","InstanceType":"c4.xlarge","Name":"Master"}]' --configurations file://./.envs/Production.json --ebs-root-volume-size 64 --service-role EMRRole --enable-debugging --name 'Application' --auto-terminate --scale-down-behavior TERMINATE_AT_TASK_COMPLETION --region <region>
Now Production.json looks like this:
[
{
"Classification": "yarn-env",
"Properties": {},
"Configurations": [
{
"Classification": "export",
"Properties": {
"FOO": "bar"
}
}
]
},
{
"Classification": "spark-defaults",
"Properties": {
"spark.executor.memory": "2800m",
"spark.driver.memory": "900m"
}
}
]
And steps.json like this :
[
{
"Name": "Job",
"Args": [
"--deploy-mode","cluster",
"--master","yarn","--py-files",
"s3://<bucket>/code/dependencies.zip",
"s3://<bucket>/code/__init__.py",
"--conf", "spark.yarn.appMasterEnv.SPARK_YARN_USER_ENV=SHAPE=TRIANGLE",
"--conf", "spark.yarn.appMasterEnv.SHAPE=RECTANGLE",
"--conf", "spark.executorEnv.SHAPE=SQUARE"
],
"ActionOnFailure": "CONTINUE",
"Type": "Spark"
}
]
When I try to access the environment variable inside my __init__.py code, it simply prints empty. As you can see I am running the step using spark with yarn cluster in cluster mode. I went through these links to reach this position.
How do I set an environment variable in a YARN Spark job?
https://spark.apache.org/docs/latest/configuration.html#environment-variables
https://spark.apache.org/docs/latest/configuration.html#runtime-environment
Thanks for any help.
Use classification yarn-env to pass environment variables to the worker nodes.
Use classification spark-env to pass environment variables to the driver, with deploy mode client. When using deploy mode cluster, use yarn-env.
(Dear moderator, if you want to delete the post, let me know why.)
To work with EMR clusters I work using the AWS Lambda, creating a project that build an EMR cluster when a flag is set in the condition.
Inside this project, we define the variables that you can set in the Lambda and then, replace this to its value. To use this, we have to use the AWS API. The possible method you have to use is the AWSSimpleSystemsManagement.getParameters.
Then, make a map like val parametersValues = parameterResult.getParameters.asScala.map(k => (k.getName, k.getValue)) to have a tuple with its name and value.
Eg: ${BUCKET} = "s3://bucket-name/
What this means, you only have to write in your JSON ${BUCKET} instead all the name of your path.
Once you have replace the value, the step JSON can have a view like this,
[
{
"Name": "Job",
"Args": [
"--deploy-mode","cluster",
"--master","yarn","--py-files",
"${BUCKET}/code/dependencies.zip",
"${BUCKET}/code/__init__.py",
"--conf", "spark.yarn.appMasterEnv.SPARK_YARN_USER_ENV=SHAPE=TRIANGLE",
"--conf", "spark.yarn.appMasterEnv.SHAPE=RECTANGLE",
"--conf", "spark.executorEnv.SHAPE=SQUARE"
],
"ActionOnFailure": "CONTINUE",
"Type": "Spark"
}
]
I hope this can help you to solve your problem.

AWS Elastic Beanstalk - switching environment variables between environments

I am using Elastic Beanstalk's aws:elasticbeanstalk:application:environment namespace to configure my environment with env vars. How can I set different values for different environments (e.g. development versus production)?
Development:
option_settings:
aws:elasticbeanstalk:application:environment:
REDIS_HOST: localhost
Production:
option_settings:
aws:elasticbeanstalk:application:environment:
REDIS_HOST: prod.redis.server.com
The AWS CLI has a convenient way of doing this for you as the update-environment command allows you to set env vars from a specially formatted json file. Create a separate json file for each environment you will be deploying to.
Example json file named deploy-dev.json:
[
{
"Namespace": "aws:elasticbeanstalk:application:environment",
"OptionName": "NODE_ENV",
"Value": "dev"
},
{
"Namespace": "aws:elasticbeanstalk:application:environment",
"OptionName": "LOG_LEVEL",
"Value": "silly"
}
]
Deploy app and then update env vars:
aws elasticbeanstalk create-application-version --application-name "$EB_APP_NAME" --version-label "$EB_VERSION"
aws elasticbeanstalk update-environment --environment-name "$EB_ENV_NAME" --version-label "$EB_VERSION" --option-settings file://.ebextensions/deploy-dev.json
How it's changed depends on your deployment method.
One option you can use is to set a bogus value in your .config file:
option_settings:
aws:elasticbeanstalk:application:environment:
REDIS_HOST: change me
Then, after deployment, modify the variable either using the AWS Management Console or using the EB CLI:
eb setenv REDIS_HOST=prod.redis.server.com
If you are using CloudFormation to deploy your EB application, you can feed the value as part of the OptionSettings field in your CloudFormation template:
"EBConfigurationTemplate" : {
"Type" : "AWS::ElasticBeanstalk::ConfigurationTemplate",
"Properties" : {
"ApplicationName" : {
"Ref" : "EBApplication"
},
"Description" : "Configuration Template",
"OptionSettings" : [
{
"Namespace" : "aws:elasticbeanstalk:application:environment",
"OptionName" : "REDIS_HOST",
"Value" : {
"Ref" : "RedisHostInputParameter"
}
},
]
There are probably other methods too, but they will depend on the method of deployment.

Map 'ec2-register snapshot' syntax onto 'register-image AMI 'syntax using awscli

What is the correct syntax for mapping a snapshot onto an AMI using awscli?
More explicitly, how do I map the old syntax
'ec2-register -s snap-9abc1234 --kernel 99abcdef' onto the new syntax
'aws ec2 --register-image' ?
It's the following:
aws ec2 register-image --kernel-id <your-kernel> --root-device-name /dev/sda1 --block-device-mappings [list in JSON shown below]
[
{
"VirtualName": "string",
"DeviceName": "string",
"Ebs": {
"SnapshotId": "string",
"VolumeSize": integer,
"DeleteOnTermination": true|false,
"VolumeType": "standard"|"io1",
"Iops": integer
},
"NoDevice": "string"
}
...
]
You can run aws ec2 register-image help for help on the command.
Make sure you are using the awscli python package on not the aws package as that one is different (not the official one)
Here's a link to the github repo:
https://github.com/aws/aws-cli