I want to securely fetch configuration files from S3 using a secure VPC from my Docker container. But I want to determine inside the application which configuration file to fetch and use based on the region I am on. Is there a good/best practice to go on about describing the current container's region?
I understand that you can use the AWS SDK/CLI to describe the ECS instances, but that doesn't tell me which one the container is specifically deployed on.
Use the metadata server to query the availability-zone from which you can get the region.
$ curl 169.254.169.254/latest/meta-data/placement/availability-zone/
us-east-1a
One example if you are using python SDK is:
import os
az = os.popen("curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone/").read()
print az[:-1]
>>> us-east-1
Within EC2, you can retrieve the instance metadata using a simple curl command to a local (internal) web API. Region and AZ are some of the data points you can get:
http://169.254.169.254/latest/meta-data/services/domain
http://169.254.169.254/latest/meta-data/placement/availability-zone
See this page for full details about instance metadata.
Within ECS, I'd be interested to see if these might still work -- my hunch is they would, as the container should query the host machine's API for the answer, and the ECS host is most certainly an EC2 instance.
Let us know if that works?
Related
As of the date of this question I'm using the most recent version of the AWS CLI (2.4.6) running on macOS. According to the v2 docs the Instances that are returned should include properties like InstanceLifecycle, Licenses, MetadataOptions -> PlatformDetails and several others that are missing for me. While I'm getting back most data, some fields are absent... I've tried this is two separate AWS accounts and I have admin IAM creds that I'm using locally, why does the aws ec2 describe-instances call not return all of the fields listed in the docs?
Not all outputs is available for every ec2 instance, it depends on the way of provisioning of your ec2 instances.
Ex:
InstanceLifecycle: is exclusive if you provisioned the ec2 instance as spot instance or reserved one.
Licenses: If you used BYOL when provisioning EC2 (Bring your own license)
Extra.. The docs describe every possible output from querying ec2 api endpoint, but it depends on the different parameters of your provisioned ec2 instance.
For example, try to provision a spot instance, and query the instance lifecycle.
I'd like to know if there is an approach to automatically join an EC2 instance (launched via ElasticBeantalk) to an Active Directory already hosted in an EC2 Windows Server.
Following this document below I could see that is possible using AD Connector or AWS Simple AD (either will give a "directoryId)," but in my case, the AD already is hosted in an EC2.
https://aws.amazon.com/blogs/security/how-to-configure-your-ec2-instances-to-automatically-join-a-microsoft-active-directory-domain/
You can do it by following the same path - but instead of relying to the default SSM document for Managed AD, you should create a custom one.
In it, you can omit the directory id and rely on DNS name / IP address of your EC2 AD and respective domain name, username and password with which you join.
This approach is partially described in the following blog post. It has an actual SSM document as a CloudFormation stack which you can use as a base.
In order to fully automate it you can create an Amazon EventBridge rule that will be looking for EC2 Launch instance events and has SSM Run Command as a target (Run Command can start your custom SSM document). There are some dedicated Beanstalk events that may be useful in this regard as well.
I have my microservice running in AWS ECS, and I want to tell which region this service is running in. Do they have a meta data service for me get my microservice region?
There are two ways to do this. The first is to use the Metadata file. This feature is disabled by default so you'll need to turn it on.
Run cat $ECS_CONTAINER_METADATA_FILE on linux after enabling it to see the metadata. The ENV var stores the file location.
The second is to use the HTTP metadata endpoint. There are two potential endpoints here (version 2 and 3) depending on how the instance is launched, so check the docs.
In either case the region is not a specific property of the metadata, but it can be inferred from the ARN.
I have an EC2 instance that is running a few processes. I also have a Lambda script that is triggered through various means. I would like this Lambda script to talk to my EC2 instance and get a list of running processes from it (Essentially run ps aux on the EC2 box, and read the output).
Now this is easy enough with just one instance and its instance-id. Just SSH in, run the command, get the output, and be on my way. However, I would like to scale this to multiple EC2 instances, for which only the instance-id is known and SSH keys may not be given.
Is such a configuration possible with Lambda and Boto (or other libraries)? Or do I just have to run a microserver on each of my instances that will reply with the given information (something I'm really trying to avoid)
You can do this easily with AWS Systems Manager - Run Command
AWS Systems Manager provides you safe, secure remote management of your instances at scale without logging into your servers, replacing the need for bastion hosts, SSH, or remote PowerShell.
Specifically:
Use the send-command API from Lambda function to get list of all processes on a group of instances. You can do this by providing a list of instances or even a tag query
You can also use CloudWatch Events to trigger a Run Command directly
I don't think there is something available out of the box for this scenario.
Instead of querying, try an alternate approach. Install an agent on all ec2 instances, which reports the required information to a central service or probably a DynamoDB table, with HashKey as InstanceId.
You may want to bake this script as a cron job, (executed probably hourly?) in the AMI itself.
With this implementation, you reduce the complexity of managing and running a separate web service on each EC2 instance.
Query the DynamoDB table on demand. There will be a lag, as data may not be real time, but you can always reduce the CRON interval per your needs.
Like Yeshodhan mentioned, There is no direct approach for this.
However, There is one more approach.
1) Save your private key file to an s3 bucket, Create a lambda function and use python fabric module to login to the remote machines from lambda function and execute commands.
The above-mentioned approach is possible but I highly recommend launching a separate machine and use a configuration management system (Preferably ansible) and get the results from remote machines.
I know from an AWS API call (similar to aws rds describe-db-instances --db-instance-identifier instanceIdentifier) that my AWS RDS Postgres instance has dbInstanceClass equal to "db.t2.large".
Using another API call, I want to get the attributes of a "db.t2.large" instance class especially the allocated memory.
I can get the value from the AWS documentation but I want the value programmatically.
What would be the API call or the equivalent on the command line?
An RDS instance is some kind of specialized EC2 instance but I did not find anything either under the EC2 API.
Thanks
Olivier
From other people, I got the answer that there is no API for this. You have to rely on screen scraping...
Outside of the AWS documentation page, this web site http://www.ec2instances.info/rds gives the values for EC2 and RDS and you can download all the values or specific columns as a CSV file.