I have my microservice running in AWS ECS, and I want to tell which region this service is running in. Do they have a meta data service for me get my microservice region?
There are two ways to do this. The first is to use the Metadata file. This feature is disabled by default so you'll need to turn it on.
Run cat $ECS_CONTAINER_METADATA_FILE on linux after enabling it to see the metadata. The ENV var stores the file location.
The second is to use the HTTP metadata endpoint. There are two potential endpoints here (version 2 and 3) depending on how the instance is launched, so check the docs.
In either case the region is not a specific property of the metadata, but it can be inferred from the ARN.
Related
CONTEXT:
We have a platform where users can create their own projects - multiple projects per user. We need to provide them with a browser-based IDE to edit those projects.
We decided to go with coder-server. For this we need to configure an auto-scalable cluster on AWS. When the user clicks "Edit Project" we will bring up a new container each time.
https://hub.docker.com/r/codercom/code-server
QUESTION:
How to pass parameters from the url query (my-site.com/edit?project=1234) into a startup script to pre-configure the workspace in a docker container when it starts?
Let's say the stack is AWS + ECS + Fargate. We could use kubernetes instead of ECS if it helps.
I don't have any experience in cluster configuration. Will appreciate any help or at least a direction where to dig further.
The above can be achieved using multiple ways in AWS ECS. The basic requirements for such systems are to launch and terminate containers on the fly while persisting the changes in the files. (I will focus on launching the containers)
Using AWS SDK's:
The task can be easily achieved using AWS SDKs, Using a base task definition. AWS SDK allows starting tasks with overrides on the base task definition.
E.G. If task definition has a memory of 2GB then the SDK can override the memory to parameterised value while launching a task from task def.
Refer to the boto3 (AWS SDK for Python) docs.
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs.html#ECS.Client.run_task
Overall Solution
Now that we know how to run custom tasks with python SDK (on demand). The overall flow for your application is your API calling AWS lambda function whit parameters to spin up and wait to keep checking task status and update and rout traffic to it once the status is healthy.
API calls AWS lambda functions with parameters
Lambda function using AWS SDK create a new task with overrides from base task definition. (assuming the base task definition already exists)
Keep checking the status of the new task in the same function call and set a flag in your database for your front end to be able to react to it.
Once the status is healthy you can add a rule in the application load balancer using AWS SDK to route traffic to the IP without exposing the IP address to the end client (AWS application load balancer can get expensive, I'll advise using Nginx or HAProxy on ec2 to manage dynamic routing)
Note:
Ensure your Image is lightweight, and the startup times are less than 15 mins as lambda cannot execute beyond that. If that's the case create a microservice for launching ad-hoc containers and hosting them on EC2
Using Terraform:
If you looking for infrastructure provisioning terraform is the way to go. It has a learning curve so recommend it as a secondary option.
Terraform is popular for parametrising using variables and it can be plugged in easily as a backend for an API. The flow of your application still remains the same from step 1, but instead of AWS Lambda API will be calling your ad-hoc container microservice, which in turn calls terraform script and passing variables to it.
Refer to the Terrafrom docs for AWS
https://registry.terraform.io/providers/hashicorp/aws/latest
Currently I have several redis clusters for different env. In my code, I will write data to redis inside my lambda function, if I deploy this lambda to my aws account, how can it update the corresponding redis cluster? Since every environment has its own redis cluster.
You can have a config file with Redis cluster name and their hostnames and in code, you can pick different clusters based on the env provided.
If you are using roles in the AWS account, for each environment then you should also do STS on requirred role
Your resource files are just files. They will be loaded by your application following different strategies, depend on your application's framework and how it has been configured.
Some application, at build-time, apply the correct configuration, by passing a flag to the build for example. --uat or --prod. If your application is one of those kind, you can just build the correct version and push it to AWS. They will connect to the correct Redis, given that you put the redis configuration into the ENV files
The other option is to use Environment Variable
Bear with me, what I am requesting may be impossible. I am a AWS noob.
So I am going to describe to you the situation I am in...
I am doing a freelance gig and was essentially handed the keys to AWS. That is, I was handed the root user login credentials for the AWS account that powers this website.
Now there are 3 EC2 instances. One of the instances is a linux box that, from what I am being told, is running a Django Python backend.
My new "service" if you will must exist within this instance.
How do I introduce new source code into this instance? Is there a way to pull down the existing source code that lives within it?
I am not be helped by any existing/previous developers so I am kind of just handed the AWS credentials and have no idea where to start.
Is this even possible. That is, is it possible to pull the source code from an EC2 instance and/or modify the code? How do I do this?
EC2 instances are just virtual machines. So you can use SSH/SCP/SFTP files to and from. You can use the AWS CLI tools to copy stuff from S3. Dealers choice...
Now to get into this instance... If you look in the web console you can find its IP(s), what the security groups (firewall rules), and the key pair name. Hopefully they gave you the keys. You need these to SSH in.
You'll also want to check to make sure there's a security group applied that has SSH open. Hopefully only to your IP :)
If you don't have the keys you'll have to create an AMI image of the instance so you can create a new one with a key pair you do have.
Amazon has a set of tools for you in Amazon CodeSuite.
The tool used for "deploying" the code is Amazon CodeDeploy. By using this service you install an agent onto your host, then when triggered it will pull down an artifact of a code base and install it matching hosts. You can even specify additional commands through the hook system.
But you also want to trigger this to happen, maybe even automatically? CodeDeploy can be orchestrated using the CodePipeline tool.
I'm attempting to setup a Consul 1.0 cluster in ECS using Terraform. I am able to get Consul up and running as a cluster, but I am running into ACL errors, as documented here. The problem I am having is running the associated curl scripts to create a token with the proper rules, saving that outputted token, and running it on every member of the autoscale group both for the first time and every time the group scales up.
Does anyone have any suggestions on how to get this knocked out?
So what I ended up doing was creating a lambda script to handle 2 types of events: bootstrap and adding new nodes, which is triggered by either a local_exec in TF (bootstrap) or an autoscaling group sns notification (add new node). The bootstrap function stored the acl_agent_token in an SSM Parameter Store and applied it initially to the members of the cluster. The function that adds new nodes queries the parameter store and adds the node via the rest api.
One way of implementing this is storing the created token in Vault or S3 (encrypted with KMS) add a few lines to user data to retrieve it back, locking it down with the appropriate IAM policies.
I want to securely fetch configuration files from S3 using a secure VPC from my Docker container. But I want to determine inside the application which configuration file to fetch and use based on the region I am on. Is there a good/best practice to go on about describing the current container's region?
I understand that you can use the AWS SDK/CLI to describe the ECS instances, but that doesn't tell me which one the container is specifically deployed on.
Use the metadata server to query the availability-zone from which you can get the region.
$ curl 169.254.169.254/latest/meta-data/placement/availability-zone/
us-east-1a
One example if you are using python SDK is:
import os
az = os.popen("curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone/").read()
print az[:-1]
>>> us-east-1
Within EC2, you can retrieve the instance metadata using a simple curl command to a local (internal) web API. Region and AZ are some of the data points you can get:
http://169.254.169.254/latest/meta-data/services/domain
http://169.254.169.254/latest/meta-data/placement/availability-zone
See this page for full details about instance metadata.
Within ECS, I'd be interested to see if these might still work -- my hunch is they would, as the container should query the host machine's API for the answer, and the ECS host is most certainly an EC2 instance.
Let us know if that works?