I have a basic node app that I've wrapped in a Dockerfile
FROM node:lts-alpine3.15
WORKDIR /app
COPY package.json ./
RUN npm install
COPY . .
EXPOSE 8080
CMD [ "npm", "run", "serve" ]
I push that to Gitlab's container registry. I'm trying to deploy it from there to AWS, but running into problems on the ECS side. In ECS I have:
a cluster (frontend)
a service (frontend)
both of which are configured in terraform
resource "aws_ecs_cluster" "frontend" {
name = "frontend"
setting {
name = "containerInsights"
value = "enabled"
}
}
resource "aws_ecs_service" "frontend" {
name = "frontend"
cluster = aws_ecs_cluster.frontend.id
deployment_controller {
type = "EXTERNAL"
}
tags = {
Name = "WebAppFrontend"
}
}
The web app is in a different repository from the terraform infrastructure. In my .gitlab-ci.yml I'm trying to register a new task definition for the web app I'm trying to register a new task definition with a json file.
I want when there's been changes to the web app I was to perform a rolling update so both the new version and old version are running, but I can't get one version deployed to ecs. My .gitlab-ci.yml is
deploy_ecs:
stage: deploy_ecs
script:
- aws ecs register-task-definition --cli-input-json file://task_definition.json
task_definition.json is:
{
"family": "frontend",
"containerDefinitions": [
{
"name": "frontend",
"image": "registry.gitlab.com/myproject/application/myimage:latest",
"memory": 300,
"portMappings": [
{
"containerPort": 8080,
"hostPort": 80
}
],
"essential": true,
"environment": [
{
"name": "Frontend",
"value": "dev"
}
]
}
]
}
Attempting to create a service from the console I get this error
The selected task definition is not compatible with the selected compute strategy.
Manually on the ec2 instance infrastructure for the ecs cluster I can run
docker run -d -p 80:8080 myimage
which does run the app. Am I able to:
Deploy the task definition file as above and run the service in my cluster
Deploy in a way so that there will be both versions in a rolling update to avoid any downtime
Do both of the above from my .gitlab-ci.yml
The ec2 instance is confirmed to be running the ecs-agent and I can see the container instance showing correctly so I know ecs is running.
I used console and the service was created successfully.
{
"requiresCompatibilities": [
"EC2"
],
"family": "frontend",
"containerDefinitions": [
{
"name": "frontend",
"image": "registry.gitlab.com/myproject/application/myimage:latest",
"memory": 300,
"portMappings": [
{
"containerPort": 8080,
"hostPort": 80
}
],
"essential": true,
"environment": [
{
"name": "Frontend",
"value": "dev"
}
]
}
]
}
The task eventually failed with access denied but the rest everything worked. Plus you need to add the " ecsTaskExecutionRole" for the task to function.
Related
I have successfully pushed my 3 docker images on ECR.
Configured an ECS cluster.
Created 3 task definitions for those 3 images stored in respective ECR repositories.
Now, I want to run a public image of redis on the same cluster as a different task. I tried created a task definition of the same using the following URL: public.ecr.aws/ubuntu/redis:latest
But as soon as I run it as a new task I get the following error:
Essential container in task exited
Any specific reason for this error or am I doing something wrong?
Ok, so the redis image needs to either set a password (I recommend this as well) or allow it to connect to the redis cluster without a password.
To configure a password or disable passwords auth, you need to set environment variables in the image. You can read the documentation under the heading Configuration
Luckily, this is easy in ECS. You need to specify the environment variable in the task definition. So either:
{
"family": "",
"containerDefinitions": [
{
"name": "",
"image": "",
...
"environment": [
{
"name": "ALLOW_EMPTY_PASSWORD",
"value": "yes"
}
],
...
}
],
...
}
or for a password:
{
"family": "",
"containerDefinitions": [
{
"name": "",
"image": "",
...
"environment": [
{
"name": "REDIS_PASSWORD",
"value": "your_password"
}
],
...
}
],
...
}
For more granular configuration you should read the documentation of the redis docker image I posted above.
I was trying to set up a ECS service running a container image on a cluster, but could not get the setup working.
I have basically followed the guide on https://docs.aws.amazon.com/AmazonECS/latest/developerguide/create-blue-green.html, except that I was trying to host the containers on EC2 instances.
I wonder if the issue is related to the network mode (used "awsvpc").
Expectation
It should show something on index.html on access witht eh ALB link
Observation
When I tried to access with the load balancer link, it gives HTTP 503, and the health-check also showed unhealthy
And it seems ECS keeps "re-creating" the conatiners? (Forgive me as I am still not familiar with ECS)
Tried to access the container instance directly but also could not reach
I had a look on the ECS agent log (/var/logs/ecs-agent.log) on the container instance, the image should have been pulled sucessfully
And the task should have been started
ECS service events
It seems it kept register and deregister target
Security groups have been set to accept HTTP traffic
Setup
Tomcat server on container starts on port 80
ALB
Listener
Target group
ECS task definition creation
{
"family": "TestTaskDefinition",
"networkMode": "awsvpc",
"containerDefinitions": [
{
"name": "TestContainer",
"image": "<Image URI>",
"portMappings": [
{
"containerPort": 80,
"hostPort": 80,
"protocol": "tcp"
}
],
"essential": true
}
],
"requiresCompatibilities": [
"EC2"
],
"cpu": "256",
"memory": "512",
"executionRoleArn": "<ECS execution role ARN>"
}
ECS service creation
{
"cluster": "TestCluster",
"serviceName": "TestService",
"taskDefinition": "TestTaskDefinition",
"loadBalancers": [
{
"targetGroupArn": "<target group ARN>",
"containerName": "TestContainer",
"containerPort": 80
}
],
"launchType": "EC2",
"schedulingStrategy": "REPLICA",
"deploymentController": {
"type": "CODE_DEPLOY"
},
"networkConfiguration": {
"awsvpcConfiguration": {
"assignPublicIp": "DISABLED",
"securityGroups": [ "sg-0f9b629686ca3bd08" ],
"subnets": [ "subnet-05f47b367df4f50d4", "subnet-0fd76fc8e47ea3be7" ]
}
},
"desiredCount": 1
}
Based on the comments.
To investigate the issue, it was recommended to tested the ECS service without ALB. Based on the test, it was found that the ALB was treating the ECS service as unhealthy due to long application starting time.
The issue was solved by increasing ALB health-check grace period to (e.g. 300s).
not sure if EC2 launch type must use "bridge"
You can use awsvpc on EC2 instances as well, but bridge is easier to use in this case.
Below is my AWS Task Definition for ECS.
I need every EC2 instance of this task to have port 3026 publicly accessible to the world. How can I modify this JSON to do that?
Currently, after service is running this task, I manually find the EC2 instance(s) and then I manually add a security group that allows ingress from 0.0.0.0/0 on that port.
But I really want to know how to make this JSON do it so I no longer have to do it manually.
{
"family": "myproj",
"requiresCompatibilities": [
"EC2"
],
"containerDefinitions": [
{
"memory": 500,
"memoryReservation": 350,
"name": "myproj",
"image": "blah.dkr.ecr.us-east-1.amazonaws.com/myproj:latest",
"essential": true,
"portMappings": [
{
"hostPort": 3026,
"containerPort": 8000,
"protocol": "tcp"
}
],
"entryPoint": [
"./entrypoint_deployment.sh"
],
"environment" : [
{ "name" : "DB_HOST", "value" : "blah.blah.us-east-1.rds.amazonaws.com" }
],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/myproj",
"awslogs-region": "us-east-1",
"awslogs-stream-prefix": "ecs"
}
}
}
]
}
My suggested approach is to configure an ECS service associated to your task, and then use an Application Load Balancer (ALB) to route the public traffic to this service.
This guide should help you: https://aws.amazon.com/blogs/compute/microservice-delivery-with-amazon-ecs-and-application-load-balancers/
Another (cheaper) option is to use the EC2 instance metadata API provided by Amazon, read the instance_id value from that API and use the "aws-cli" utility to update the security group when your container starts. A script like this should work (to run inside the container):
export SECURITY_GROUP=sg-12345678
export INSTANCE_ID=$(curl http://169.254.169.254/latest/meta-data/instance-id)
aws ec2 modify-instance-attribute --instance-id INSTANCE_ID --groups $SECURITY_GROUP
You need to set the SECURITY_GROUP accordingly and have the aws ec2 utility installed in the docker image of the task that you are running.
Furthermore, you need to change the ENTRYPOINT of your task docker image to run the script, for example:
"entryPoint": [
"./script_to_setup_SG.sh && ./entrypoint_deployment.sh"
],
I am trying to run a dockerised Jenkins and postgres database on AWS elastic beanstalk in a multi-container t2.micro environment:
Dockerrun.aws.json
{
"AWSEBDockerrunVersion": 2,
"containerDefinitions": [
{
"name": "postgres-db",
"image": "postgres:9.5-alpine",
"essential": true,
"memory": 256,
"portMappings": [
{
"hostPort": 5432,
"containerPort": 5432
}
]
},
{
"name": "jenkins-blueocean",
"image": "<account_id>.dkr.ecr.ap-southeast-2.amazonaws.com/<image>:latest",
"essential": true,
"memory": 256,
"mountPoints": [
{
"sourceVolume": "jenkins-data",
"containerPath": "/var/jenkins_home"
}
],
"portMappings": [
{
"hostPort": 80,
"containerPort": 8080
}
],
"links": [
"postgres-db"
]
}
],
"volumes": [
{
"name": "jenkins-data",
"host": {
"sourcePath": "/var/jenkins-data"
}
}
]
}
AWS shows it deploys fine but the logs for jenkins-blueocean container has that error:
/var/log/containers/jenkins-blueocean-7ce78063214b-stdouterr.log
touch: cannot touch '/var/jenkins_home/copy_reference_file.log': Permission denied
Can not write to /var/jenkins_home/copy_reference_file.log. Wrong volume permissions?
Am I missing something to allow jenkins access to the volume?
Thanks in advance!
Not 100% sure if this is the right path but we ended up following the .ebextensions method of running commands to setup the volume path to allow the jenkins user from the jenkins-blueocean container full access to do its thing.
mkdir -p /var/jenkins-data
chmod 777 /var/jenkins-data
This was because the permissions on the location in the docker instance has r-x rights for other users, with root user having rwx.
I'm deploying an ASP.NET Core Web API app as a docker image to AWS ECS, so use a task definition file for that.
It turns out the app only works if I specify environment variable VIRTUAL_HOST with the public DNS of my EC2 instance (as highlighted here: http://docs.servicestack.net/deploy-netcore-docker-aws-ecs), see taskdef.json below:
{
"family": "...",
"networkMode": "bridge",
"containerDefinitions": [
{
"image": "...",
"name": "...",
"cpu": 128,
"memory": 256,
"essential": true,
"portMappings": [
{
"containerPort": 80,
"hostPort": 0,
"protocol": "http"
}
],
"environment": [
{
"name": "VIRTUAL_HOST",
"value": "ec2-xx-xxx-xxx-xxx.compute-1.amazonaws.com"
}
]
}
]
}
Once the app is deployed to AWS ECS, I hit the endpoints - eg http://ec2-xx-xxx-xxx-xxx.compute-1.amazonaws.com/v1/ping
with the actual public DNS of my EC2 instance in VIRTUAL_HOST all works fine
without the env variable I'm getting "503 Service Temporarily Unavailable" from nginx/1.13.0
and if I put an empty string to VIRTUAL_HOST I'm getting a "502 Bad Gateway" from nginx/1.13.0.
Now, I'd like to avoid specifying virtual host in the taskdef file - is that possible? Is my problem ASP.NET Core related or nginx related?
Amazon ECS have a secret management system using Amazon S3. You have to create a secret in your ECS interface, and then you will be able to reference it in your configuration, as an environment variable.
{
"family": "...",
"networkMode": "bridge",
"containerDefinitions": [
{
"image": "...",
"name": "...",
"cpu": 128,
"memory": 256,
"essential": true,
"portMappings": [
{
"containerPort": 80,
"hostPort": 0,
"protocol": "http"
}
],
"environment": [
{
"name": "VIRTUAL_HOST",
"value": "SECRET_S3_VIRTUAL_HOST"
}
]
}
]
}
Store secrets on Amazon S3, and use AWS Identity and Access Management (IAM) roles to grant access to those stored secrets in ECS.
Full blog post
You could also make your own NGinx Docker image, which will already contain the environment variable.
FROM nginx
LABEL maintainer YOUR_EMAIL
ENV "VIRTUAL_HOST" "ec2-xx-xxx-xxxxxx.compute1.amazonaws.com"
And you would just have to build it, ship it privately and then use it for your configuration.