How to Create AWS Task Definition JSON from Existing task definition? - amazon-web-services

I have a an existing task definition 'my-task-definition' that I can get the data for using 'aws ecs describe-task-definition --task-definition my-task-definition' (I put the output of that into my_file.json'). But my understanding is that the output from 'aws ecs describe-task-definition --task-definition my-task-definition' is not valid input for 'aws ecs register-task-definition --cli-input-json file://<path_to_json_file>/my_file.json'. What additional piece(s) of data do I have to add to that file (or remove from it). The file (with the arns changed) is below:
{
"taskDefinition": {
"taskDefinitionArn": "arn:aws:ecs:us-west-1:112233445566:task-definition/my-task-definition:64",
"containerDefinitions": [
{
"name": "my-container",
"image": "123456789023.dkr.ecr.us-west-1.amazonaws.com/monolith-repo:latest",
"cpu": 0,
"memory": 1600,
"portMappings": [
{
"containerPort": 8080,
"hostPort": 0,
"protocol": "tcp"
}
],
"essential": true,
"environment": [
{
"name": "SERVER_FLAVOR",
"value": "JOB"
}
],
"mountPoints": [],
"volumesFrom": [],
"linuxParameters": {
"initProcessEnabled": true
},
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/my-task-definition",
"awslogs-region": "us-west-1",
"awslogs-stream-prefix": "ecs"
}
}
}
],
"family": "my-task-definition",
"taskRoleArn": "arn:aws:iam::111222333444:role/my_role",
"networkMode": "bridge",
"revision": 64,
"volumes": [],
"status": "ACTIVE",
"requiresAttributes": [
{
"name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
},
{
"name": "com.amazonaws.ecs.capability.ecr-auth"
},
{
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
},
{
"name": "com.amazonaws.ecs.capability.task-iam-role"
},
{
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.25"
}
],
"placementConstraints": [],
"compatibilities": [
"EXTERNAL",
"EC2"
],
"requiresCompatibilities": [
"EC2"
]
}
}

You are getting an error because the output from the aws ecs describe-task-definition command has additional fields that are not recognized by the aws ecs register-task-definition command.
There is no built in solution to be able to easily update a running Task Definition using the AWS CLI. However, it is possible to script a solution using a tool like jq.
One possible solution is something like this:
TASK_DEFINITION=$(aws ecs describe-task-definition --task-definition "$TASK_FAMILY" --region "us-east-1")
NEW_TASK_DEFINTIION=$(echo $TASK_DEFINITION | jq --arg IMAGE "$NEW_IMAGE" '.taskDefinition | .containerDefinitions[0].image = $IMAGE | del(.taskDefinitionArn) | del(.revision) | del(.status) | del(.requiresAttributes) | del(.compatibilities)')
aws ecs register-task-definition --region "us-east-1" --cli-input-json "$NEW_TASK_DEFINITION"
These commands update the docker image in an existing task definition and delete the extra fields so that you can register a new task definition.
There is an open Github Issue that is tracking this issue. https://github.com/aws/aws-cli/issues/3064

Related

AWS ECS/EC2 task fails to deploy

I'm trying to deploy a small React app to an EC2 instance using ECS but I'm constantly getting the error message Resource handler returned message: "Error occurred during operation 'ECS Deployment Circuit Breaker was triggered'." (RequestToken: xxx-xxx-xxx-xxx, HandlerErrorCode: GeneralServiceException)
I have tried deploying the app to an EC2 instance manually (with docker run -p 80:3000 my-app) and this worked fine so I'm pretty sure the app and Dockerfile are ok. The Docker image is hosted in ECR.
I'm thinking that most likely this is an issue with my task definition. It looks like this:
{
"taskDefinitionArn": "arn:aws:ecs:eu-north-1:xxx:task-definition/my-task-definition:4",
"containerDefinitions": [
{
"name": "my-app",
"image": "xxx.dkr.ecr.eu-north-1.amazonaws.com/my-app:latest",
"cpu": 1024,
"memory": 1024,
"portMappings": [
{
"name": "my-app-3000-tcp",
"containerPort": 3000,
"hostPort": 80,
"protocol": "tcp"
}
],
"essential": true,
"environment": [],
"mountPoints": [],
"volumesFrom": [],
"disableNetworking": false,
"privileged": false,
"readonlyRootFilesystem": false,
"pseudoTerminal": false
}
],
"family": "my-app",
"taskRoleArn": "arn:aws:iam::xxx:role/EcsToEcrAccess",
"executionRoleArn": "arn:aws:iam::xxx:role/ecsTaskExecutionRole",
"networkMode": "bridge",
"revision": 4,
"volumes": [],
"status": "ACTIVE",
"requiresAttributes": [
{
"name": "com.amazonaws.ecs.capability.ecr-auth"
},
{
"name": "com.amazonaws.ecs.capability.task-iam-role"
},
{
"name": "ecs.capability.execution-role-ecr-pull"
}
],
"placementConstraints": [],
"compatibilities": [
"EC2"
],
"requiresCompatibilities": [
"EC2"
],
"registeredAt": "2023-01-31T15:30:16.919Z",
"registeredBy": "arn:aws:iam::xxx:user/me",
"tags": [
{
"key": "ecs:taskDefinition:createdFrom",
"value": "ecs-console-v2"
}
]
}

AWS service can't start task, but starting task manually works

Until now I had a backend running single tasks. I now want to switch to services starting my tasks. For two of the tasks I need direct access to them so I tried using ServiceConnect.
When I run this task standalone it starts. When I start a service without ServiceConnect with the same task inside it also starts. When I enable ServiceConnect I get this error message inside of the 'Deployments and events' tab in the service:
service (...) was unable to place a task because no container instance met all of its requirements.
The closest matching container-instance (...) is missing an attribute required by your task.
For more information, see the Troubleshooting section of the Amazon ECS Developer Guide.
When I check the attributes of all free containers with:
ecs-cli check-attributes --task-def some-task-definition --container-instances ... --cluster some-cluster
I just get:
Container Instance Missing Attributes
heyvie-backend-dev None
My task definition looks like that:
{
"family": "some-task-definition",
"taskRoleArn": "arn:aws:iam::...:role/ecsTaskExecutionRole",
"executionRoleArn": "arn:aws:iam::...:role/ecsTaskExecutionRole",
"networkMode": "awsvpc",
"cpu": "1024",
"memory": "982",
"containerDefinitions": [
{
"name": "...",
"image": "...",
"essential": true,
"healthCheck": {
"command": ["..."],
"startPeriod": 20,
"retries": 3
},
"portMappings": [
{
"name": "somePortName",
"containerPort": 4321
}
],
"mountPoints": [
{
"sourceVolume": "...",
"containerPath": "..."
}
],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "...",
"awslogs-region": "eu-...",
"awslogs-stream-prefix": "..."
}
}
}
],
"volumes": [
{
"name": "...",
"efsVolumeConfiguration": {
"fileSystemId": "...",
"rootDirectory": "/",
"transitEncryption": "ENABLED"
}
}
],
"requiresCompatibilities": ["EC2"]
}
My service definition looks like that:
{
"cluster": "some-cluster",
"serviceName": "...",
"taskDefinition": "some-task-definition",
"desiredCount": 1,
"launchType": "EC2",
"deploymentConfiguration": {
"maximumPercent": 100,
"minimumHealthyPercent": 0
},
"placementConstraints": [
{
"type": "distinctInstance"
}
],
"networkConfiguration": {
"awsvpcConfiguration": {
"subnets": [
...
],
"securityGroups": ["..."],
"assignPublicIp": "DISABLED"
}
},
"serviceConnectConfiguration": {
"enabled": true,
"namespace": "someNamespace",
"services": [
{
"portName": "somePortName",
"clientAliases": [
{
"port": 4321
}
]
}
]
},
"schedulingStrategy": "REPLICA",
"enableECSManagedTags": true,
"propagateTags": "SERVICE"
}
I also added this to the user data of my launch template:
#!/bin/bash
cat <<'EOF' >> /etc/ecs/ecs.config
ECS_ENABLE_TASK_IAM_ROLE=true
ECS_CLUSTER=some-cluster
EOF
Did anyone experience something similiar or does know what could cause that issue?
I used ServiceDiscovery, I think, it's the easiest way to replace a dynamic ip address of a task in a service (on every restart the ip address changes and that's probably what you're trying to avoid?).
With ServiceDiscovery you are creating a new DNS record and instead of ip-address:port you can just use serviceNameOfNamespace.namespace. to connect to a task. ServiceDiscovery worked without any problem on an existing task.
Hope that helps, I don't really know if there are any benefits for ServiceConnect except for higher connection counts and retry functionalities, so if anybody knows more about differences between those I'm happy to learn.

Fargate oneoff task keeps running

I'm having an issue with a fargate one off task , it's meant to run database migration and then stop but it keeps stuck in running status
this is the task definition :
resource "aws_ecs_task_definition" "migrate" {
family = "${var.project_name}-${var.environment}-migrate"
network_mode = "awsvpc"
requires_compatibilities = ["FARGATE"]
cpu = 512
memory = 1024
execution_role_arn = aws_iam_role.ecs_task_execution_role.arn
task_role_arn = aws_iam_role.ecs_task_execution_role.arn
container_definitions = <<DEFINITION
[
{
"name": "${var.project_name}-migrate",
"image": "${var.repository_url}:latest",
"cpu": 512,
"memory": 1024,
"command": [
"/bin/sh",
"-c",
"python manage.py migrate --no-input"
],
"mountPoints": [],
"environment": [
{
"name": "DJANGO_SETTINGS_MODULE",
"value": "****"
},
{
"name": "DB_HOST",
"value": "****"
},
{
"name": "DD_API_KEY",
"value": "****"
}
],
"secrets": [
{
"name": "SECRETS",
"valueFrom": "*****"
}
],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "****",
"awslogs-region": "****",
"awslogs-stream-prefix": "******"
}
},
"volumesFrom": []
}
]
DEFINITION
}
and this is how i call it from github actions
aws ecs run-task --launch-type FARGATE --cluster cs-name --task-definition $MIGRATE_TASK_ARN --network-configuration "awsvpcConfiguration={subnets=[${{ secrets.MIGRATE_TASK_SUBNET_IDA }}, ${{ secrets.MIGRATE_TASK_SUBNET_IDB }}],securityGroups=${{ secrets.MIGRATE_TASK_SECURITY_GROUP_ID }}}"
any idea what's wrong ?
I guess it depends what the command does. When the main process in the container exits the containers stops and the task will stop. One way to check the behavior would be to run something like ls (or similar) and see what happens. I am wondering if the problem is due to the fact you are calling the shell and then the python program and when the program exits the shell keeps the container alive? Have you tried just running the python program?
"command": "python manage.py migrate --no-input",

AWS CodePipeline Fails: "Exception while trying to read the task definition artifact filef rom: SourceArtifact"

I have an AWS CodePipeline setup that is meant to pull from CodeCommit, use CodeBuild, and then do a Blue/Green deployment via CodeDeploy.
I believe it to be configured correctly (will discuss specifics below), but every time I get to the "Deploy" stage, I get the error message:
Invalid action configuration: Exception while trying to read the task definition artifact file from: SourceArtifact
I've looked through other SO answers, and I've checked the following:
SourceArtifact is well under 3MB in size.
The files taskdef.json and appspec.yml are both inside the SourceArtifact (these are the names as configured in my CodePipeline definition) which is generated in the first stage of the CodePipeline.
The artifact is able to be decrypted via KMS key as the CodePipeline is configured to make use of one (since SourceArtifact comes from a different account) and the CodeBuild step is able to successfully complete (it creates a Docker image and saves to ECR).
I can see no syntax errors of any kind in taskdef.json or appspec.yml as they're essentially copies from working versions of those files from different projects. The placeholder names remain the same.
Checking CodeTrail and checking list-action-executions (via CLI) don't show any additional error information.
Here's the "Deploy" stage config (as entered via a Terraform script):
stage {
name = "Deploy"
action {
name = "Deploy"
category = "Deploy"
owner = "AWS"
provider = "CodeDeployToECS"
version = "1"
input_artifacts = ["SourceArtifact", var.charting_artifact_name]
configuration = {
ApplicationName = aws_codedeploy_app.charting_codedeploy.name
DeploymentGroupName = aws_codedeploy_app.charting_codedeploy.name
TaskDefinitionTemplateArtifact = "SourceArtifact"
AppSpecTemplateArtifact = "SourceArtifact"
TaskDefinitionTemplatePath = "taskdef.json"
AppSpecTemplatePath = "appspec.yml"
Image1ArtifactName = var.charting_artifact_name
Image1ContainerName = "IMAGE1_NAME"
}
}
}
taskdef.json (account numbers redacted):
{
"executionRoleArn": "arn:aws:iam::<ACCOUNT_NUM>:role/fargate-iam-role",
"containerDefinitions": [
{
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/sentiment-logs",
"awslogs-region": "us-east-1",
"awslogs-stream-prefix": "sentiment-charting"
}
},
"portMappings": [
{
"hostPort": 80,
"protocol": "tcp",
"containerPort": 80
}
],
"cpu": 0,
"environment": [],
"mountPoints": [],
"volumesFrom": [],
"image": "<IMAGE1_NAME>",
"name": "sentiment-charting"
}
],
"placementConstraints": [],
"memory": "4096",
"compatibilities": [
"EC2",
"FARGATE"
],
"taskDefinitionArn": "arn:aws:ecs:us-east-1:<ACCOUNT_NUM>:task-definition/sentiment-charting-taskdef:4",
"family": "sentiment-charting-taskdef",
"requiresAttributes": [
{
"name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
},
{
"name": "ecs.capability.execution-role-awslogs"
},
{
"name": "com.amazonaws.ecs.capability.ecr-auth"
},
{
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
},
{
"name": "ecs.capability.execution-role-ecr-pull"
},
{
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
},
{
"name": "ecs.capability.task-eni"
}
],
"requiresCompatibilities": [
"FARGATE"
],
"networkMode": "awsvpc",
"cpu": "2048",
"revision": 4,
"status": "ACTIVE",
"volumes": []
}
appspec.yml:
version: 0.0
Resources:
- TargetService:
Type: AWS::ECS::Service
Properties:
TaskDefinition: "<TASK_DEFINITION>"
LoadBalancerInfo:
ContainerName: "sentiment-charting"
ContainerPort: 80
PlatformVersion: "LATEST"
I'm at a bit of a loss as to how best to continue troubleshooting without spinning my wheels. Any help would be greatly appreciated.
TIA

Github actions - pass secret variables to render ECS task definition action

In order to deploy new task to ECS im using amazon-ecs-render-task-definition GitHub action.
This action receives a task-definition.json as a parameter. This JSON contain secrets that i dont want to push, is there a way to inject some parameter to this JSON? Maybe from aws secrets manager?
For example - task-definition.json
{
"containerDefinitions": [
{
"name": "wordpress",
"links": [
"mysql"
],
"image": "wordpress",
"essential": true,
"portMappings": [
{
"containerPort": 80,
"hostPort": 80
}
],
"memory": 500,
"cpu": 10
},
{
"environment": [
{
"name": "MYSQL_ROOT_PASSWORD",
"value": ****"password"**** // ITS A SECRET!
}
],
"name": "mysql",
"image": "mysql",
"cpu": 10,
"memory": 500,
"essential": true
}],
"family": "hello_world" }
Apparently there is a build in solution for using aws-scrent-manager secrets:
"secrets": [
{
"name": "DATABASE_PASSWORD",
"valueFrom": "arn:aws:ssm:us-east-1:awsExampleAccountID:parameter/awsExampleParameter"
}
]
https://aws.amazon.com/premiumsupport/knowledge-center/ecs-data-security-container-task/
Another solution is to use sed to insert your secrets
So your workflow becomes something like -
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- name: Add secrets to Task Definition
run: |
sed -i "s/<jwt_secret>/$JWT_SECRET/g" task.json
sed -i "s/<mongo_password>/$MONGO_PASSWORD/g" task.json
env:
JWT_SECRET: ${{secrets.JWT_SECRET}}
MONGO_PASSWORD: ${{secrets.MONGO_PASSWORD}}
Then you edit your to task.json to include the placeholders that sed will use for the replacement
{
"ipcMode": null,
"executionRoleArn": null,
"containerDefinitions": [
{
...
"environment": [
{
"name": "JWT_SECRET",
"value": "<jwt_secret>"
},
{
"name": "MONGO_PASSWORD",
"value": "<mongo_password>"
},
]
...
}
]
}
All repos have a place to store their secrets, see creating and using encrypted secrets. As for editing .json, preinstalled jq looks like an obvious choice here, or maybe powershell if you're more familiar with that (just remember about tweaking -Depth).