Using lambda's for ECS auto deployment - amazon-web-services

Question

I am about at my wits end with this one. I am currently working on setting up an auto deployment for a aws application. The application is running inside of ECS without a load balancer.
The idea here is that when a new docker image get pushed to ECR, cloudTrail picks up the log, cloudWatch triggers and alarm, said alarm goes to SNS, and SNS triggers a lambda to deploy. So far I have everything working except the lambda.
The code I am using is in python and is as follows:
import boto3
import pprint
import os
region = "us-east-2"
client = boto3.client('ecs', region_name=region)
response = client.list_task_definitions(familyPrefix= 'stage-dcs-task-000', status='ACTIVE')
def lambda_handler(event, context):
response = client.register_task_definition(
family='stage-dcs-task-000',
networkMode='bridge',
containerDefinitions=[
{
'name': 'dcs-stage-container-000',
'image': 'ecrUrlHere:latest',
'cpu': 10,
'memory': 500,
'portMappings': [
{
'containerPort': 8080,
'hostPort': 80,
'protocol': 'tcp'
},
],
'essential': True,
},
],
)
taskDefinitionRev = response['taskDefinition']['family'] + ':' + str(response['taskDefinition']['revision'])
response = client.update_service(
cluster='stage-secretName-conversation-service',
service='stage-dcs-service-000',
desiredCount=1,
taskDefinition=taskDefinitionRev,
deploymentConfiguration={
'maximumPercent': 200,
'minimumHealthyPercent': 100
},
forceNewDeployment=True
)
Now, this lambda fails with the error
"errorMessage": "An error occurred (InvalidParameterException) when calling the UpdateService operation: The container stage-dcs-container-000is not valid. The value for 'containerName' must already be specified in the task definition. Registry: ARN-LINK-HERE"
Looking at what gets created in AWS, the task definition was created correctly, and the container was named and configured correctly. Based on what I understand from the docs, my call to update-service is correct, I just have no idea why the container name would be invalid.
Something interesting to note, is I though the container names might have to be unique. So I started add the task revision number to the end of the container name in place of the 000, however, I always get back stage-dcs-container-000 back as the invalid name and have no idea where it might be coming from. Any ideas here?
After doing some more debugging I have figured out that it is the container name within the current task definition that is throwing the error. Based on container name rules, I cannot have dashes, so I changed them to underscores, but the same issues still persists.
resources: this is the tutorial I have been following https://medium.com/#YadavPrakshi/automate-zero-downtime-deployment-with-amazon-ecs-and-lambda-c4e49953273d

Answer 1

It took a little while for me to figure this one out, but I eventually got it solved. Truth be told I am not 100% sure what the fix was, but I know there were at least two errors.
The first was the fact that I had somehow managed to fall into a state where I had illegal characters in the name of different parts of the stack. To give an explicit example, I have dashes (-) in the name of the containers, which is an illegal character. To solve this is just ended up recreating the whole tech stack using camel casing.
The second was I had some error in the lambda function, specifically in the creation of the task revision. What I ended up noticing what that when I created things via the cli I somehow missed the docker requirement. How I got around this was I created the task revision in the AWS consul and then just copied the json into to register_task_definition method inside the lambda. This is what I started with that DID NOT WORK.
response = client.register_task_definition(
family='stage-dcs-task-000',
networkMode='bridge',
containerDefinitions=[
{
'name': 'dcs-stage-container-000',
'image': 'ecrUrlHere:latest',
'cpu': 10,
'memory': 500,
'portMappings': [
{
'containerPort': 8080,
'hostPort': 80,
'protocol': 'tcp'
},
],
'essential': True,
},
],
)
this is what I finished with that DOES WORK.
response = client.register_task_definition(
containerDefinitions=[
{
"portMappings": [
{
"hostPort": 80,
"protocol": "tcp",
"containerPort": 8080
}
],
"cpu": 10,
"environment": [],
"mountPoints": [],
"memoryReservation": 500,
"volumesFrom": [],
"image": "image uri here",
"essential": True,
"name": "productionDCSContainer"
}
],
family="productionDCSTask",
requiresCompatibilities=[
"EC2"
],
networkMode="bridge"
)
It should be noted that it is easy to fall into a state where you dead lock on the revision update, that is to say you have something like minPercentage = 100, maxPercentage = 100 and desired = 1. In this case, AWS will never kill the running instance to start the update. A way around this is have minInstance = 0, maxInstance = 200% and have forceNewDeployment = True.