How to make ALB slow_start work during ECS update service - amazon-web-services

Question

For my high-traffic containerized app running in ECS Fargate, slow ramp-up is required for new containers, to avoid out of memory situation immediately after startup. This is especially important during the update service operation when all the containers are replaced at the same time.
How can I get this to work with ECS Fargate and ALB, making sure the old containers stay around until the slow_start period for the new containers is over?
This is my current terraform setup. I enabled slow_start, but during update service the old containers are stopped too early, so that the new containers get full traffic instantly.
resource "aws_alb_target_group" "my_target_group" {
name = "my_service"
port = 8080
protocol = "HTTP"
vpc_id = data.aws_vpc.active.id
target_type = "ip"
slow_start = 120
health_check {
enabled = true
port = 8080
path = "/healthCheck"
unhealthy_threshold = 2
healthy_threshold = 2
}
}
resource "aws_ecs_service" "my_service" {
name = "my_service"
cluster = aws_ecs_cluster.my_services.id
task_definition = aws_ecs_task_definition.my_services.arn
launch_type = "FARGATE"
desired_count = var.desired_count
deployment_maximum_percent = 400
deployment_minimum_healthy_percent = 100
enable_execute_command = true
wait_for_steady_state = true
network_configuration {
subnets = data.aws_subnets.private.ids
security_groups = [aws_security_group.my_service_container.id]
}
load_balancer {
container_name = "my-service"
container_port = 8080
target_group_arn = aws_alb_target_group.my_target_group.arn
}
lifecycle {
create_before_destroy = true
ignore_changes = [desired_count]
}
}

Answer 1

aws ecs usually sends sigterm to gracefully shutdown and sends sigkill if 30 second passed.
So, you can handle this sigterm signal (example to catch this signal in python) and add delay in your code. After that, you need to adjust sigkill 30 second wait with stopTimeout in ContainerDefinition to stop aws shutting down the ecs.