Fargate task stops about 10s after is starts with no log output - amazon-web-services

My Fargate task keeps stopping after it's started and doesn't output any logs (awslog driver is selected).
The container does start up and stay running when i execute docker locally.
Docker-compose file:
version: '2'
services:
asterisk:
build: .
container_name: asterisk
restart: always
ports:
- 10000-10099:10000-10099/udp
- 5060:5060/udp
Dockerfile:
FROM debian:10.7
RUN {stuff-that-works-is-here}
# Keep Asterisk running in the foreground
ENTRYPOINT ["asterisk", "-f"]
# SIP port
EXPOSE 5060:5060/udp
# RTP ports
EXPOSE 10000-10099:10000-10099/udp
my task execution role has full cloudwatch access for debugging.

Click on the ECS task instance, expand the container section, the error should be shown there. I have attached a screen shot of it. Here is a screenshotScrenshot

The AWS log driver alone is not enough.
Unfortunately, Fargate doesn't create the log group for you unless you tell it to
See Creating a log group at https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_awslogs.html

I had a similar problem, and the cause was the Health Check.
ECS dont have Health Check for UDP, so when you open a UDP port if you use Docker for the deploy (docker compose), it create a Health Check pointing to a TCP port, and since there was no open TCP ports for that range, the container reset itself due to Health Check.
I had to add a custom Resource to docker-compose:
x-aws-cloudformation:
Resources:
AsteriskUDP5060TargetGroup:
Type: "AWS::ElasticLoadBalancingV2::TargetGroup"
Properties:
HealthCheckProtocol: TCP
HealthCheckPort: 8088
Basically I have a Health Check for a UDP port pointing to a TCP port. Its a "hack" to bypass this problem when the deploy is made with Docker.

Related

How to address another container in the same task definition in AWS ECS on Fargate?

I have an MQTT application which consists of a broker and multiple clients. The broker and each client run in their own container. Locally I am using Docker compose to set up my application:
services:
broker:
image: mqtt-broker:latest
container_name: broker
ports:
- "1883:1883"
networks:
- engine-net
db:
image: database-client:latest
container_name: vehicle-engine-db
networks:
- engine-net
restart: on-failure
networks:
engine-net:
external: false
name: engine-net
The application inside my clients is written in C++ and uses the Paho library. I use the async_client to connect to the broker. It takes two arguments, namely:
mqtt::async_client cli(server_address, client_id);
Hereby, server_address is the IP of the broker + port, and the client_id is the "name" of the client that is connecting. While using the compose file, I can simply use the service name given in the file to address the other containers in the network (here "broker:1883" does the trick). My containers work, and now I want to deploy to AWS Fargate.
In the task definition, I add my containers and give a name to them (the same names like the services in the Docker compose file. However, the client does not seem to be able to connect to the broker, as the deployment fails. I am quite sure that it cannot connect because it cannot resolve the broker IP.
AWS Fargate uses network mode awsvpc which - to my understanding - puts all containers of a task into the same VPC subnet. Therefore, automatic name resolution like in Docker compose would make sense to me.
Has anybody encountered the same problem? How can I resolve it?
Per the documentation, containers in the same Fargate task can address each other on 127.0.0.1 at the container's respective ports.

Why does AWS ECS allows inbound traffic to ALL ports by default?

I am deploying the following relatively simple docker-compose.yml file on AWS ECS via the Docker CLI.
It uses tomcat server image which can be also replaced by any other container which does not exits of startup.
services:
tomcat:
image: tomcat:9.0
command: catalina.sh run
ports:
- target: 8080
published: 8080
x-aws-protocol: http
Commands used
docker context use mycontextforecs
docker compose up
The cluster, services, task, target, security groups and application load balancer are automatically created as expected.
But, the security group created by AWS ECS allows inbound traffic on ALL ports by default instead of only the exposed 8080.
Following is a screenshot of the security group, which also has a comment -
"tomcat:8080/ on default network"
But port range is "All" instead of 8080
I've read the following and some other stackoverflow links but could not get an answer.
https://docs.docker.com/cloud/ecs-compose-features/
https://docs.docker.com/cloud/ecs-architecture/
https://docs.docker.com/cloud/ecs-integration/
I understand that the default "Fargate" instance type gets a public ip assigned.
But why does ECS allow traffic on all ports?
If I add another service in the docker-compose file, the default security group gets shared between both of them.
As a result, anyone can telnet into the port exposed by the service due to this security group rule.

How to manage start/stop of docker-compose services when instance is not used

How to manage start/stop of docker-compose services when AWS instance is not used
We use docker-compose services during 9AM - 6PM, but would like to stop during the rest of the day
How to start/stop services when they are not needed
On a modern Linux system you can use systemd socket activation to start a service on-demand. For example you could put this in my-docker-service.socket.
[Socket]
ListenStream=443 # listen on HTTPS port both on ipv4 and ipv6
Accept=no # Don't spawn a new instance for each incoming connection
[Install]
WantedBy=sockets.target
[Service]
ExecStart=/usr/sbin/docker-compose # the service to start with arguments
User=docker
Group=docker
You can do it by adding restart policy in the docker-compose.yml file, so whenever you start the server, the docker containers will restart themselves.
add line restart: always
webserver:
image: xxxxx
user: user-name (Optional)
restart: always
....
Ref: https://docs.docker.com/compose/compose-file/compose-file-v3/
I have always used this policy and all the containers come up after server reboots.

a service in ecs fails to find another service within the same network

I have a very simple docker-compose for locust (python package for load testing). It starts a 'master' service and a 'slave' service. Everything works perfectly locally but when I deploy it to AWS ECS a 'slave' can't find a master.
services:
my-master:
image: chapkovski/locust
ports:
- "80:80"
env_file:
- .env
environment:
LOCUST_MODE: master
my-slave:
image: chapkovski/locust
env_file:
- .env
environment:
LOCUST_MODE: slave
LOCUST_MASTER_HOST: my-master
So apparently I need to refer from my-slave service not to my-master when I am on ECS. What's wrong here?
Everything works perfectly locally but when I deploy it to AWS ECS
a 'slave' can't find a master.
I assume that slave needs to access master both must be in the same task definition to access like this or you can explore service discovery?
"links": [
"master"
]
links
Type: string array
Required: no
The link parameter allows containers to communicate with each other
without the need for port mappings. Only supported if the network mode
of a task definition is set to bridge. The name:internalName construct
is analogous to name:alias in Docker links.
Note
This parameter is not supported for Windows containers or tasks using the awsvpc network mode.
Important
Containers that are collocated on a single container instance may be
able to communicate with each other without requiring links or host
port mappings. Network isolation is achieved on the container instance
using security groups and VPC settings.
"links": ["name:internalName", ...]
container_definition_network

Kafka on AWS ECS, how to handle advertised.host without known instance?

I'm trying to get Kafka running on an AWS ECS container. I have this setup already / working fine on my local docker environment, using the spotify/kafka image
To get this working locally, I needed to ensure the ADVERTISED_HOST environment variable was set. ADVERTISED_HOST needed to be set as the containers external IP, otherwise when I try to connect it was just giving me connection refused.
My local docker-compose.yaml has this for the kafka container:
kafka:
image: spotify/kafka
hostname: kafka
environment:
- ADVERTISED_HOST=192.168.0.70
- ADVERTISED_PORT=9092
ports:
- "9092:9092"
- "2181:2181"
restart: always
Now the problem is, I don't know what the IP is going to be, as I dont know which instance this will run on. So how do I set that environment variable?
Your entrypoint script will need to call the EC2 Metadata Service on startup (in this case http://169.254.169.254/latest/meta-data/local-hostname) to get the external-to-docker hostname and set that variable.
Sample:
[ec2-user ~]$ curl http://169.254.169.254/latest/meta-data/local-hostname
ip-10-251-50-12.ec2.internal