I have trouble showing log from my docker container in ecs.
What I did:
1) ssh into an ec2 instance of the cluster.
2) docker logs my service
Then this message is showing up:
FATA[0000] Error executing 'logs': Failed to get log configuration: Container 'my-container': Must specify log driver as awslogs
What I am trying to do is to show log in the console.
What I don't understand is that for some container, the command docker logs works fine.
open docker daemon file in /etc/docker/daemon.json and add log driver:
{
"log-driver": "awslogs"
}
and restart docker with sudo systemctl restart docker
You should choose the docker log driver to json-log while creating the task definition revision in ECS, If you want to see the docker logs with docker logs container-id command. you will get the container id from the docker ps command.
But if you want to push docker logs to cloudwatch logs then you have to choose aws-logs as docker log driver.
For some containers, it might work fine because those have docker log driver as json-file set in their task definition.
How to create a task definition?
Reference: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/create-task-definition.html
Do let me know if you are still facing the issue.
Related
I have a AWS instance with Docker installed on it. And some containers are running.I have setup one Laravel project inside docker.
I can access this web application through AWS IP address as well as DNS address(GoDaddy).
I have also designed gitlab CI/CO to publish the code to AWS instance.
When I try to push the code through Gitlab pipelines, I am getting following error in pipeline.
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
I checked the docker, it is running properly. Any clues please.
.gitlab-ci.yml
http://pastie.org/p/7ELo6wJEbFoKaz7jcmJdDp
the pipeline failing at deploy-api-staging: -> script -> scripts/ci/build
build script
http://pastie.org/p/1iQLZs5GqP2m5jthB4YCbh
deploy script
http://pastie.org/p/2ho6ElfN2iWRcIZJjQGdmy
From what I see, you have directly installed and registered the GitLab runner on your EC2 instance.
I think the problem is that you haven't already given permissions to your GitLab Runner user to use Docker.
From the official Docker documentation:
The Docker daemon binds to a Unix socket instead of a TCP port. By default that Unix socket is owned by the user root and other users can only access it using sudo. The Docker daemon always runs as the root user.
If you don’t want to preface the docker command with sudo, create a Unix group called docker and add users to it. When the Docker daemon starts, it creates a Unix socket accessible by members of the docker group.
Well, GitLab Runners use the user gitlab-runner by default when they're running any CI/CD Pipeline and that user won't use sudo (neither it should be in the sudoers file!) so we have to correctly configure it.
First of all, create a Docker group on the EC2 where the GitLan Runner is registered:
sudo groupadd docker
Then, we are going to add the user gitlab-runner to that group:
sudo usermod -aG docker gitlab-runner
And we are going to verify that the gitlab-runner user actually has access to Docker:
sudo -u gitlab-runner -H docker info
Now your Pipelines should be able to access without any problem to the Unix socket under unix:///var/run/docker.sock.
Additional Steps if using Docker Runners
If you're using the Docker executor in your runner, you have to now mount that Unix socket on the Docker image you're using.
[[runners]]
url = "https://gitlab.com/"
token = REGISTRATION_TOKEN
executor = "docker"
[runners.docker]
tls_verify = false
image = "docker:19.03.12"
privileged = false
disable_cache = false
volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
Take special care of the contents in the volume clause.
I would like to launch an interactive shell into a public Docker image on my AWS ECS/Fargate cluster to run network/connectivity tests from inside the cluster.
It seems the official way to do this is with aws ecs run-task followed by aws ecs execute-command [1][2]
I'd like to use existing, public Docker Hub images rather than build custom images if possible.
If I run do run-task with no command or the default command, the task exits and execute-command won't work on an exited task.
"Essential container in task exited"
If I set a Docker command of sleep 10000, I get:
"CannotStartContainerError: ResourceInitializationError: failed to create new container runtime task: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: \"sleep 10000\": executable file not found in $PATH: unknown",
Ideally, run-task and execute-command would be combined in one step. I don't want a background task running indefinitely, I want a shell to run a few commands interactively, that is cleaned up when I'm finished. How would I achieve this?
[1] https://aws.amazon.com/blogs/containers/new-using-amazon-ecs-exec-access-your-containers-fargate-ec2/
[2] https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-exec.html
I had the same issue. I was finally able to get a container to sit "idle" with the following command inside the Task Definition:
"tail", "-F", "/dev/null"
Then I could connect in with an interactive execute-command.
I am using terraform to build infrastructure on AWS provider. I am using ECR to push my local docker images using AWSCLI.
Now, I have a Application load balancer which would route traffic to ECS_service. I want ECS to manage my Docker Containers using Fargate. But, the docker containers are exited by saying "Essential Docker container exited".
Thats the only log printed out.
If i change the docker image to be nginx:latest(which is fetched from dockerhub). It works.
PS: My docker container is a simple node application with node:alpine as base image. Is it something related to this, i am wrong !
Can anyone provide me with some insight on what is wrong with my approach.
I get the following error in AWS Logs:
docker-standard-init-linux-go211-exec-user-process-caused-exec-format-error.
My Dockerfile
FROM node:alpine
WORKDIR /app
COPY . .
RUN npm install
# Expose a port.
EXPOSE 8080
# Run the node server.
ENTRYPOINT ["npm", "start"]
They say, its issue with the start script. I am just running this command. npm start to start the server.
It’s not your approach, your image is just not working.
Try running it locally and see the output otherwise you will need to ship the logs to Cloudwatch and see what they say
Is there easy way to run an ECS Task attached or to follow the logs only while the container is Running (ie. Detach after displaying all of the logs associated)?
Using the AWS CLI (1.17.0) and ecs-cli (1.21.0), I have gotten decently close with the following two commands:
aws ecs run-task --cluster "mycluster" --task-definition testhelloworldjob --launch-type FARGATE --network-configuration etc.etc.etc.
ecs-cli logs --task-id {TASK_ID_HERE_FROM_OUTPUT_OF_PREVIOUS_COMMAND} --follow
I am currently have two issues with the above approach:
There is a race condition being that the logs are not available when the task is in a pre "running" state. Instead of ecs-cli logs waiting for the logs to exist, there is an error immediately thrown.
Even after waiting for the task to be in a running state, and issuing the ecs-cli logs the command refuses to detach even AFTER the task is finished and in a Post Running status.
For the first issue I could poll until there is a post activating/pending status, prior to calling logs. For the second issue I could draft some type of threaded call that would poll to stop the following of a log after the container in question is no longer running.... But there has to be an easier way?
To clarify I am coming from numerous other container orchestration tools/technologies that seemingly supported this very seamlessly. Here are some examples of tools and their associated commands that would yield me my intended results:
Docker CLI:
docker run hello-world
Docker-Compose Yaml:
docker-compose up
K8 Kubectl Yaml:
kubectl apply -f ./hello-k8.yaml && kubectl logs --follow hello-world
I think ecs-cli is the best option available at the moment.
Apart from that, you can change the logs driver of the AWS ECS task to syslog and then watch the logs file from the terminal after doing SSH into the EC2 container instance in which it is running.
Another thing you can do is SSH into the EC2 container instance in which it was running before and then run the container of that AWS ECS task by yourself in it using docker run and once the testing is done, you can stop and remove that container and then get that task started via AWS ECS.
Note: You can use AWS SSM Session Manager in order to avoid using EC2 key pair and adding an inbound rule for SSH.
I have a few docker containers running with docker-compose on an AWS EC2 instance. I am looking to get the logs sent to AWS CloudWatch. I was also having issues getting the logs from docker containers to AWS CloudWatch from my Mac running Sierra so I've moved over to EC2 instances running Amazon AMI.
My docker-compose file:
version: '2'
services:
scraper:
build: ./Scraper/
logging:
driver: "awslogs"
options:
awslogs-region: "eu-west-1"
awslogs-group: "permission-logs"
awslogs-stream: "stream"
volumes:
- ./Scraper/spiders:/spiders
When I run docker-compose up I get the following error:
scraper_1 | WARNING: no logs are available with the 'awslogs' log driver
but the container is running. No logs appear on the AWS CloudWatch stream. I have assigned an IAM role to the EC2 container that the docker-containers run on.
I am at a complete loss now as to what I should be doing and would apprecaite any advice.
The awslogs works without using ECS.
you need to configure the AWS credentials (the user should have IAM roles appropriate [cloudwatch logs]).
I used this tutorial, it worked for me: https://wdullaer.com/blog/2016/02/28/pass-credentials-to-the-awslogs-docker-logging-driver-on-ubuntu/
I was getting the same error but when I checked the cloudwatch logs, I was able to see the logs in cloudwatch. Did you check that if you have the logs group created in cloudwatch. Docker doesn't support console logging when we use the custom logging drivers.
The section on limitations here says that docker logs command is only available for json-file and journald drivers, and that's true for built-in drivers.
When trying to get logs from a driver that doesn't support reading, nothing hangs for me, docker logs prints this:
Error response from daemon: configured logging driver does not support reading
There are 3 main steps involved it to it.
Create an IAM role/User
Install CloudAgent
Modify docker-compose file or docker run command
I have referred an article here with steps to send the docker logs to aws cloudwatch.
The AWS logs driver you are using awslogs is for use with EC2 Container Service (ECS). It will not work on plain EC2. See documentation.
I would recommend creating a single node ECS cluster. Be sure the EC2 instance(s) in that cluster have a role, and the role provides permissions to write to Cloudwatch logs.
From there anything in your container that logs to stdout will be captured by the awslogs driver and streamed to Cloudwatch logs.