How to logging in Amazon Web Service ( AWS )? - amazon-web-services

Question

I have a project built in Golang and deployed on a Docker instance in AWS.
Internally I create a log file where the program write several logs.
How can I access that log file?
Is there another correct way to logging?
Thanks

Answer 1

You could mount the log file from your container to your EC2 host. You can do this by using the -v flag when running your container:
docker run -v /var/log/my_host_log_file.log:/var/log/your_container_log_file.log your-image
Alternatively, you can configure your app to log to stdout and use syslog as your log driver (using the --log-driver=syslog switch). Your container logs will then be written to /var/log/messages on your host.

Answer 2

If you use AWS, i would suggest to send Logs direct to AWS CloudWatch.
First create a new Log-Group in AWS Cloudwatch, for example "Production". In your Docker-Compose.yml (or via docker run..) add the AWS Logdriver:
logging:
driver: "awslogs"
options:
awslogs-region: "eu-central-1"
awslogs-group: "Production"
awslogs-stream: "MyApp"
Next creat a IAM user with Access to AWS Cloudwatch and add to the Dockerhost the credentials.
Example IAM Policy:
"Version" "2012-10-17"
"Statement"
"Action" "logs:CreateLogStream" "logs:PutLogEvents" "Effect" "Allow" "Resource"
On Ubuntu with systemd:
"Version" "2012-10-17"
"Statement"
"Action"
"logs:CreateLogStream"
"logs:PutLogEvents"
"Effect"
"Allow" "Resource"
And add to the File:
[Service] Environment"AWS_ACCESS_KEY_ID=<aws_access_key_id>"
Environment"AWS_SECRET_ACCESS_KEY=<aws_secret_access_key>"
Run:
systemctl daemon-reload
service docker restart
Now your logs should appear in AWS Cloudwatch.

Answer 3

Thanks for reply.
After a while looking for the solution to the problem, I found it!
Firstly, I needed to mount the file that is inside the instance in the docker-host.
To do this I add a Json file in the root folder of my project called Dockerrun.aws.json
( http://docs.aws.amazon.com/es_es/elasticbeanstalk/latest/dg/create_deploy_docker_image.html#create_deploy_docker_image_dockerrun )
That is the file that declares the shared folder (volumes) (beetwen docker-host and instance) where I save my log file . This line is equivalent to adding -v flag in the docker run command (https://docs.docker.com/engine/tutorials/dockervolumes/#mount-a-host-directory-as-data-volume). I do this this way because I can not add mount to a running instance and i cant stop it by ssh.
{
"AWSEBDockerrunVersion": "1",
"Volumes": [
{
"HostDirectory": "/var/log/",
"ContainerDirectory": "/go/src/app/log"
}
]
}
Then to tell aws that I want to download my log file when I request records. (Tail (last 100 lines), bundle or rotate) I add these files to the .ebextension folder in my project directory. ( http://docs.aws.amazon.com/en_us/elasticbeanstalk/latest/dg/using-features.logging.html#health-logs-extend )
Log_bundle.conf
Files:
"/opt/elasticbeanstalk/tasks/bundlelogs.d/log_bundle.conf":
Mode: "000755"
Owner: root
Group: root
Content: |
/var/log/application.log
Log_rotate.config
Files:
"/opt/elasticbeanstalk/tasks/bundlelogs.d/log_rotate.conf":
Mode: "000755"
Owner: root
Group: root
Content: |
/var/log/application.log
Log_tail.config
Files:
"/opt/elasticbeanstalk/tasks/publishlogs.d/log_tail.conf":
Mode: "000755"
Owner: root
Group: root
Content: |
/var/log/application.log
Finally, I dont try Amazon Could Watch but is the next step.
Regards

Answer 4

If you use ELK (Elasticsearch, Logstash, Kibana), I would suggest to use "logrus"
Get the library
go get github.com/sirupsen/logrus
Then in your project
package main
import (
logrus "github.com/sirupsen/logrus"
)
var log = logrus.New()
func main() {
conn, _ := net.Dial("tcp", "logstash-address")
hook := logrustash.New(conn, logrustash.DefaultFormatter(logrus.Fields{"type": "my-app"}))
log.Hooks.Add(hook)
log.Info("Hello World!")
}