Elastic Beanstalk not streaming to CloudWatch - amazon-web-services

Question

I've had an Elastic Beanstalk instance streaming my logs to Cloud Watch for about a year. This week the logs stopped streaming. This may have been because I 'rebuilt' the environment in Beanstalk. No configuration changes were made at the same time.
I've double checked that my Beanstalk role has the correct permissions in IAM (it has CloudWatchFullAccess).
I also tried deleting all of my existing group logs. I then went into the Beanstalk 'Instance log streaming to CloudWatch Logs' area, changed my log retention period and restarted the App Server. Sure enough my log groups were recreated (with the new retention period), so I'm pretty sure the permissions look OK. Despite this, no log messages are appearing in the log groups.
I have requested the recent logs though Beanstalk and I can see messages are being written to the logs on the App Server OK.
My platform is Tomcat 8 with Java 8 running on 64bit Amazon Linux/2.6.2
I'm not sure where to go from here. I have no error messages to work off, or any good ideas for what to check next.
Edit: Here is my custom config for CloudWatch, as defined here
files:
"/etc/awslogs/config/company_log.conf" :
mode: "000600"
owner: root
group: root
content: |
[/var/log/tomcat8/company.log]
log_group_name = `{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "var/log/tomcat8/company.log"]]}`
log_stream_name = {instance_id}
file = /var/log/tomcat8/company.*

Answer 1

It's probably because it's not allowed to. Make sure the role used by the EC2 instances is allowed to access:
logs:CreateLogGroup
logs:CreateLogStream
logs:GetLogEvents
logs:PutLogEvents
logs:DescribeLogGroups
logs:DescribeLogStreams
logs:PutRetentionPolicy
With Elastic Beanstalk the IAM role used by the EC2 instances is probably aws-elasticbeanstalk-ec2-role. Give that role access to a new policy: ec2-cloudwatch-logs-stream (if someone knows a better name let me know) using JSON (as suggested here):
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:GetLogEvents",
"logs:PutLogEvents",
"logs:DescribeLogGroups",
"logs:DescribeLogStreams",
"logs:PutRetentionPolicy"
],
"Resource": [
"*"
]
}
]
}
And it should work after restarting the awslogs service with sudo services awslogs restart. If not; check the logs. You can find it here: /var/log/awslogs.log

Answer 2

You might have upgraded your AWS AMI platform, the Tomcat location is therefore different (e.g. /var/log/tomcat instead of /var/log/tomcat8).
I've detailed in a new Medium blog how this all works and an example .ebextensions file and where to put it.
Below is an excerpt that you might be able to use, though the article explains how to determine the right folder/file(s) to stream.
packages:
yum:
awslogs: []
option_settings:
- namespace: aws:elasticbeanstalk:cloudwatch:logs
option_name: StreamLogs
value: true
- namespace: aws:elasticbeanstalk:cloudwatch:logs
option_name: DeleteOnTerminate
value: false
- namespace: aws:elasticbeanstalk:cloudwatch:logs
option_name: RetentionInDays
value: 90
files:
"/etc/awslogs/awscli.conf" :
mode: "000600"
owner: root
group: root
content: |
[plugins]
cwlogs = cwlogs
[default]
region = `{"Ref":"AWS::Region"}`
"/etc/awslogs/config/logs.conf" :
mode: "000600"
owner: root
group: root
content: |
[/var/log/tomcat/localhost.log]
log_group_name = `{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "var/log/tomcat/localhost.log"]]}`
log_stream_name = {instance_id}
file = /var/log/tomcat/localhost.*
[/var/log/tomcat/catalina.log]
log_group_name = `{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "var/log/tomcat/catalina.log"]]}`
log_stream_name = {instance_id}
file = /var/log/tomcat/catalina.*
[/var/log/tomcat/localhost_access_log.txt]
log_group_name = `{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "var/log/tomcat/access_log"]]}`
log_stream_name = {instance_id}
file = /var/log/tomcat/access_log.*
commands:
"01":
command: systemctl enable awslogsd.service
"02":
command: systemctl restart awslogsd