Cloudwatch agent not sending logs to cloudwatch - amazon-web-services

I am trying to send two lots of logs up to CloudWatch.
Here are the two logs:
/var/log/apache2/access.log
/var/log/apache2/error.log
I used the amazon-cloudwatch-agent-config-wizard to create the config file, and here is a snippet of the file showing the correct file path:
"collect_list": [
{
"file_path": "/var/log/apache2/access.log",
"log_group_name": "*group_name*",
"log_stream_name": "apache-access"
},
{
"file_path": "/var/log/apache2/error.log",
"log_group_name": "group-name*",
"log_stream_name": "apache-error"
}
]
I loaded in the config with:
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json -s
And ran into no errors and no errors and showing in the amazon-cloudwatch-agent.log.
Checking the status of the Amazon CloudWatch Agent shows it is running and has not errors. Also states the schema is valid.
The unique part of all of this is that I have removed the old CloudWatch agent and installed the new one. I have done this on two EC2 instances, one of which everything is working perfectly on, and the other one is not sending the logs to CloudWatch.
In a nutshell, why aren't the logs going up to CloudWatch? What can I do to troubleshoot this?
Any help will be appreciated.

So the problem turned out to be permission-based. The CloudWatch config wizard defaults to using cwagent as the user that runs CloudWatch, this is also reiterated in official guides.
Changing the running using to root resolved the issue even though the files in question all had 777 permissions at the time of trying to get it running.
The config file you edit is:
sudo nano /opt/aws/amazon-cloudwatch-agent/bin/config.json
At the top of the file you will see:
"agent": {
"metrics_collection_interval": 60,
"run_as_user": "cwagent"
},
You need to change run_as_user to root, like:
"agent": {
"metrics_collection_interval": 60,
"run_as_user": "root"
},
Once you have changed that, you simply reload the config file:
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json -s
And then restart the service:
sudo systemctl restart amazon-cloudwatch-agent.service
You should then see the logs coming into CloudWatch. Expect some backfilling.

Check
the CloudWatch Agent log- /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log.
Here you should see some lines similar to - Reading from /var/log/apache2/access.log for both the files.
the agent toml file to make sure both the files path are configured and check the region as well-
/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml.

Related

What does "Error" mean in Systems Manager Run Command (Document: AmazonCloudWatch-ManageAgent)?

I basically followed this web page. https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/install-CloudWatch-Agent-on-EC2-Instance-fleet.html
Steps
Launch an EC2 instance (Amazon Linux 2) with an IAM role (Permissions: CloudWatchAgentServerRole, AmazonSSMManagedInstanceCore).
See "Download the CloudWatch agent package" section in the documentation and run "AWS-ConfigureAWSPackage".
Go to Systems Manager Parameter Store and create a parameter.
Name: AmazonCloudWatch-linux
Parameter: see below
{
"metrics": {
"append_dimensions": {
"ImageId": "${!aws:ImageId}",
"InstanceId": "${!aws:InstanceId}",
"InstanceType": "${!aws:InstanceType}"
},
"metrics_collected": {
"mem": {
"measurement": [
"mem_used_percent"
],
"metrics_collection_interval": 60
}
}
}
}
See "Start the CloudWatch agent" section in the documentation and run "AmazonCloudWatch-ManageAgent". I input "AmazonCloudWatch-linux" to the "Optional Configuration Location" box.
To check the status of CloudWatch Agent, I run sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a status, and this below is returned. This means CloudWatch Agent is running successfully.
{
"status": "running",
"starttime": "2022-07-20T15:06:12+0000",
"configstatus": "configured",
"cwoc_status": "running",
"cwoc_starttime": "2022-07-20T15:06:11+0000",
"cwoc_configstatus": "configured",
"version": "1.247353.0b251941"
}
I also go to CloudWatch Metrics and confirm I get a new metric.
However, the execution history of "AmazonCloudWatch-ManageAgent" (Step 4) has some messages in "Error".
Created symlink from /etc/systemd/system/multi-user.target.wants/cwagent-otel-collector.service to /etc/systemd/system/cwagent-otel-collector.service.
Redirecting to /bin/systemctl restart cwagent-otel-collector.service
2022/07/20 15:06:12 D! [EC2] Found active network interface
2022/07/20 15:06:12 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/ssm_AmazonCloudWatch-linux.tmp ...
2022/07/20 15:06:12 I! Valid Json input schema.
2022/07/20 15:06:12 D! [EC2] Found active network interface
Created symlink from /etc/systemd/system/multi-user.target.wants/amazon-cloudwatch-agent.service to /etc/systemd/system/amazon-cloudwatch-agent.service.
Redirecting to /bin/systemctl restart amazon-cloudwatch-agent.service
I also check "Output" of the execution history. In my understanding, this does not show any issues.
****** processing cwagent-otel-collector ******
Successfully fetched the config and saved in /opt/aws/amazon-cloudwatch-agent/cwagent-otel-collector/etc/cwagent-otel-collector.d/default.tmp
cwagent-otel-collector config has been successfully fetched.
cwagent-otel-collector has already been stopped
****** processing amazon-cloudwatch-agent ******
/opt/aws/amazon-cloudwatch-agent/bin/config-downloader --output-dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d --download-source ssm:AmazonCloudWatch-linux --mode ec2 --config /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml --multi-config default
I! Trying to detect region from ec2
Region: ap-northeast-1
credsConfig: map[]
Successfully fetched the config and saved in /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/ssm_AmazonCloudWatch-linux.tmp
Start configuration validation...
/opt/aws/amazon-cloudwatch-agent/bin/config-translator --input /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json --input-dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d --output /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml --mode ec2 --config /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml --multi-config default
I! Detecting run_as_user...
I! Trying to detect region from ec2
No csm configuration found.
No log configuration found.
Configuration validation first phase succeeded
/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent -schematest -config /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml
Configuration validation second phase succeeded
Configuration validation succeeded
amazon-cloudwatch-agent has already been stopped
Question
Even though there is no problem at Step 5, 6 and 8, why is there a message in Error at Step 7?

Docker port mapping on AWS Elastic Beanstalk

I have a Docker container specified through Dockerfile which has to be run with the -p flag in order to function properly. On my local machine, after running
docker run -d -p 5000:5000 blagtagger:v0.4.3
the port mapping is shown as 0.0.0.0:5000->5000/tcp and everything works smoothly.
Now, I need to deploy it on AWS Elastic Beanstalk, so I prepared my Dockerrun.aws.json file as follows:
{
"AWSEBDockerrunVersion": "1",
"Ports": [
{
"ContainerPort": 5000,
"HostPort": 5000
}
]
}
However, the container port does not seem to map to the host port correctly. The port mapping is shown as 5000/tcp.
How can I ask Beanstalk to set the mapping to 0.0.0.0:5000->5000/tcp?
It turns out there is no elegant way to accomplish this. Along the lines of the answers here and here, I added the following in a file named .ebextensions/01-commands.config:
commands:
00001_add_port_mapping:
cwd: /tmp
command: 'sed -i "s/docker run -d/docker run -p 5000:5000 -d/" /opt/elasticbeanstalk/hooks/appdeploy/enact/00run.sh'
and discarded the Dockerrun.aws.json file.

Unable to access jarfile - Docker on Elastic Beanstalk

I am trying to deploy a Docker image of a Spring Boot application to AWS Elastic Beanstalk and I'm encountering this error in /var/log/eb-activity.log:
Docker container quit unexpectedly after launch: Docker container quit unexpectedly on Wed Jun 22 11:56:25 UTC 2016:
Error: Unable to access jarfile /home/packedit/app/packed-it.jar. Check snapshot logs for details. (Executor::NonZeroExitStatus)
This is a single container on Elastic Beanstalk with the following Dockerrun.aws.json:
{
"AWSEBDockerrunVersion": "1",
"Image": {
"Name": "packedit/packedit-api",
"Update": "true"
},
"Ports": [
{
"ContainerPort": "8080"
}
],
"Volumes": [
{
"HostDirectory": "/var/app/packedit",
"ContainerDirectory": "/home/packedit/app"
}
],
"Logging": "/home/packedit/app/logs"
}
This is the Dockerfile:
FROM java:8
MAINTAINER my#email.com
VOLUME /tmp
EXPOSE 8080
ENV USER_NAME packedit
ENV APP_HOME /home/$USER_NAME/app
ENV APP_FILENAME packed-it.jar
RUN useradd -ms /bin/bash $USER_NAME
RUN mkdir -p $APP_HOME/data
ADD $APP_FILENAME $APP_HOME/$APP_FILENAME
RUN chown -R $USER_NAME $APP_HOME/
USER $USER_NAME
WORKDIR $APP_HOME
RUN bash -c 'touch $APP_FILENAME'
# Can't use $APP_FILENAME here because ENTRYPOINT does not do ENV replacement
# See: http://stackoverflow.com/a/28854410/336752
ENTRYPOINT ["java","-Djava.security.egd=file:/dev/./urandom","-jar","packed-it.jar"]
I have successfully deployed the Docker image to an EC2 instance using ECS but I have not succeeded with Elastic Beanstalk. My guess is that I am doing something wrong with the volumes but I am struggling to understand the documentation. I originally started with a multicontainer configuration but have simplified to try and isolate my issue.
Thanks for any advice.
You need to remove the line
"ContainerDirectory": "/home/packedit/app"
from your Dockerrun.aws.json.
It seems like the confusion is with how docker volumes work. The volumes are allocated at runtime and persist on consecutive runs on the same machine.
Here is what is happening. The docker image is built with jar in /home/packedit/app but since you have defined a volume in the same location, an empty volume is created when it is run and mounted in that location. Hence, the same directory of the image is ignored.
Here is how you can reproduce the issue locally:
docker build .
docker run -v /home/packedit/app IMAGEID_FROM_OUTPUT_OF_PREVIOUS_COMMAND

Elastic Beanstalk with Docker deployment failure

I'm trying to deploy my application with docker and elastic beanstalk. My Dockerrun.aws.json file looks like
{
"AWSEBDockerrunVersion": "1",
"Image": {
"Name": "jvans/maven_weekly",
"Update": "true"
},
"Ports": [
{
"ContainerPort": "5000"
}],
"Volumes": [
{
"HostDirectory": "/Users/jamesvanneman/Code/maven_weekly/maven_weekly",
"ContainerDirectory": "/maven_weekly"
}
],
"Logging": "/var/log/nginx"
}
I created this application with eb create and when I run eb deploy I get
Docker container quit unexpectedly after launch: Docker container quit
unexpectedly on Mon Sep 21 01:15:12 UTC 2015:. Check snapshot logs for details.
Hook /opt/elasticbeanstalk/hooks/appdeploy/enact/00run.sh failed. For more detail, check /var/log/eb-activity.log using console or EB CLI.
In var/log/eb-activity.log I see the following errors:
Docker container quit unexpectedly after launch: Docker container quit unexpectedly on Mon Sep 21 01:08:52 UTC 2015:. Check snapshot logs for details. (ElasticBeanstalk::ExternalInvocationError)
caused by: 83ea9b7f9a069eeb8351fef7aaedb8374f7dfe300a5e0aaeba0fe17600583175
[2015-09-21T01:08:52.205Z] INFO [2246] - [Application deployment/StartupStage1/AppDeployEnactHook/00run.sh] : Activity failed.
So it seems like there's a problem with a startup script. If i ssh into the container and try to run it manually I don't really get any extra help from error messages.
eb ssh
sudo /opt/elasticbeanstalk/hooks/appdeploy/enact/00run.sh
Docker container quit unexpectedly after launch: Docker container quit unexpectedly on Mon Sep 21 01:34:52 UTC 2015:. Check snapshot logs for details.
Msg: Docker container quit unexpectedly after launch: Docker container quit unexpectedly on Mon Sep 21 01:34:52 UTC 2015:. Check snapshot logs for details.
Are snapshot logs something different than what's in var/log/eb-activity.log? Any Idea what is going on/how I can debug this further?
Docker dumps are stored in the host box at /var/log/eb-docker/containers/.
Go there and you'll find the docker startup crash log that should indicate the root cause of your problem.
You want to look at
/var/log/eb-docker/containers/eb-current-app/unexpected-quit.log
in the bundle downloaded by eb logs --all or using eb ssh. This log file will have the stdout and stderr of your application before it crashed.
In my case, unexpected-quit.log was empty and the other log files in /var/log/ didn't help.
So I found out why the container was crashing by manually running the image on the server.
[ec2-user#ip-xxx-xxx-xxx-xxx ~]$ nano Dockerfile # or clone the entire project
[ec2-user#ip-xxx-xxx-xxx-xxx ~]$ sudo docker build -t test .
[ec2-user#ip-xxx-xxx-xxx-xxx ~]$ sudo docker run --rm test
Exception in thread "main" …
I fixed that exception, redeployed a new version and now the container starts without issues.
In my case I did an eb ssh to connect into the ec2 instance and looked at my server logs. It was an application error. Nothing to do with AWS. Fixed it. Tried again and it's working!

Error deploying Play Framework on AWS Beanstalk Docker

Im running a Play Framework app on AWS Beanstalk with Docker (64bit Amazon Linux 2015.03 v1.4.1 running Docker 1.6.0).
Docker File:
FROM relateiq/oracle-java8
MAINTAINER XXXX
EXPOSE 9000
ADD files /
WORKDIR /opt/docker
RUN ["chown", "-R", "daemon", "."]
RUN ["chmod", "+x", "bin/app"]
USER daemon
ENTRYPOINT ["bin/app"]
CMD []
Dockerrun.aws.json
{
"AWSEBDockerrunVersion": "1",
"Ports": [{
"ContainerPort": "9000"
}]
}
When the instance first starts I get about 1 minute where its deployed as normal, then after I browse a few pages the error shows:
502 Bad Gateway
nginx/1.6.2
The error in the ElasticBeanstalk logs is:
Play server process ID is 1 This application is already running (Or delete /opt/docker/RUNNING_PID file).
I also get in the /var/log/docker-events.logthe following messages every 30 seconds:
2015-05-30T20:07:58.000000000Z d0425e47095e5e2637263a0fe9b49ed759f130f31c041368ea48ce3d99d1e947: (from aws_beanstalk/current-app:latest) start
2015-05-30T20:08:15.000000000Z d0425e47095e5e2637263a0fe9b49ed759f130f31c041368ea48ce3d99d1e947: (from aws_beanstalk/current-app:latest) die
2015-05-30T20:08:16.000000000Z d0425e47095e5e2637263a0fe9b49ed759f130f31c041368ea48ce3d99d1e947: (from aws_beanstalk/current-app:latest) start
2015-05-30T20:08:31.000000000Z d0425e47095e5e2637263a0fe9b49ed759f130f31c041368ea48ce3d99d1e947: (from aws_beanstalk/current-app:latest) die
Can anyone see my issue? Cheers.
Adding the following to build.sbt should resolve the issue:
javaOptions in Universal ++= Seq("-Dpidfile.path=/dev/null")