awslogs agent can't keep up - amazon-web-services

Question

I'm running the awslogs agent on a server, and when I look in CloudWatch logs in the AWS console, the logs are about 60 minutes behind. Our server produces about 650MB of data per hour, and it appears that the agent is not able to keep up.
Here is our abbreviated config file:
[application.log]
datetime_format = %Y-%m-%d %H:%M:%S
time_zone = UTC
file = var/output/logs/application.json.log*
log_stream_name = {hostname}
initial_position = start_of_file
log_group_name = ApplicationLog
[service_log]
datetime_format = %Y-%m-%dT%H:%M:%S
time_zone = UTC
file = var/output/logs/service.json.log*
log_stream_name = {hostname}
initial_position = start_of_file
log_group_name = ServiceLog
Is there a common way to speed of the awslogs agent?

Answer 1

The amount of data (> 0.2MB/s) is not an issue for the agent. The agent has a capacity of about 3MB/s per log file. However, if you're using the same log stream for multiple log files, the agents write to the same stream, and end up blocking each other. The throughput more than halves when you share a stream between log files.
Also, there are a few properties that can be configured that may have an impact on performance:
buffer_duration = <integer>
batch_count = <integer>
batch_size = <integer>
To solve my issue, I did two things:
Drastically increase the batch size (defaults to 32768 bytes)
Use a different log stream for each log file
And the agent had no problems keeping up. Here's my final config file:
[application.log]
datetime_format = %Y-%m-%d %H:%M:%S
time_zone = UTC
file = var/output/logs/application.json.log*
log_stream_name = {hostname}-app
initial_position = start_of_file
log_group_name = ApplicationLog
batch_size = 524288
[service_log]
datetime_format = %Y-%m-%dT%H:%M:%S
time_zone = UTC
file = var/output/logs/service.json.log*
log_stream_name = {hostname}-service
initial_position = start_of_file
log_group_name = ServiceLog
batch_size = 524288

Answer 2

awslogs agent supports log rotation
so this:
file = var/output/logs/application.json.log*
would pick up too many files?
Try:
file = var/output/logs/application.json.log
to speed up the process.