Upload multi-lined JSON log to AWS CloudWatch Log - amazon-web-services

Question

The put-log-events expect the JSON file need to wrap by a [ & ]
e.g.
# aws logs put-log-events --log-group-name my-logs --log-stream-name 20150601 --log-events file://events
[
{
"timestamp": long,
"message": "string"
}
...
]
However, my JSON file is in multi-lined format like
{"timestamp": xxx, "message": "xxx"}
{"timestamp": yyy, "message": "yyy"}
Is it possible to upload without writing my own program?
[1] https://docs.aws.amazon.com/cli/latest/reference/logs/put-log-events.html#examples

Answer 1

An easy way to handle publish the batch without any coding would be by using jq to do the necessary transformation in the file. jq is a command line utility to do the JSON processing.
cat events | jq -s '.'> events-formatted.json
aws logs put-log-events --log-group-name my-logs --log-stream-name 20150601 --log-events file://events-formatted.json
With this the data should be formatted and could be ingested to CloudWatch.

Answer 2

If you want to keep those lines as a single event, you can cast the lines to string, join them with \n and send them that way.
Since lines look like self sufficient json themselves, sending them as an array of events (hence [...]) might not be that bad, since they will get into same log group and will be easy to find as a batch.

Answer 3

You will need to escape it as suggested, and remove the new lines. Even though there is allot of JSON these days used as the consumer format, it isn't a great raw representation when it comes to logs. Reason being is that logs can get truncated.
Try parsing truncated JSON, no fun at all!
You also don't want to have timestamp embedded in your logs either, this will break the filter and search ability that you get with cloudwatch.
You can stream a RAW format to cloudwatch logs, and then use streams to parse that raw data, format it, filter it or whatever you want to do, into a service such as Elastic Search. I would recommend streaming to Elastic Search service on AWS if you are wanting to do more with your logs than what cloudwatch gives you, and you can do your embedded timestamp format as well if you so wish.