What is the best way to monitor stacks on AWS? - amazon-web-services

Question

I am currently working on projects with Laravel/PHP that I manage with AWS.
I deploy several instances thanks to the CloudFormation service. So i have a lot of LogsGroups.
If I have an error in these LogsGroups, I have to manually search for the error logs.
So, my need is :
When I have an error API or PH message in one of my LogsGroups, like this:
[2021-11-24T13:03:48.075879+00:00] technical.ERROR: TYPE: Trzproject\Trzutils\Exceptions\InvalidJWTException
MESSAGE: Provided JWT has since expired, as defined by the "exp" claim
FILE: /var/task/vendor/trzproject/trzcore/src/Trzutils/JWT/JWTService.php LINE: 64 TRACE: stack trace disabled ____________________________________________________________________
I want to be alerted with a message (slack, mail, whatever) that tells me in which LogGroup is the error.
I can’t create a LogInsight query because I have a lot of clients and, LogInsight does not allow to make query on a very large amount of LogsGroups.
Thank you in advance for your advice.
Edit : something like this ; https://theithollow.com/2017/12/11/use-amazon-cloudwatch-logs-metric-filters-send-alerts/
but without create x alarms for x logGroups
(sorry for my english)

Answer 1

I finally did this and it's a solution to my needs
I created a Lambda that is triggered by CloudWatch Logs. I’ve combined the log groups I want to monitor with CloudWatch Logs as well as a pattern to retrieve only certain messages.
serverless.yml
functions:
persistLogMessage:
handler: lambda/PersistLogMessage.php
timeout: 899 # 14min 59s
events:
- cloudwatchLog:
logGroup: 'MyLogsGroup01'
filter: '?ERROR ?WARN ?5xx'
- cloudwatchLog:
logGroup: 'MyLogsGroup02'
filter: '?ERROR ?WARN ?5xx'
...
layers:
- arn:aws:lambda:#{AWS::Region}:#{AWS::AccountId}:layer:php-73:1
role: PersistLogMessageRole
...
resources:
Conditions:
Resources:
#
PersistLogMessageRole:
Type: AWS::IAM::Role
Properties:
RoleName: ${opt:stage}-${opt:client}-PersistLogMessageRole
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action: sts:AssumeRole
Policies:
- PolicyName: PersistLogMessagePolicy
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: "Allow"
Action:
- logs:CreateLogGroup
- logs:CreateLogStream
- logs:PutLogEvents
- logs:PutRetentionPolicy
- logs:DescribeLogStreams
- logs:DescribeLogGroups
Resource: "*"
- Effect: Allow
Action:
- sns:Publish
Resource:
- "arn:aws:sns:eu-west-3:<account_id>:SnsTopic"
My Lambda:
public function __invoke(array $events): void
{
$data = $events['awslogs']['data'];
$this->logger->info('events', $events);
$dataDecoded = base64_decode($data);
$logMessage = zlib_decode($dataDecoded);
/** #var stdClass $stdClassLogMessage */
$stdClassLogMessage = json_decode($logMessage);
dump($stdClassLogMessage);
$params = [
'Message' => $stdClassLogMessage->logEvents[0]->message,
'TopicArn' => 'arn:aws:sns:eu-west-3:<account_id>:SnsTopic'
];
$this->snsClient->publish($params);
}