How I can send or export logs from my AWS Lambda function to S3 without using by CloudWatch?
Is there another option?
Yes, you can stop your cloudwatch log ingesting from function and send the log to another destination.
Please check- Disabling logging to CloudWatch Logs
This AWS Sample shows you have to use a Lambda extension to send logs to S3. But there is currently no way to stop logs from going to CloudWatch.
Related
I use a lot StackDriver sinks to BigQuery. It helps on keeping the Data and have them in a convenient queryable form.
I am searching for the equivalent on AWS using CloudWatch but it seems only S3 is integrated. Any workarounds or should I code it?
This can be done with a workaround. It needs only one code component reading from S3 and posting to Stackdriver, then GCP sinks can pickup.
Create a CloudWatch Sink to S3 bucket.
The S3 bucket will be integrated to send SNS messages on object Creation.
A process should listen to those SNS notifications and copy the contents of the bucket into StackDriver.
Since the guarantees of SNS and Cloudwatch to S3 are there, a Lambda function on the SNS notification messages can make the solution more seamless.
I have a question about streaming CloudFront logs to CloudWatch. I know that it is possible to save CloudFront logs to S3 bucket and later, I can use lambda function which can be triggered by S3 to send logs to CloudWatch. There is also a possibility to look at streamed logs using Amazon Kinesis Data Streams. However I want to know that if some CloudFront configurations exist to send the logs to CloudWatch. Helpful answers are appreciated. Thanks in advance!
I created a custom app that automatically uploads logs to s3.
Is there a way to push those logs to cloudwatch from s3 for analysis and alerting?
I'm aware that I can use a cloudwatch agent to push directly to cloudwatch from the app but there are complications involved in that option.
Thank you!
You could probably use Cloudwatch Events to listen to S3 changes. Not sure about if you can get the data from the S3 file, or just a trigger saying that a new log has been added.
You could also use S3 event notifications (https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html) connected either to a lambda or SQS, and from there create the logs to Cloudwatch. (similar to what was suggested by #marcin)
A better solution, but one that is a bit beyond the scope of the question, would be to send your logs through Kinesis Firehose and from there add the Cloudwatch and S3 logs.
I'm not aware of any out-of-the-box mechanism for that provided by AWS. But I think it could be relatively easy to develop.
Namely, you can create S3 notification for a PUT of a new log file from your app to S3. The event would trigger a lambda function. The function would get the file and using AWS SDK, e.g. boto3's put_log_events, it would send the log events to CloudWatch logs.
I have to read and process a file in an AWS Lambda function from an SFTP server that is not on AWS.
Some external source is putting the file in the SFTP server which is not in AWS, and whenever the file is uploaded completely, we have to check it via AWS CloudWatch and then trigger an AWS Lambda for processing this file.
Is this approach right? Is this even possible?
If this is possible, please suggest some steps. I checked in AWS CloudWatch but I was not able to found any trigger which checks the file outside the AWS.
You need to create some sort of job that will monitor your SFTP directory (e.g., using inotify) and then invoke your AWS Lambda function by using AWS access keys of an IAM user created with programmatic access enabled and sufficient permissions to invoke that AWS Lambda function.
You can also create an AWS CloudWatch event that will be triggered on scheduled basis like every 5 mins that will trigger the AWS Lambda function to check for any news file by maintaining a history somewhere for example on AWS DynamoDB but I would rather trigger the AWS Lambda from SFTP server as using AWS based file-upload on SFTP detection sounds better if AWS Transfer for SFTP is used instead of on-premises SFTP server because it uses AWS S3 as an SFTP store and AWS S3 as the feature of creating an event on files/objects upload and trigger an AWS Lambda function.
Can you modify the external source script ?
If yes, you can send a SNS notification to specific topic using the aws cli or a specific language sdk.
Then you can have a lambda to process your file, triggered by the SNS topic.
I've setup some aws cloudwatch agents to send logs to a cloudwatch log group and what I want to do now is get those into an elasticsearch cluster (on ec2). Ideally, what I want to do is write a lambda function that parses it and then sends it to elasticsearch, but also have the lambda function be triggered anytime a message is written to the log group. Is it possible to trigger lambda from a cloudwatch log group?
Subscribing AWS Lambda functions to CloudWatch Logs is documented here. Also, streaming CloudWatch Logs to the AWS ElasticSearch service is documented here.
Here you can find the documentation about adding a subscription to CloudWatch logs for Lambda.
For streaming to AWS ElasticSearch see this (as mentioned before).