I know its is possible to stream CloudWatch Logs Data to Amazon Elasticsearch Service. It is documented here, But is it possible to stream the logs data to a custom AWS Glue Job, or to an EMR Job?
The way streaming of CloudWatch Logs (CWLs) to ElasticSearch (ES) works, is that AWS creates a lambda function for you. So CWLs will stream to lambda first, and the lambda will then upload the log events to ES.
For Glue, you don't need lambda function, as glue can get its streaming data from kinesis streams. So you would have to setup CWL subscription to kinesis stream. The stream would be used as a source in a streaming Glue job.
For EMR you could also just stream log events from CWL to kinesis in the same way as for Glue. But to read the stream data in EMR, you would probably have to use EMR Input Connector for Kinesis.
Related
I have been Stocked on how to send Lambda logs(Prints) directly to Amazon Kinesis Data Stream. I have Found the way to send Logs from Cloud watch but I would like to send every single prints to kinesis data streams. I have a doubt if I send data from cloud watch does it stream real time prints records to kinesis or not? On this case I would like to use lambda as producer and through the kinesis data S3 as a consumer .
below I have attached a flow work of my conditions.
You can also check the lambda extensions, which helps into direct ingestion of the logs to custom destinations. Its helpful incase you want to avoid cloudwatch costs
https://aws.amazon.com/blogs/compute/using-aws-lambda-extensions-to-send-logs-to-custom-destinations/
You have to create CouldWatch Subscription filter for the Lambda's log stream you want to save to S3. So you would do:
CW Logs subscription ---> Firehose ---> S3
I have events that keep coming which I need to put to S3. I am trying to evaluate if I muse use Kinesis Stream or Firehose. I also want to wait for few minutes before writing to S3 so that the object is fairly full.
Based on my reading of Kinesis Data stream, I have to create an analytics app which will then be used to invoke a lambda. I will then have to use the lambda to write to S3. Or Kinesis Data Streams can directly write to lambda somehow? I could not find anything indicating the same.
Firehose is not charged by hour(while stream is). So is firehose a better option for me?
Or Kinesis Data Streams can directly write to lambda somehow?
Data Streams can't write directly to S3. Instead Firehose can do this:
delivering real-time streaming data to destinations such as Amazon Simple Storage Service (Amazon S3), Amazon Redshift, Amazon Elasticsearch Service (Amazon ES), Splunk, and any custom HTTP endpoint or HTTP endpoints owned by supported third-party service providers, including Datadog, MongoDB, and New Relic.
What's more Firehose allows you to buffer the records before writing them to S3. The writing can happen based on buffer size or time. In addition to that you can process the records using lambda function before writing to S3.
Thus, colectively it seems that Firehose is more suited to your use-case then Data Streams.
I just created a new AWS kinesis stream and have it set up to receive data I am pulling in a lambda function. Theres is some data in my cloud watch logs that came in before I had my stream set up. Is there a way to get manually send this to kinesis?
There are ways as shown in this answer to export CloudWatch logs as JSON/txt/etc. afterwhich you could design some sort of script to iterate through those logs and put them into your Kinesis stream
Firehose is fully managed whereas Streams is manually managed.
If other people are aware of other major differences, please add them. I'm just learning.
Thanks..
Amazon Kinesis Data Firehose can send data to:
Amazon S3
Amazon Redshift
Amazon Elasticsearch Service
Splunk
To do the same thing with Amazon Kinesis Data Streams, you would need to write an application that consumes data from the stream and then connects to the destination to store data.
So, think of Firehose as a pre-configured streaming application with a few specific options. Anything outside of those options would require you to write your own code.
I am using AWS Kinesis Stream that is containing customer transactions. I want to backup the transactions into S3 before start processing them. How can I use bare Kinesis Firehose to backup the transactions from Kinesis Stream without running any Lambda or other computing component for it?
You can reverse the order of your Kinesis building blocks:
Instead of writing into Kinesis Stream, write into Kinesis Firehose that is directed to S3.
Run Kinesis Analytics (KA) application to read the events from your Kinesis Firehose (KF), and write them to a Kinesis Stream (KS). You can use the functionality of KA to do some of the filtering, aggregation and joins that you would otherwise run in your code (Lambda or KCL).