How to connect Kinesis Data Stream with Azure Function App as consumer, without using intermediary services?
You can use the Kinesis Client Library (KCL) and its documentation is here
You can also use Kinesis SDK (Python, Java or Nodejs) and use "Get Records" API to fetch records from the stream.
Get Records documentation is here
Related
I am trying to use AWS amplify analytics library to stream analytics data. I ran into issues with pinpoint not sending custom attributes and decided to switch to AWS Kinesis. Now I am having issues configuring a custom kinesis backend to amplify analytics library. This docs only provides documentation only for configuring custom Pinpoint backend. How do I configure custom Kinesis backend?
I want to expose a data streaming endpoint with 'Transfer-Encoding: chunked', a constantly open connection that streams data to a client.
My first though was to create something on Spring like here which works but we are using API Gateway and I found out that it does not support chunked transfer mode. Is there any other way to expose such an endpoint within the AWS ecosystem like Lambda or Kinesis or something else? Thanks in advance.
I deployed a ES cluster to AWS EKS and it can be access through application load balancer. I have created an index in the cluster. I am going to deploy a Kinesis firehose to use http_endpoint as the target to stream data to ES cluster.
But how can I configure what http verb firehose uses to put data to ES endpoint? For example, my ES endpoint is http://xxxx.com/myindex and I'd like the firehose to send POST request to this endpoint. How can I configure it to use POST rather than PUT?
Kinesis Firehose doesn't give you that much control over what the HTTP requests look like, unfortunately, since there are only a handful of configuration options for the HTTP Endpoint destination.
Kinesis sends HTTP requests according to their own specification, and expects responses to look in a specific way. You can find the specification here.
This specification is not compatible with Elasticsearch's Index API as you discovered. On a side note, you will probably want to use the Bulk API for performance reasons, since Kinesis will buffer events and deliver multiple ones in one request.
So, you will most likely need something in-between Kinesis and your Elasticsearch cluster to handle the differences in regards to the HTTP requests and responses. This could be a simply nginx proxy, for instance, or it could be something like an API Gateway which invokes a Lambda function. This Lambda function then sends a request to your index' Bulk API and constructs a response for Kinesis according to the HTTP specification.
I hope that helps.
I am building real-time dashboard using aws services, currently my application using MySQL database(RDS), which service and how would be designed real-time dashboard using Amazon web service, currently my approach is to use kinesis with redshift and connect my application via JDBC connector or use kinesis with s3 and use Athena to show real-time aggregators.
please help.
Thanks in advance.
Although the Amazon Kinesis Docs mention that Kinesis streams can be used to send updates to dashboards, Kinesis has no native mechanism to do this alone. For some very good security reasons it's unwise to allow clients (i.e. from a webpage you serve) to access backend services like Kinesis directly.
Instead, you'll want to set up your application layer (Java, in this case) to listen to the Kinesis streams and expose any relevant events to your client, storing any changes you need to keep track of in your database (RDS, in this case). For a real-time dashboard I'd recommend using something like WebSockets to send events in real time to your webpage from your server, as they're widely supported and easy to use. Heres a tutorial on how to implement WebSockets on GlassFish
Any API allow to send SQL query from my server to kinesis analytics? I can not find related API in official SDK document.
You specify a SQL statement when you create an application via the API.
See doc: http://docs.aws.amazon.com/kinesisanalytics/latest/dev/API_CreateApplication.html#API_CreateApplication_RequestSyntax
HTH