We are evaluating real-time event processing engines (like twitter storm)
One of the options is recently released Amazon Kinesis.
I'm wondering if there is any sort of emulator/sandbox environment available that would allow to play around with kinesis a bit without the need to set up AWS account and paying for the use of the service
Thank you in advance
I've been using kinesalite for a while now and it seems to work quite well.
Since I don't really like to have node packages into my machine, I've also created a docker image for it.
Hope this helps.
EMR recently launched Kinesis integration for tools in Hadoop ecosystem, you may want to try and see if your usecases can be addressed there. http://aws.typepad.com/aws/2014/02/process-streaming-data-with-kinesis-and-elastic-mapreduce.html
Related
My company is currently looking to build a power dialer using AWS Amazon Connect but I haven't been able to find support for agent extensions or voicemail. I'm, I going crazy? Someone please point me in the right direction.
We faced similar challenges and after several attempts discovered https://amazonconnectvoicemail.com . It was easy to integrate into our current Amazon Connect instance.
AWS has recently released Voicemail for Amazon Connect, which is deployed using a CloudFormation template.
You only pay for the standard AWS charges that the solution uses, there's no additional platform/service charges.
https://aws.amazon.com/solutions/voicemail-for-amazon-connect/
I'm executing a Flink Job with this tools.
I think both can do exactly the same with the proper configuration. Does Kinesis Data Analytics do something that EMR can not do or vice versa?
Amazon Kinesis Data Analytics is the easiest way to analyze streaming data, gain actionable insights, and respond to your business and customer needs in real time.
Amazon Elastic Map Reduce provides a managed Hadoop framework that makes it easy, fast, and cost-effective to process vast amounts of data across dynamically scalable Amazon EC2 instances. You can also run other popular distributed frameworks such as Apache Spark, HBase, Presto, and Flink in EMR.
The major difference is maintainability and management from your side.
If you want more independent management and more control then I would say go for AWS EMR. Where its your responsibility to manage the EMR infrastructure as well as the Apache Flink cluster in it.
But if you want less control and more focus on application development and you need to deliver faster(tight deadline) then KDA is the way to go. Here AWS provides all the bells and whistles you need for running your application. This also easily sets up with AWS s3 as code source and provides a bare minimum Configuration Management using the UI.
It scales automatically as well.(Need to understand KCU though).
It provides the same Flink dashboard where you can monitor your application and AWS Cloudwatch integration for debugging your application.
Please go through this nice presentation and let me know it that helps.
Please let me know.
https://www.youtube.com/watch?v=c_LswkrwOvk
I will say one major difference between the two is that Kinesis does not provide a hosted Hadoop service unlike Elastic MapReduce (now EMR)
Got this same question also. This video was helpful in explaining with a real architecture scenario and AWS explanation here tries to explain how Kinesis and EMR can fit together with possible use cases.
I'm trying to figure out what AWS services I need for the mobile application I'm working on with my startup. The application we're working on should go into the app-/play-store later this year, so we need a "best-practice" solution for our case. It must be high scaleable so if there are thousands of requests to the server it should remain stable and fast. Also we maybe want to deploy a website on it.
Actually we are using Uberspace (link) servers with an Node.js application and MongoDB running on it. Everything works fine, but for the release version we want to go with AWS. What we need is something we can run Node.js / MongoDB (or something similar to MongoDB) on and something to store images like profile pictures that can be requested by the user.
I have already read some informations about AWS on their website but that didn't help a lot. There are so many services and we don't know which of these fit our needs perfectly.
A friend told me to just use AWS EC2 for the Node.js server + MongoDB and S3 to store images, but on some websites I have read that it is better to use this architecture:
We would be glad if there is someone who can share his/her knowledge with us!
To run code: you can use lambda, but be careful: the benefit you
don't have to worry about server, the downside is lambda sometimes
unreasonably slow. If you need it really fast then you need it on EC2
with auto-scaling. If you tune it up properly it works like a charm.
To store data: DynamoDB if you want it really fast (single digits
milliseconds regardless of load and DB size) and according to best
practices. It REQUIRES proper schema or will cost you a fortune,
otherwise use MongoDB on EC2.
If you need RDBMS then RDS (benefits:
scalability, availability, no headache with maintenance)
Cache: they have both Redis and memcached.
S3: to store static assets.
I do not suggest CloudFront, there are another CDN on market with better
price/possibilities.
API gateway: yes, if you have an API.
Depending on your app, you may need SQS.
Cognito is a good service if you want to authenticate your users at using google/fb/etc.
CloudWatch: if you're metric-addict then it's not for you, perhaps standalone EC2
will be better. But, for most people CloudWatch is abcolutely OK.
Create all necessary alarms (CPU overload etc).
You should use roles
to allow access to your S3/DB from lambda/AWS.
You should not use the root account but create a separate user instead.
Create billing alarm: you'll know if you're going to break budget.
Create lambda functions to backup your EBS volumes (and whatever else you may need to backup). There's no problem if backup starts a second later, so
Lambda is ok here.
Run Trusted Adviser now and then.
it'd be better for you to set it up using CloudFormation stack: you'll be able to deploy the same infrastructure with ease in another region if/when needed, also it's relatively easier to manage Infrastructure-as-a-code than when it built manually.
If you want a very high scalable application, you may be need to use a serverless architecture with AWS lambda.
There is a framework called serverless that helps you to manage and organize all your lambda function and put them behind AWS Gateway.
For the storage you can use AWS EC2 and install MongoDB or you can go with AWS DynamODB as your NoSql storage.
If you want a frontend, both web and mobile, you may be want to visit the react native approach.
I hope I've been helpful.
Does amazon AWS have decent service for internet-of-things type applications (e.g. NEST thermostat, Wifi controlled appliances)? We would like to connect up to 2 million devices through the cloud. I can see how you might be able to do this with Amazon SQS and the Elastic Beanstalk, however I was hoping that there might be a better way that is less custom. For example, is there a good rules engine for SQS messaging?
I know that the NEST thermostat has solved a similar problem.
Thanks,
Mike
Try Temboo. Temboo offers a platform containing methods for 100+ APIs, databases, and code utilities, the Library and the Temboo platform are accessible via the Temboo SDK in seven programming languages, the Temboo REST API, and the Arduino Yún.
I am totally new to AWS.
I have $8700 credit from Amazon AWS. We are 87 people that I want to share this $8700 among us. (preferably $40 for each person).
1- Please guide me how I can create accounts for them and allocate like $40 to each? Or assume that they have already AWS accounts and I want to allocate $40 to each.
2- We are going to use Amazon AWS to get familiar with map-reduce. I don't know what service we should use (like EC2, Elastic ...). We prefer the easiest one. We want to use 1 computer first and run a map-reduce function on a big data-set and see how long the process takes and test it again using 4 and 8 systems to see the difference.
3- What language should we use for map reduction. Is it possible to use JAVA or C++? Where should we write our code (In netBeans, Microsoft VS, ...)? And where I can find some sample codes?
4- I am not sure about the data set either. Should it be on Oracle? Microsoft SQL SERVER ...
Thank you so much for your help in advance.
I really appreciate your help.
1 - you need to understand AWS Identity and Access Manager http://aws.amazon.com/iam/. You can create users linking the same billing account. I'm not sure if you can allot credits to individual users, but you can control access.
2 - The service you want is AWS Elastic MapReduce http://aws.amazon.com/elasticmapreduce/
3- What language are you most comfortable with?
Q: What programming languages does Amazon Elastic MapReduce support?
You can use Java to implement Hadoop custom jars. Alternatively, you may use other languages including Perl, Python, Ruby, C++, PHP,
and R via Hadoop Streaming. Please refer to the Developer’s Guide for
instructions on using Hadoop Streaming.
http://aws.amazon.com/elasticmapreduce/faqs/#dev-8
4- I think you mean databases...I favor MySQL because its cheaper to run than MSSQL or Oracle. have you read about Amazon RDS? http://aws.amazon.com/rds/
For amount sharing you can look at IAM role.Put yourself as Admin.
For Mapreduce EMR is best option as you don't have
to take care about complicated time sync and DNS issues.
Yes it is possible to use JAVA/C++.For sample code you can refer s3 bucket s3n://elasticmapreduce/samples And also documentation from amazon is pretty good http://docs.aws.amazon.com/ElasticMapReduce/ . You can write your code on AWS instances or you can create your local dev environment and build your code on EMR instances.
Is it dataset and database ?