Are there any cloud services that will greatly simplify Periscope-like application development? We also need to record (archive) live streams for further playback on mobile devices.
I see that many well-known cloud services like Azure Media Services or AWS Elemental Media Live have a low default limit for maximum number of live channels, like 5-10.
See a response to a similar question here. Others have tried using products from Wowza, building and managing their own deployment of VMs running their streaming engine.
Related
I am working on an AWS cloud project which involves video streaming. For this case, a media player needs be to used is Exoplayer along with required AWS services but I am not able to find anything regarding Exoplayer's use on Websites. Can anyone help me with the usage of Exoplayer for web applications?
ExoPlayer is an Android player - for websites, you would typically use a HTML5/JavaScript player assuming your service is using a typical video streaming technology such as HLS or DASH.
OpenSource HTML5/JavaScript players are readily available:
https://github.com/videojs/video.js
https://github.com/shaka-project/shaka-player
There are also multiple commercial players also like BitMovin, TheoPlayer, JWPlayer etc.
I want to plan an architecture based on GCP cloud platform. Below are the subject areas what I have to cover. Can someone please help me to find out the proper services which will perform that operation?
Data ingestion (Batch, Real-time, Scheduler)
Data profiling
AI/ML based data processing
Analytical data processing
Elastic search
User interface
Batch and Real-time publish
Security
Logging/Audit
Monitoring
Code repository
If I am missing something which I have to take care then please add the same too.
GCP offers many products with functionality that can overlap partially. What product to use would depend on the more specific use case, and you can find an overview about it here.
That being said, an overall summary of the services you asked about would be:
1. Data ingestion (Batch, Real-time, Scheduler)
That will depend on where your data comes from, but the most common options are Dataflow (both for batch and streaming) and Pub/Sub for streaming messages.
2. Data profiling
Dataprep (which actually runs on top of Dataflow) can be used for data profiling, here is an overview of how you can do it.
3. AI/ML based data processing
For this, you have several options depending on your needs. For developers with limited machine learning expertise there is AutoML that allows to quickly train and deploy models. For more experienced data scientists there is ML Engine, that allows training and prediction of custom models made with frameworks like TensorFlow or scikit-learn.
Additionally, there are some pre-trained models for things like video analysis, computer vision, speech to text, speech synthesis, natural language processing or translation.
Plus, it’s even possible to perform some ML tasks in GCP’s data warehouse, BigQuery in SQL language.
4. Analytical data processing
Depending on your needs, you can use Dataproc, which is a managed Hadoop and Spark service, or Dataflow for stream and batch data processing.
BigQuery is also designed with analytical operations in mind.
5. Elastic search
There is no managed Elastic search service directly provided by GCP, but you can find several options on the marketplace, like an API service or a Kubernetes app for Google’s Kubernetes Engine.
6. User interface
If you are referring to a user interface for your own use, GCP’s console is what you’d be using. If you are referring to a UI for end-users, I’d suggest using App Engine.
If you are referring to a UI for data exploration, there is Datalab, which is essentially a managed notebook service, and Data Studio, where you can build plots of your data in real time.
7. Batch and Real-time publish
The publishing service in GCP, for both synchronous and asynchronous messages is Pub/Sub.
8. Security
Most security concerns in GCP are addressed here. Which is a wide topic by itself and should probably need a separate question.
9. Logging/Audit
GCP uses Stackdriver for logging of most of its products, and provides many ways to process and analyze those logs.
10. Monitoring
Stackdriver also has monitoring features.
11. Code repository
For this there is Cloud Source Repositories, which integrate with GCP’s automated build system and can also be easily synched with a Github repository.
12. Analytical data warehouse
You did not ask for this one, but I think it's an important part of a data analysis stack.
In the case of GCP, this would be BigQuery.
I wish to monitor all the APIs that I created on one of my docker containers. That Docker container is using Django REST framework for its services.. and I am running it on Azure. I want to monitor my API by means of if it is working or if there are too many requests it will throw an alert.. what is its request per second something like that.
We are using sysdig for monitoring our containers but I don't think it has the capability to monitor all our APIs of our Django Rest Framework
To monitor your API performance and downtime, you could create custom scripts to ping your API and alert you if there's downtime, or you could use a third-party service to monitor remotely. This is the simpler option, as it doesn't require writing and maintaining code.
One third-party service you could use is mine, https://assertible.com. They provide frequent health checks (1/5/15 minute), deep data validation, integrations with other services like Slack and GitHub, and a nice way to view/manage test failures.
If you want to integrate with your own code or scripts, you can use Trigger URLs and/or the Deployments API to programatically run your tests whenever and wherever:
$ curl 'https://assertible.com/apis/{API_ID}/run?api_token=ABC'
[{
"runId": "test_fjdmbd",
"result": "TestPass",
"assertions": {
"passed": [{...}],
"failed": [{...}]
},
...
}]
Hope it helps!
You can use the monitoring functionality from Postman. For more information check out the following link [1].
[1] https://learning.getpostman.com/docs/postman/monitors/intro_monitors/
Since you're running on Azure, you should take a look at Application Insights:
Application Insights is an extensible Application Performance
Management (APM) service for web developers on multiple platforms. Use
it to monitor your live web application. It will automatically detect
performance anomalies. It includes powerful analytics tools to help
you diagnose issues and to understand what users actually do with your
app. It's designed to help you continuously improve performance and
usability. It works for apps on a wide variety of platforms including
.NET, Node.js and J2EE, hosted on-premises or in the cloud. It
integrates with your devOps process, and has connection points to a
variety of development tools. Source
API monitoring is described here.
I would like to build an app and collect some events from the app, and then show some event statistics like frequency, duration etc.
I`ve just investigated the aws Cognito web service, but it stores only a set of key-value pairs of a limited total size.
I can build, of course, my own REST web service on the top of the database and store all my events there. But I wonder if there are some aws web service(s) that I can leverage to build such a solution. (In case if someone familiar with Azure, it would be nice to see the possible solution there too!)
Any ideas, suggestions?
Haven't used any packaged web service for this; however, I do use REST methods for statistics in my apps and find it works well....low overhead and easy to add, change and collect.
I would suggest you to have a look at AWS Mobile Analytics service (http://aws.amazon.com/mobileanalytics/)
Have a look at the Getting Started page http://aws.amazon.com/mobileanalytics/getting-started/
Seb
I've started my journey with cloud related technologies very recently. I'm trying to understand the basics as to be able to prepare the foundation for a basic cloud setup in my Internet of Things oriented company.
While browsing the Internet I've stumbled upon the following two groups of open source projects:
WSO2 / Mule / ...
OpenStack / CouldStack / Eucalyptus / ...
I'm trying to understand:
what kind of service do they offer? (IaaS, PaaS, SaaS, other?)
what are the differences between them?
what do they have in common?
how do the play with other cloud related technologies like Amazon AWS?
which one would you recommend to get some basic experience and for some early proof-of-concept? (I'm looking for the easiest option first)
Cloud stack and Open stack are open source softwares designed to manage, deploy virtual machines and networks which can deliver cloud services. Mainly these provide Infrastructure as a Service (IaaS). There are alot of comparisons on the internet on these two. So these softwares needs to be intalled on your hardware and maintain it and you provide a cloud service from it. When it comes Amazon AWS it is a readily available service where you don't do installations or maintain hardware, you just take service from them.
WSO2 and MuleSoft are different from above two and they are software platforms where several products(such as ESB). Both provide cloud platform facilities to deploye their products.
We cannot say which one to use but base on your requirements you may choose one or two (WSO2 products deployed on Amazon AWS or WSO2 products deployed on CloudStack VM's). Since you are willing to set up Internet of things, i think you may need to refer about products provided by above providers. Following source [1] will give you an idea about Iot platform setup by several free open source WSO2 products.
[1] http://wso2.com/landing/internet-of-things/