I'm looking at the possibly of replacing/moving our existing Apache Kafka set up (version 2.1.0) to Amazon's MSK and for it work on EKS.
I've been looking around to see if this is actually possible and if someone has done this or attempted it but so far I've only seen reference to using Apache Kafka on EKS. Does anyone know if it is possible/makes sense to use MSK on EKS?
Many thanks.
Amazon MSK provides fully-managed Kafka clusters, which means that from your side, you do not have to operate the cluster at all. Broker and Zookeeper nodes are packaged, deployed, created, updated and patched for you.
This step-by-step tutorial illustrates the creation of a cluster.
The answer is not, MSK is a fully managed service provided by AWS, you cannot install managed service :-) but you can run your own Kafka cluster on top of Kubernetes cluster in AWS, eg. on EKS service while installing a Kafka Operator:
https://banzaicloud.com/docs/supertubes/kafka-operator/
I haven’t done it for MSK before but surely done it for AWS Aurora Postgres. Not sure why you can’t define your external persistence (in this case MSK) as a service with no selector then manually register an Endpoint object pointing to the MSK host.
https://kubernetes.io/docs/concepts/services-networking/service/#services-without-selectors
Related
I am wondering if there is a support for elastic search cross cluster replication on AWS and Azure?
I see AWS announcement where said that they are going to support cross cluster search (don't sure that is tis related to my query though).
Could you please advice if it is supported or are there any news pointing that it might be supported in the nearest future?
Highly appreciate any help.
AWS Elasticsearch doesn't have CCR(Cross Cluster Replication).
However you can achieve (depends on your RTO & RPO) the same by taking manual snapshots into your own S3 there by replicating the data into new Domain.
For more info on manual snapshots , you can go through here
At last Amazon releases new opensearch service for elasticsearch cross cluster replication. Here is announce - https://aws.amazon.com/about-aws/whats-new/2021/10/amazon-opensearch-service-amazon-elasticsearch-service-cross-cluster-replication/
I used ECS Fargate and it provides containerization, auto-scaling based on request count, CPU and Memory.
It is working as expected.
I start to explore the AWS EKS feature and I didn't see any advantage in using this as all are provided by ECS Fargate.
Could someone help me understand where to use ECS Fargate and Where to use AWS EKS?
Anyhelp is appreciated.
Thanks,
Harry
You would use AWS EKS if you want to use Kubernetes.
Since Kubernetes is a standard, you could in theory move your application from AWS EKS to other cloud providers like Azure, Google Cloud, or DigitalOcean easily since they all support Kubernetes.
If you don't care about Kubernetes then I find that AWS ECS with the AWS Fargate [Serverless compute for containers] deployment type is currently the easiest method of running Docker containers on AWS.
Note that Amazon is actively working on adding the Fargate deployment type to the EKS service.
I would check back after the AWS re:invent conference next month to see how things have changed in this area.
We hear these questions often and I tried to capture some of the core principles of these comparisons/positioning in this blog post.
I'm looking for a way to establish custom metrics over StatsD protocol for Amazon Elastic Container Service. I've found a documentation on how to establish Amazon CloudWatch Agent on EC2. It works well. However I'm failing to find a correct configuration for Dockerfile. Quite probably some set of custom IAM permissions will also be required there.
Is it possible to have Docker containers working from AWS ECS with custom metrics using StatsD reporting to AWS CloudWatch?
Rather than building your own container, you can use the one provided by Amazon. This article explains how, including a link to an example daemon service task configuration.
This might sound like a newbie question but I have a neo4j instance running on Amazon cloud. The instance is set to Autoscale at 80% usage. That means Amazon one the usage is reaches 80%, Amazon will create another instance on Neo4j with the same configuration and will keep adding more once this one reaches 80%..
My questions are -
1) Does this setup on Amazon means we have a cluster of neo4j in place?
2) Do I need to do anything else in order to have neo4j cluster, what I have read is that you need some tool like zookeeper to maintain the cluster..
3) Does this current setup on Amazon will have both instances as Master or will it be more like master/slave setup..
Any help, feedback, suggestions would be helpful.
Thanks in advance,
Ravi
Yes, if you are using Auto Scaling group for Neo4j you need to set a cluster. As #stefan-armbruster mentioned, you need to Neo4j Enterprise edition for that. In that case it's master/slave setup.
Neo4j has its own solution for Cluster management, instead of Zookeeper.
But with AWS and EC2 there are few open question, how to properly deploy Neo4j with Auto Scaling group.
From configuration file perspective
* You need to maintain unique clusterId for each machine in cluster.
* You need to know ip addresses/hostnames of other machines in cluster.
Neo4j Enterprise edition features clustering, see the docs on this. With some well written scripts around that to configure the new instances properly I don't see a reason why AWS autoscale should not work.
Can IBM Integration Bus((and /or Websphere message Broker) be implemeted on AWS ? Can my on-premise ESB be migrated to AWS Cloud ?
Thanks in Advance
AWS EC2 allows importing VMs into an AMI then you can start an EC2 instance using that image. If you are new to AWS you can check the link below
https://aws.amazon.com/ec2/vm-import/
However, you should be careful about IIB license and how many machines you can install it on before regesting the AMI in a launch configuration and create an autoscaling group and set a scaling policy that can start instances more that what you purchased.
That's very much possible. There are several possible approaches.
1. IIB on EC2
Installing and configuring IIB on an EC2 instance is very much similar to doing the same in on-premise servers. Only difference is that the physical server is in AWS Cloud. While this approach gives you maximum flexibility to design your architecture any way, it does not take advantage of the basic features of the cloud.
2. Quick Start
IIB is available for deployment under AWS Quick Start. You can read more about this here. This helps you get started quickly by setting up the entire environment in a few clicks. But, if you're planning to migrate your existing architecture to AWS, this may not suit you as the architecture is pre-defined with limited options for customization.
3. IIB on Containers
ACE 11 provides better support for containerization. You can read more about running IIB 10 on containers here and ACE 11 on containers here. After this, the containers can be deployed into fully managed containers such as AWS Elastic Container Service or your own container configuration such as Docker on EC2.
Yes of course, AWS provides the IAAS and you just install whatever you want inside. Make sure you open ports, use specific credentials for the instalation (dont use admin) and everything should work.
IBM also provides docker images of integration bus v10 and APP Connect Enterprise v11. This is true for all their integration tools, MQ, API Management and more.
Not restricted to AWS.