This might sound like a newbie question but I have a neo4j instance running on Amazon cloud. The instance is set to Autoscale at 80% usage. That means Amazon one the usage is reaches 80%, Amazon will create another instance on Neo4j with the same configuration and will keep adding more once this one reaches 80%..
My questions are -
1) Does this setup on Amazon means we have a cluster of neo4j in place?
2) Do I need to do anything else in order to have neo4j cluster, what I have read is that you need some tool like zookeeper to maintain the cluster..
3) Does this current setup on Amazon will have both instances as Master or will it be more like master/slave setup..
Any help, feedback, suggestions would be helpful.
Thanks in advance,
Ravi
Yes, if you are using Auto Scaling group for Neo4j you need to set a cluster. As #stefan-armbruster mentioned, you need to Neo4j Enterprise edition for that. In that case it's master/slave setup.
Neo4j has its own solution for Cluster management, instead of Zookeeper.
But with AWS and EC2 there are few open question, how to properly deploy Neo4j with Auto Scaling group.
From configuration file perspective
* You need to maintain unique clusterId for each machine in cluster.
* You need to know ip addresses/hostnames of other machines in cluster.
Neo4j Enterprise edition features clustering, see the docs on this. With some well written scripts around that to configure the new instances properly I don't see a reason why AWS autoscale should not work.
Related
I know the question is normally the other way around, but it seems important to know how can we go back to a provisioned solution from a serverless cluster.
We will migrate to an Postgres RDS serverless cluster in a few days and we would like to know how can we go back to a provisioned cluster if something goes wrong with the serverless solution. I didn't find any answers, as DMS doesn't seem to allow serverless sources.
Is there any way to achieve this, other than using pg_dump?
Thank you!
It depends on the version of AWS RDS Aurora Serverless that you are using:
For Serverless v1, just create a snapshot and restore it to a provisioned cluster. Snapshots are compatible between provisioned clusters and Serverless v1.
Serverless v2 uses the same clusters as provisioned instances (see the first paragraph of the AWS Docs for Serverless v2). Select the serverless instance of the cluster, click the "Modify" button and select another "DB instance class" under "Instance configuration". Note: You can also have serverless and provisioned instances in the same cluster, which allows for more complex migration paths like first adding a new instance, remove the old one and using writer instance promotion.
You may first want to test those methods with test data even though I validated them for you with empty databases.
Please also keep in mind that Severless v2 only released in April 2022 (after a long beta testing period solely with MySQL), so you may want to thoroughly test it if you want to use it in production.
I am wondering if there is a support for elastic search cross cluster replication on AWS and Azure?
I see AWS announcement where said that they are going to support cross cluster search (don't sure that is tis related to my query though).
Could you please advice if it is supported or are there any news pointing that it might be supported in the nearest future?
Highly appreciate any help.
AWS Elasticsearch doesn't have CCR(Cross Cluster Replication).
However you can achieve (depends on your RTO & RPO) the same by taking manual snapshots into your own S3 there by replicating the data into new Domain.
For more info on manual snapshots , you can go through here
At last Amazon releases new opensearch service for elasticsearch cross cluster replication. Here is announce - https://aws.amazon.com/about-aws/whats-new/2021/10/amazon-opensearch-service-amazon-elasticsearch-service-cross-cluster-replication/
I'm looking at the possibly of replacing/moving our existing Apache Kafka set up (version 2.1.0) to Amazon's MSK and for it work on EKS.
I've been looking around to see if this is actually possible and if someone has done this or attempted it but so far I've only seen reference to using Apache Kafka on EKS. Does anyone know if it is possible/makes sense to use MSK on EKS?
Many thanks.
Amazon MSK provides fully-managed Kafka clusters, which means that from your side, you do not have to operate the cluster at all. Broker and Zookeeper nodes are packaged, deployed, created, updated and patched for you.
This step-by-step tutorial illustrates the creation of a cluster.
The answer is not, MSK is a fully managed service provided by AWS, you cannot install managed service :-) but you can run your own Kafka cluster on top of Kubernetes cluster in AWS, eg. on EKS service while installing a Kafka Operator:
https://banzaicloud.com/docs/supertubes/kafka-operator/
I haven’t done it for MSK before but surely done it for AWS Aurora Postgres. Not sure why you can’t define your external persistence (in this case MSK) as a service with no selector then manually register an Endpoint object pointing to the MSK host.
https://kubernetes.io/docs/concepts/services-networking/service/#services-without-selectors
I am running a storm cluster on AWS. But I want the storm cluster to expand automatically when the need comes. I figured out mesos is something like that. But I do not have much knowledge about mesos and its deployment on AWS.
Can mesos on AWS automatically increase the parallelism of my topology tasks by launching new instances and shutting them down when not necessary? If it can, how do we configure mesos for the same.
Mesos does not directly handle autoscaling itself, but allows frameworks running on top of it to receive new resource offers and react to them by launching new task instances. I haven't used it personally, but you could try the Storm-Mesos framework for running Storm on Mesos: https://github.com/mesos/storm
Once you have Storm running on Mesos, ready to launch new instances as resources come available, you're ready to autoscale within the existing cluster's capacity. You'll probably want to take advantage of Amazon's Auto-Scaling Groups (ASGs) to scale up the number of Mesos nodes based on your need. As the ASG scales up more Mesos nodes, the resources from those nodes will be automatically offered to the Storm-Mesos framework, which can launch more Storm instances.
Yes, you're heading in the right direction. However I'd suggest to use Marathon rather than the low-level Mesos API.
See for example the GitHub repo obaidsalikeen/storm-marathon, which is particularly well done in terms of completeness and documentation richness.
Can IBM Integration Bus((and /or Websphere message Broker) be implemeted on AWS ? Can my on-premise ESB be migrated to AWS Cloud ?
Thanks in Advance
AWS EC2 allows importing VMs into an AMI then you can start an EC2 instance using that image. If you are new to AWS you can check the link below
https://aws.amazon.com/ec2/vm-import/
However, you should be careful about IIB license and how many machines you can install it on before regesting the AMI in a launch configuration and create an autoscaling group and set a scaling policy that can start instances more that what you purchased.
That's very much possible. There are several possible approaches.
1. IIB on EC2
Installing and configuring IIB on an EC2 instance is very much similar to doing the same in on-premise servers. Only difference is that the physical server is in AWS Cloud. While this approach gives you maximum flexibility to design your architecture any way, it does not take advantage of the basic features of the cloud.
2. Quick Start
IIB is available for deployment under AWS Quick Start. You can read more about this here. This helps you get started quickly by setting up the entire environment in a few clicks. But, if you're planning to migrate your existing architecture to AWS, this may not suit you as the architecture is pre-defined with limited options for customization.
3. IIB on Containers
ACE 11 provides better support for containerization. You can read more about running IIB 10 on containers here and ACE 11 on containers here. After this, the containers can be deployed into fully managed containers such as AWS Elastic Container Service or your own container configuration such as Docker on EC2.
Yes of course, AWS provides the IAAS and you just install whatever you want inside. Make sure you open ports, use specific credentials for the instalation (dont use admin) and everything should work.
IBM also provides docker images of integration bus v10 and APP Connect Enterprise v11. This is true for all their integration tools, MQ, API Management and more.
Not restricted to AWS.