I have built a single cassandra node on AWS and its working fine. We want to build 5 more of it so we have cloned the first one into 5 other cassandra servers. I would like to know what all the changes we possibly need to make so that they will run into 5 new cassandra servers.
1) delete all the data in data directories, saved_caches and commitlog
2) update cassandra.yaml file with listen, broadcast and rpc_addresses.
3) what change should we make at system level like hostname or gateway or any other things for the new nodes????? Kindly suggest these, i don't have much knowledge of system administration.
4) I have stopped the original node cassandra service and ds agent service before taking the clones.
please add any other things i need to change to make the cluster work with new 5 nodes.
Many Thanks.
Datastax has a guide for planning an Amazon EC2 cluster. After reviewing the Datastax guide, make sure you configure the cassandra.yaml file with the correct options. If all of the nodes will have duplicate directory structures you can use the same cassandra.yaml file on each node. Note that making changes to the cassandra.yaml will require a restart.
For more information on configuration of the nodes in a single data center see " Initializing a multiple node cluster in a single data center." For multiple data centers see "Initializing a multiple node cluster, multiple data centers."
If you follow the steps outlined in the documentation and guides you should be able to easily add new nodes to your instance.
Related
I have two AWS Elasticache instances, One of the instances (lets say instance A) has very important data sets and connections on it, downtime is unacceptable. Because of this situation, instead of doing normal migration (like preventing source data from new writes, getting dump on it, and restore it to the new one) I'm trying to sync instance A's data to another Elasticache instance (lets say instance B). As I said, this process should be downtime-free. In order to do that, I tried RedisShake, but because AWS restrict users to run certain commands (bgsave, config, replicaof,slaveof,sync etc), RedisShake is not working with AWS Elasticache. It's giving the error below.
2022/04/04 11:58:42 [PANIC] invalid psync response, continue, ERR unknown command `psync`, with args beginning with: `?`, `-1`,
[stack]:
2 github.com/alibaba/RedisShake/redis-shake/common/utils.go:252
github.com/alibaba/RedisShake/redis-shake/common.SendPSyncContinue
1 github.com/alibaba/RedisShake/redis-shake/dbSync/syncBegin.go:51
github.com/alibaba/RedisShake/redis-shake/dbSync.(*DbSyncer).sendPSyncCmd
0 github.com/alibaba/RedisShake/redis-shake/dbSync/dbSyncer.go:113
github.com/alibaba/RedisShake/redis-shake/dbSync.(*DbSyncer).Sync
... ...
I've tried rump for that matter, But it doesn't have enough stability to handle any important processes. First of all, it's not working as a background process, when the first sync finished, it's being closed with signal: exit done, so it will not be getting ongoing changes after the first finish.
Second of all, it's recognizing created/modified key/values in each run, for example, in first run key apple equals to pear, it's synced to the destination as is, but when I deleted the key apple and its value in source and ran the rump syncing script again, it's not being deleted in destination. So basically it's not literally syncing the source and the destination. Plus, last commit to the rump github repo is about 3 years ago. It seems a little bit outdated project to me.
After all this information and attempts, my question is, is there a way to sync two Elasticache for Redis instances, as I said, there is no room for downtime in my case. If you guys with this kind of experience have a bulletproof suggestion, I would be much appreciated. I tried but unfortunately didn't find any.
Thank you very much,
Best Regards.
If those two Elasticache Redis clusters exist in the same account but different regions, you can consider using AWS Elasticache global-datastore.
https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/Redis-Global-Datastores-Console.html
It has some restrictions on the regions, type of nodes and that both the clusters should have same configurations in terms of number of nodes, etc.
Limitations - https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/Redis-Global-Datastores-Getting-Started.html
Otherwise, there's a simple brute-force mechanism and you would be able to code it yourself I believe.
Create a client EC2 (let's call this Sync-er) pub-sub channel from your EC Redis instance A.
Whenever there is a new data, Sync-er would make WRITE commands on EC Redis instance B.
NOTE - You'll have to make sure that the clusters are in connectable VPCs.
Elasticache is only available to the resources within the VPC. If your Instance A and Instance B are in different VPCs, you'll have to peer them or connect them via TransitGateway.
Is there any way of viewing cache data in kubernetes cluster ? Does it get stored in a folder somewhere or can It be shown via a monitoring tool (Prometheus) ?
And how to clean the cache data in k8s cluster ?
There is cache directory in Kubernetes; however, it is held within the Master Node. As shown in this github link, the issue was found when debugging the kube-controller which is handled by the Master Node.
I am not familiar with Amazon's cloud service; however, within GKE, the Master node is a managed service and customers do not have access to it. So to answer your question, yes there is a cache directory in Kubernetes, but you cannot view it within GKE.
with this command you can see the buff/cahe memory for each pod
kubectl exec -it my-cluster-kafka-1 -- bash
and then just top
It would help if you could specify which kind of cache data you would like to clear, and what you would like to achieve.
For kubectl there is a default cache directory$HOME/.kube/cache 1, and $HOME/.kube/http-cache for the http cache 2.
There’s also a cache server like Memcached that is used for caching web content in k8s reference 3.
I am really hoping to use Presto in an ETL pipeline on AWS EMR, but I am having trouble configuring it to fully utilize the cluster's resources. This cluster would exist solely for this one query, and nothing more, then die. Thus, I would like to claim the maximum available memory for each node and the one query by increasing query.max-memory-per-node and query.max-memory. I can do this when I'm configuring the cluster by adding these settings in the "Edit software settings" box of the cluster creation view in the AWS console. But the Presto server doesn't start, reporting in the server.log file an IllegalArgumentException, saying that max-memory-per-node exceeds the useable heap space (which, by default, is far too small for my instance type and use case).
I have tried to use the session setting set session resource_overcommit=true, but that only seems to override query.max-memory, not query.max-memory-per-node, because in the Presto UI, I see that very little of the available memory on each node is being used for the query.
Through Google, I've been led to believe that I need to also increase the JVM heap size by changing the -Xmx and -Xms properties in /etc/presto/conf/jvm.config, but it says here (http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-presto.html) that it is not possible to alter the JVM settings in the cluster creation phase.
To change these properties after the EMR cluster is active and the Presto server has been started, do I really have to manually ssh into each node and alter jvm.config and config.properties, and restart the Presto server? While I realize it'd be possible to manually install Presto with a custom configuration on an EMR cluster through a bootstrap script or something, this would really be a deal-breaker.
Is there something I'm missing here? Is there not an easier way to make Presto allocate all of a cluster to one query?
As advertised, increasing query.max-memory-per-node, and also by necessity the -Xmx property, indeed cannot be achieved on EMR until after Presto has already started with the default options. To increase these, the jvm.config and config.properties found in /etc/presto/conf/ have to be changed, and the Presto server restarted on each node (core and coordinator).
One can do this with a bootstrap script using commands like
sudo sed -i "s/query.max-memory-per-node=.*GB/query.max-memory-per-node=20GB/g" /etc/presto/conf/config.properties
sudo restart presto-server
and similarly for /etc/presto/jvm.conf. The only caveats are that one needs to include the logic in the bootstrap action to execute only after Presto has been installed, and that the server on the coordinating node needs to be restarted last (and possibly with different settings if the master node's instance type is different than the core nodes).
You might also need to change resources.reserved-system-memory from the default by specifying a value for it in config.properties. By default, this value is .4*(Xmx value), which is how much memory is claimed by Presto for the system pool. In my case, I was able to safely decrease this value and give more memory to each node for executing the query.
As a matter of fact, there are configuration classifications available for Presto in EMR. However, please note that these may vary depending on the EMR release version. For a complete list of the available configuration classifications per release version, please visit 1 (make sure to switch between the different tabs according to your desired release version). Specifically regarding to jvm.config properties, you will see in 2 that these are not currently configurable via configuration classifications. That being said, you can always edit the jvm.config file manually per your needs.
Amazon EMR 5.x Release Versions
1
Considerations with Presto on Amazon EMR - Some Presto Deployment Properties not Configurable:
2
We have an Openstack infrastructure consisting of one controller node, eight compute nodes and a network node. This last node is having hardware problems (disk write failures). Unfortunately it has only one disk without replication. And there's no option now to modify it for HA support.
We already tried to "dd" that disk to another but it didn't bring up. So, we agreed that the better choice was to build a new network node (using the same hardware specs)
Failing network node is running the following:
CentOS 7.1.1503
Openstack-neutron-openvswitch-2014.2.2-1 (Juno release)
Openvswitch-2.1.2-2
New network node:
CentOS 7.3.1611
Openstack-neutron-openvswitch-2014.2.3-1 (Juno release)
Openvswitch-2.3.1-2
We managed to export the database, just copying the conf.db file located in /etc/openvswitch into the new node. We had to convert the db to a newer schema since the nodes have different ovs versions. But we can't make it work like the old one, since it adds new interfaces to the database record and doesn't use the ones imported from the old hardware, even having the same exact names.
Is there a way to replicate the ovs configuration in the new node and make it work? That considering that both hardware are the same. Any of you had any experience trying to move/import/export an ovs database? I can attach the database dump if necessary.
I think you must config manually from the beginning, because that database is hashed and encrypted,
when you install openstack throught different machine, the openstack generate keystone randomly,
if you use packstack it the keystone stone can be same with you generate the answer-file
packstack --gen-answer-file=openstack.conf
and adit the option you want in openstack.conf
I want to fake an enterprise environment with OpenShift Origin V3 to test some stuff.
I'm going to try the advanced installation with multiple masters, etcds and multiple nodes.
https://docs.openshift.org/latest/install_config/install/advanced_install.html
I already did the quick installation once (running OpenShift in a container) and I did the advanced installation a few times (one host which contains a master + a node, and some nodes).
First of all, I'm installing the whole environment on AWS EC2 instances with CentOS7 as OS. I have 2 masters (master1.example.com and master2.example.com) and 3 nodes (node1.example.com, node2.example.com, ...)
I want to seperate my masters and nodes. So containers and images will only be on the nodes. (So no host which contains a master and a node).
My masters needs to be HA. So they will use a virtual IP and pacemaker. But how do I have to configure this? There are some tutorials to use pacemaker with apache. But there is nothing that describes the configuration of pacemaker and vip for using it in OpenShif.
great news, I had to deal with pacemaker as well but now Pacemaker is not the native HA method for Openshift anymore(from v3.1). So we can get rid of the tedious pcs configuration and fencing tunning.
Now ansible installation playbook's take care of multiple masters installation, with what they called Openshift HA native method. No additional setup is required for a standard configuration.
Openshift HA method takes advantage of etcd to select the active leader every 30s(by default)
There is a method to migrate from Pacemaker to native HA
https://docs.openshift.com/enterprise/3.1/install_config/upgrading/pacemaker_to_native_ha.html#install-config-upgrading-pacemaker-to-native-ha