Getting "Connection refused" when running nodetool on Cassandra cluster on AWS EC2 - amazon-web-services

I just did the setup of a cassandra cluster.
I have changed the seed, listen_adress and broadcast_adress in the cassandra.yalm file. But when I run the command
$ nodetool flush system
Cmd retrun this error,
nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException:
'Connection refused (Connection refused)'.
in the file etc/cassandra/cassandra-env.sh I made the modification JVM_OPTS as on the screenshot
I use aws at the beginning I was on a t2.micro server. But I switched to a t2.large as recommended in many articles.
Finally, my ports are open as shown in this screenshot and I'm use ubuntu.

By default, remote JMX connections to Cassandra nodes are disabled. If you're running nodetool commands on the EC2 instance itself, it isn't necessary to modify the JVM options cassandra-env.sh.
In fact, we discourage allowing remote JMX connections for security reasons. Only allow remote access if you're an expert and know what you're doing. Cheers!

Related

How to connect a ec2 instante with a localhoster docker swarm cluster

I've been trying to connect an ec2 machine from aws to my localhost (WSL on windows) docker swarmer cluster, but i keep getting displayerd : Error response from daemon: Timeout was reached before node joined. The attempt to join the swarm will continue in the background. Use the "docker info" command to see the current swarm status of your node." and the ec2 is not being added as a node (even if later I try to add it again, it says that it is already part of a cluster, on my localhost it does not appear added).
What I've been tried:
Open the doors 2377, 7946 and 4789 (required by docker) on my wsl and ec2.
approved all traffic to all ports on my ec2 firewall.
desable my windows firewall (Tried to init a windows cluster to add ec2, but did not worked too.
Aditional information:
to open the doors on my wsl/ec2 I mainly used ufw and telnet.
I was able to connect my windows docker to my wsl cluster.
I'm being able to ping my ec2 ipv4 adress from mylocalhost, but not my localhost ip from ec2.
Any suggestions and solutions are welcome, i'm seriously HOURS in this, any progress will make me happy
Systems: I'm using ubuntu 18-04 on wsl and ecs, and windows 11
Please find below ports and protocols open on the security group my ec2 is using

Unable to SSH into EC2 server after reboot

I have an ubuntu 18.04 based EC2 instance using an Elastic IP Address. I am able to SSH into the instance without any problems.
apt is executing some unattended updates on the instance. If I reboot the system after the updates, I am no longer able to SSH into the system. I am getting the error ssh: connect to host XXX port 22: Connection refused
Few points:
Even after the updates, I am able to SSH before the reboot
Method of restart does not make a difference. sudo shutdown -r now and EC2 dashboard have the same result.
There are no problems with sshd_config. I've detached the volume and attached it to a new working instance. sshd -t did not report any problems either
I am able to do sudo systemctl restart ssh.service after the updates but before the system restart.
I've tried with and without Elastic IP. Same result
From the system logs, I see that SSH is trying to start, but failing for some reason
I want to find out why the ssh daemon is not starting. Any pointers?
Update:
System Logs
Client Logs
No changes in the security groups before and after reboot
EC2 > Network & Security > Security Groups > Edit inblound rules > SSH 0.0.0.0/0
Step 1: EC2 > Instances > Actions > Image and templates > Create image
Step 2: Launch a new instance using the AMI image.
I missed the error Failed to start Create Static Device Nodes in /dev. in system logs. The solution given at https://askubuntu.com/questions/1301750/ubuntu-16-04-failed-to-start-create-static-device-nodes-in-dev helped solve my problem

Redis cli unable to connect to AWS ElastiCache due to redis memory usage but application still able to communicate

Is it possible that redis cli is given less priority to connect when memory consumption is high but application is allowed to communicate?
I am unable to connect via cli so can't check anything. Also, don't have the redis server access.
We connect without authentication -
redis-cli -h <hostname>
I ran a process which inserted too many redis keys and that caused this situation. Now, I am not able to delete those keys. I am afraid, the other necessary keys would get evicted as old and system would start doing processing for things not available in redis.
Not able to connect via telnet as well.
Is it possible to connect via a Python script at this point?
If I restart the Java application, will it be able to connect anymore?
Will redis server access via AWS console be able to delete any of the key patterns? I don't have the access currently, so not able to confirm myself. Never used via it also.
Update
Following are graphs taken from AWS console, over the last 1 day since this issue happened -
Update
I went through the FAQ of elasticache, but did not find any mention of being able to manage the data at key value pair level or presence of some special privilege users like root in case of MySql which is able to connect when no other users are able to connect.
All I found is cluster level management capabilities.
From the question it is not clear whether the redis-cli -h <host> command you're running is from within the EC2 or it is from you local machine (Outside AWS VPC).
Accessing from EC2
You will have to ensure following points:
Both the EC2 instance and Redis Instance are on the same VPC.
The security group on EC2 should be allowing port 6379 (It should already be if an application is able to access Redis on the same EC2)
Accessing from outside Amazon VPC
This is not something that's preconfigured and I will suggest that you go through the Accessing Your Cluster docs under the heading "How to Access ElastiCache Resources from Outside AWS".
First, check the connectivity from source(Generally Ec2 instance) to Target (Redis Host).
We can use simple command for that like
#curl -v hostIP(or dnsName):Port
#curl -v myredis.com:6379 or curl -v 192.17.37.42:6379
If you see "Connected" then there is no issue with the network otherwise you have to look into network configurations like firewalls.
Next, you can connect to Redis using redis-cli with the below command:
#redis-cli -h myredis.com -p 6379

AWS SSH into EC2 server timing out

About 6 months ago I created an AWS EC2 instance to mess around with on the free tier. After months of having no issues remoting into my AWS EC2 server, I've recently been unable to access it via SSH. I am using the following command:
ssh -i my-key-pair.pem ec2-user#ec2-**-**-***-***.us-****-*.compute.amazonaws.com
...and after a minute or two, am getting this response
ssh: connect to host ec2-**-**-***-***.us-****-*.compute.amazonaws.com port 22: Operation timed out
What's strange is that
1) I can read and write to my RDS database just fine
2) I can ping into the server
3) My port 22 is open
4) The instance is running and healthy
5) In the Inbound section of the security group of the EC2 server it allows for all traffic and SSH from any location via port 22.
6) I'm using the same key-pair as always
I went through this documentation (https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/TroubleshootingInstancesConnecting.html) and can confirm that the VPC, subnet, network ACL and route tables all line up (I haven't changed anything with those since the SSH stopped working). Any insight would be extremely helpful!
Sometimes the instance fails, you can check the screen of it via AWS
console.
Run another instance in the same security group and try to
connect to it and then from there to your original one - to verify if
ssh is still open (even if you do not have the ssh key, the error
will not be 'timeout')
You can create a snapshot of your instance and
attach it as another volume in a new one and you can investigate
logs, maybe something went wrong.
You can restart the instance, if
for example i ran out of memory it will most likely work after the
reboot (hopefully for a long enough time for you to investigate).
You can contact AWS support.

'Waiting for SSH to be available' during creation of a generic docker machine with amazon aws

I'm trying to use docker-machine with my docker instance hosted on amazon aws.
I run the following command:
$ sudo docker-machine create --driver generic --generic-ip-address={EC2 IP} --generic-ssh-key ~/.ssh/id_rsa dockeraws
Running pre-create checks...
Creating machine...
(dockering) Importing SSH key...
Waiting for machine to be running, this may take a few minutes...
Detecting operating system of created instance...
Waiting for SSH to be available...
Error creating machine: Error detecting OS: Too many retries waiting for SSH to be available. Last error: Maximum number of retries (60) exceeded
But It stucks on 'Waiting for SSH to be available...' and I don't know why.
I've also opened the ports '22' and '2376' but it's still not working.
For my instance I'm using the template stated on the docker page here -> https://docs.docker.com/docker-for-aws/
Try adding your machine ip address to allowed host to the security group used by your ec2 instance. This solved the issue for me.
Generate a ssh key (if you didn't):
ssh-keygen
Then install your public key in the server using ssh-copy-id:
ssh-copy-id user#remote-server
Where user is your remote user and remote-server your server IP/URL.