I have a nodeJS app deployed to Amazon Elastic Beanstalk and uses Application Load Balancer.
The problems is - once in a while instance stop responding. When I try to get logs, I see:
requestEnvironmentInfo is starting
and a timeout as a result.
I simply cannot access logs, even when I do eb shh I get:
INFO: Attempting to open port 22.
INFO: SSH port 22 open.
INFO: Running ssh -i /Users/Rostyslav/.ssh/cards-eb-keypair ec2-user#xx.xxx.xxx.xx
ssh_exchange_identification: read: Connection reset by peer
ERROR: An error occurred while running: ssh.
I also have set up logs auto-transfer to S3, but after instance get stacked - no new logs there anymore.
If I kill the EC2 instance and Load Balancer creates a new one and problem get fixed. However in usually ~1 day the same thing is happening.
Any ideas about why it is happening? Thank you!
Related
I was no longer able to SSH into a Google Cloud Compute Engine VM instance that previously showed no problems.
The error logs show the following
#type: "type.googleapis.com/google.protobuf.Struct" value: {
conditionNotMet: { userVisibleMessage: "Supplied fingerprint does not
match current metadata fingerprint."
Trying SSH through the console showed
Code: 4003 Reason: failed to connect to backend Please ensure that:
your user account has iap.tunnelInstances.accessViaIAP permission
VM has a firewall rule that allows TCP ingress traffic from the IP range XXX.0/20, port: 22
you can make a proper https connection to the IAP for TCP hostname: https://tunnel.cloudproxy.app You may be able to connect without using
the Cloud Identity-Aware Proxy.
The VM instance logs showed the following
Error watching metadata: Get
http://metadata.google.internal/computeMetadata/v1//?recursive=true&alt=json&wait_for_change=true&timeout_sec=60&last_etag=XXX:
net/http: request canceled (Client.Timeout exceeded while awaiting
headers)
After stopping and restarting the instance I was able to ssh again but I would like to understand the reason for the problem in the first place.
The error message you received indicates that the metadata server's response caused the connection to the Google Compute Engine VM instance to time out. This could be because the server was taking too long to respond or there was a problem with the network. You can try to resolve this issue by either increasing the timeout value by using this doc or waiting for the instance to become healthy using the gcloud compute wait command.
The instance was unable to reach the metadata server, as suggested by the timeout error message. This could be a problem with the instance itself or with the network connection. A firewall or network configuration issue could have prevented the instance from connecting to the metadata server, or an issue with the underlying infrastructure could have rendered the instance temporarily unavailable.
To prevent this issue from happening again, you can increase the timeout value or use the gcloud compute wait command to wait for the instance to become healthy.it is recommended that you regularly update the SSH key used to connect to the instance, and check that the instance can reach the metadata server by making an HTTPS request to the IAP for TCP hostname. Additionally, it is important to ensure that your user account has the "iap.tunnelInstances.accessViaIAP" permission, and that the VM has a firewall rule that allows TCP ingress traffic from the IP range XXX.0/20, port: 22.
If you are using windows vm try troubleshooting steps mentioned in this doc.
I just did the setup of a cassandra cluster.
I have changed the seed, listen_adress and broadcast_adress in the cassandra.yalm file. But when I run the command
$ nodetool flush system
Cmd retrun this error,
nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException:
'Connection refused (Connection refused)'.
in the file etc/cassandra/cassandra-env.sh I made the modification JVM_OPTS as on the screenshot
I use aws at the beginning I was on a t2.micro server. But I switched to a t2.large as recommended in many articles.
Finally, my ports are open as shown in this screenshot and I'm use ubuntu.
By default, remote JMX connections to Cassandra nodes are disabled. If you're running nodetool commands on the EC2 instance itself, it isn't necessary to modify the JVM options cassandra-env.sh.
In fact, we discourage allowing remote JMX connections for security reasons. Only allow remote access if you're an expert and know what you're doing. Cheers!
I have created my ec2 instance successfully in AWS.
here my instance end point is ec2-18-222-211-176.us-east-2.compute.amazonaws.com
The problem is when I try to hit this end point it is showing refused to connect. I don't have any Idea why it is not working like this
where as I successfully connect instance using SSH key, here it is working fine.
here my instance security group inbound rules are listed below
Looks like it is working on both the DNS and the public IP http://18.222.211.176
It was the Apache web server not running. To find this, type this from your root#ip directory:
service httpd status
To start the server, type:
service httpd start
About 6 months ago I created an AWS EC2 instance to mess around with on the free tier. After months of having no issues remoting into my AWS EC2 server, I've recently been unable to access it via SSH. I am using the following command:
ssh -i my-key-pair.pem ec2-user#ec2-**-**-***-***.us-****-*.compute.amazonaws.com
...and after a minute or two, am getting this response
ssh: connect to host ec2-**-**-***-***.us-****-*.compute.amazonaws.com port 22: Operation timed out
What's strange is that
1) I can read and write to my RDS database just fine
2) I can ping into the server
3) My port 22 is open
4) The instance is running and healthy
5) In the Inbound section of the security group of the EC2 server it allows for all traffic and SSH from any location via port 22.
6) I'm using the same key-pair as always
I went through this documentation (https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/TroubleshootingInstancesConnecting.html) and can confirm that the VPC, subnet, network ACL and route tables all line up (I haven't changed anything with those since the SSH stopped working). Any insight would be extremely helpful!
Sometimes the instance fails, you can check the screen of it via AWS
console.
Run another instance in the same security group and try to
connect to it and then from there to your original one - to verify if
ssh is still open (even if you do not have the ssh key, the error
will not be 'timeout')
You can create a snapshot of your instance and
attach it as another volume in a new one and you can investigate
logs, maybe something went wrong.
You can restart the instance, if
for example i ran out of memory it will most likely work after the
reboot (hopefully for a long enough time for you to investigate).
You can contact AWS support.
I have created AWS environment for my DEV and QA. Both the environments are created using the same CloudFormation template. DEV Environment is up and running. But I am unable to launch the QA Instance due to which our testing stopped.
When I try to login to the QA Machine it says
Network connection refused
I have checked the security groups opened port 22 on the machine. I have checked the VPC Configurations and NACLS. Everything looks fine.
What might be the issue?