Stopped reason "Error response from daemon: create ecs-parser-api-dev-45-allsorter-efs-c4c8df9386aaff820100: Post http://%2Frun%2Fdocker%2Fplugins%2Famazon-ecs-volume-plugin.sock/VolumeDriver.Create: dial unix /run/docker/plugins/amazon-ecs-volume-plugin.sock: connect: connection refused"
AWS ECS EC2 based:
It was working fine. suddenly a new revision came and it didn't run the new revision but it stopped with above error
This indicates a problem with the ECS volume plugin and not necessarily a permissions/security group issue between ECS and EFS.
Replace your instances and ensure you are running the latest ECS AMI (or ECS agent if you have a custom AMI). That should resolve the issue.
Related
I have the following AWS infrastructure,
VPC
ECS Fargate container running an ASP.net app
MemoryDB Redis Cluster
Elastic Beanstalk Spring Boot Web App using Spring Boot Redis
The problem is the Elastic Beanstalk Spring Boot app cannot connect to the MemoryDB cluster. It always fails with the following error
o.s.d.r.l.RedisMessageListenerContainer : Connection failure occurred.
I am able to connect to my local Redis from the Spring Boot App without any issues. Also, the ASP.net app running in ECS Fargate can connect to the MemoryDB cluster without any issues. Based on this I figured it would be relatively easy to get a Spring Boot app running in Elastic Beanstalk to connect to a MemoryDB cluster without too much trouble, boy was I wrong. I have gone over and over everything configuration related to the VPC and Elastic Beanstalk application. They are on the same VPC, they use the same subnet, they have the necessary inbound and outbound rules in the added security groups. I've added the security group created by the Elastic Beanstalk app to the MemoryDB cluster. I've tried connection through the JedisConnectionFactory with both normal and cluster configurations, I've tried with SSL on and off. I've read over the following Stackoverflow articles,
Connect to Redis (AWS) from Elastic Beanstalk instance
Connecting to AWS MemoryDB
How to connect AWS Elasticache Redis cluster to Spring Boot app?
How run docker redis in cluster mode?
aws elastic beanstalk with spring boot app
Use java Jedis connect to aws elasticache redis
What format to use when entering an IP address into an EC2 Security Group rule?
JedisCluster : redis.clients.jedis.exceptions.JedisNoReachableClusterNodeException: No reachable node in cluster
I did not find anything helpful in any of these articles. It seems to me that there is no documentation anywhere that outlines a simple base example of how to connect a Spring Boot app running in Elastic Beanstalk to an AWS MemoryDB Redis cluster.
My properties file
My RedisConfig Jedis connection factory creations functions
At this point it seems to me the problem is very likely AWS related but I cannot be certain as I don't even know what else to try at this point since I can connect without issue from ECS Fargate and from my local development environment to a local redis db. Does anyone have any suggestions or a very simple tutorial on how to connect a Springboot App running in Elastic Beanstalk to a MemoryDB cluster?
EDIT 1
After looking at the Elastic Beanstalk stdout web log im seeing this error when I use RedisClusterConfiguration connection method
Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.data.redis.RedisConnectionFailureException: No reachable node in cluster; nested exception is redis.clients.jedis.exceptions.JedisNoReachableClusterNodeException: No reachable node in cluster] with root cause
Jul 4 17:15:53 ip-172-31-26-91 web: redis.clients.jedis.exceptions.JedisNoReachableClusterNodeException: No reachable node in cluster
Jul 4 17:15:53 ip-172-31-26-91 web: at redis.clients.jedis.JedisSlotBasedConnectionHandler.getConnection(JedisSlotBasedConnectionHandler.java:86) ~[jedis-3.3.0.jar!/:na]
Jul 4 17:15:53 ip-172-31-26-91 web: at redis.clients.jedis.JedisSlotBasedConnectionHandler.getConnectionFromSlot(JedisSlotBasedConnectionHandler.java:103) ~[jedis-3.3.0.jar!/:na]
Also in the nginx error log from beanstalk im seeing this
connect() failed (111: Connection refused) while connecting to upstream, client: 73.33.4.162, server: , request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:5000/", host: "odyssey-env.eba-yq3g2trt.us-east-1.elasticbeanstalk.com"
Edit 2
When I use RedisStandaloneConfiguration to connect I get this error,
Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.data.redis.RedisConnectionFailureException: Cannot get Jedis connection; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool] with root cause
Jul 4 17:59:26 ip-172-31-26-91 web: java.net.ConnectException: Connection refused (Connection refused)
I have deployed an AWS EKS Cluster, and I was able to read logs from cli without any issues. After deploying an application, I started getting this error from the CLI.
while running the logs command, I was not getting any output. but after waiting for a while . I started getting this error:
logs command : kubectl logs "appname" -n "namespace"
error: Error from server: Get https://x.x.x.x:10250/containerLogs/"namespace"/"appname": dial tcp x.x.x.x:10250: i/o timeout
after editing the security group for the server (cli) which is running the EKS cluster, adding the specific PORTs . it started working .
I have aws linux ubuntu ec2 t2.micro instance with django application deployed, recently I installed boto3 and django-storages on my ec2 instance and created s3 bucket, after which I was unable to makemigrations on my django project as it kept saying
"-bash: fork: Cannot allocate memory issue in EC2 "
after which I stopped and started ec2 server and then I am unable to access ec2 via command line or ftp, even with new public ip. it says
"ssh_exchange_identification: read: Connection reset by peer"
ftp connects sometimes but then few moments later it again says
"Error: Connection reset by peer
Error: Could not connect to server"
need some help here that if issue is due to my storage or usage limit or for some other reason?
Thanks,
Bilal
I have a nodeJS app deployed to Amazon Elastic Beanstalk and uses Application Load Balancer.
The problems is - once in a while instance stop responding. When I try to get logs, I see:
requestEnvironmentInfo is starting
and a timeout as a result.
I simply cannot access logs, even when I do eb shh I get:
INFO: Attempting to open port 22.
INFO: SSH port 22 open.
INFO: Running ssh -i /Users/Rostyslav/.ssh/cards-eb-keypair ec2-user#xx.xxx.xxx.xx
ssh_exchange_identification: read: Connection reset by peer
ERROR: An error occurred while running: ssh.
I also have set up logs auto-transfer to S3, but after instance get stacked - no new logs there anymore.
If I kill the EC2 instance and Load Balancer creates a new one and problem get fixed. However in usually ~1 day the same thing is happening.
Any ideas about why it is happening? Thank you!
I am trying to run storm on my AWS EC2 instances as a cluster. However, I got this error:
ERROR in ch.qos.logback.core.util.ContextUtil#550571f6 - Failed to get local hostname java.net.UnknownHostException: ip-xxx-xxx-xxx-xxx: ip-xxx-xxx-xxx-xxx: Name or service not known
I also got similar error (java.net.UnknownHostException) when try to run kafka-console-consumer.sh.
What's wrong with this? Thanks