I'm a newbie in some of the AWS services. I was following this documentation link:
http://docs.aws.amazon.com/AmazonElastiCache/latest/UserGuide/GettingStarted.ConnectToCacheNode.Redis.html
And I already installed redis-cli with brew in my computer(I'm in a mac) and I'm still having the same error when trying to connect to the node:
$ redis-cli -h mynode.abcdef.0001.usw2.cache.amazonaws.com -p 6379
Error:
Could not connect to Redis at mynode.abcdef.0001.usw2.cache.amazonaws.com:6379: Operation timed out
Yes, I have configured the VPC Security Group to allow all inbound traffic to my Node and the problem persist.
Security Group Conf:
Node Description:
Any ideas?
You can't connect to eleasticache from outside of aws. It just the way it is setup. Would be nice to do for debugging and development, but for production it doesn't really make sense to introduce that much latency into a system that main purpose is to give as-fast-as-possible results.
From AWS FAQ:
Please note that IP-range based access control is currently not
enabled for Cache Clusters. All clients to a Cache Cluster must be
within the EC2 network, and authorized via security groups as
described above.
http://aws.amazon.com/elasticache/faqs/
External access to Elasticache resources is possible yet discouraged:
Elasticache is a service designed to be used internally to your VPC.
External access is discouraged due to the latency of Internet traffic
and security concerns. However, if external access to Elasticache is
required for test or development purposes, it can be done through a
VPN.
Guide: Accessing ElastiCache Resources from Outside AWS
Related
I'm trying to access lambda functions from a Windows VM I have created in EC2 for dev purposes but even a simple 'list functions' command fails to connect
I have tried using the AWS CLI through PowerShell, the dotnet sdk and the VS AWS Toolkit but each of these times out after a long waiting period. I can, however, list other services such as my databases and S3 buckets.
aws cli failure message
VS toolkit failure message
I have tried creating a new VM with the same results. I've disabled windows firewall altogether, allowed all traffic through the security group and have VPC endpoints for my subnet (ssm, ec2messages, lambda, ec2).
I have no trouble connecting to the lambda service through my own computer. On the VM, I have modified the .aws/credentials file to match the one on my computer for both the admin and current user but I still can't connect. This tells me that the problem isn't related to my access key credentials.
I'm reaching the end of the troubleshooting options I can think of so any help would be very much appreciated!
Update: using telnet, I cannot connect to lambda.ap-southeast-2 but I can connect to s3.ap-southeast-2 and lambda.ap-southeast-1. It seems lambda.ap-southeast-2 is being blocked somewhere but it isn't windows firewall because it's off and the same problem happens on Ubuntu VMs.
In the VPC Management Console, I haven't set up any firewalls under network or dns filewalls and my network ACL allows all traffic.
I have set up an EKS cluster and I am trying to connect application pod to ElastiCache endpoint. I put both in same VPC and configured in/out security groups for them. Unfortunately while trying to telnet from pod to cache endpoint, it says "xxx.yyy.zzz.amazonaws.com: Unknown host". Is it even possible to make such a connection?
Yes, if the security groups allow connectivity then you can connect from EKS pods to Elasticache. However, be aware that the DNS name may not resolve for some time (up to around 15 minutes) after you launch the Elasticache instance/cluster.
I found an answer in the issue from cortexproject(monitoring tool based on Grafana stack).
Solved it using "addresses" instead "host" with the address of my memcached. It worked.
PS: "addresses" option isn't documented in the official documentation.
It has to view like this:
memcached_client:
addresses: memcached.host
Is it possible that redis cli is given less priority to connect when memory consumption is high but application is allowed to communicate?
I am unable to connect via cli so can't check anything. Also, don't have the redis server access.
We connect without authentication -
redis-cli -h <hostname>
I ran a process which inserted too many redis keys and that caused this situation. Now, I am not able to delete those keys. I am afraid, the other necessary keys would get evicted as old and system would start doing processing for things not available in redis.
Not able to connect via telnet as well.
Is it possible to connect via a Python script at this point?
If I restart the Java application, will it be able to connect anymore?
Will redis server access via AWS console be able to delete any of the key patterns? I don't have the access currently, so not able to confirm myself. Never used via it also.
Update
Following are graphs taken from AWS console, over the last 1 day since this issue happened -
Update
I went through the FAQ of elasticache, but did not find any mention of being able to manage the data at key value pair level or presence of some special privilege users like root in case of MySql which is able to connect when no other users are able to connect.
All I found is cluster level management capabilities.
From the question it is not clear whether the redis-cli -h <host> command you're running is from within the EC2 or it is from you local machine (Outside AWS VPC).
Accessing from EC2
You will have to ensure following points:
Both the EC2 instance and Redis Instance are on the same VPC.
The security group on EC2 should be allowing port 6379 (It should already be if an application is able to access Redis on the same EC2)
Accessing from outside Amazon VPC
This is not something that's preconfigured and I will suggest that you go through the Accessing Your Cluster docs under the heading "How to Access ElastiCache Resources from Outside AWS".
First, check the connectivity from source(Generally Ec2 instance) to Target (Redis Host).
We can use simple command for that like
#curl -v hostIP(or dnsName):Port
#curl -v myredis.com:6379 or curl -v 192.17.37.42:6379
If you see "Connected" then there is no issue with the network otherwise you have to look into network configurations like firewalls.
Next, you can connect to Redis using redis-cli with the below command:
#redis-cli -h myredis.com -p 6379
I am setting up a Tomcat application in EC2. For reliability, I am running two or more instances. If one server goes down, my users should be redirected to the other instance. This suggests that session state should be kept in an external source, or mirrored between the servers.
AWS offers a hosted service, Elasticache, which seems like it would work well. I even found a nice library, memcached-session-manager. However, I soon ran into some issues.
Unless someone can convince me otherwise, I need the session states to be encrypted in transit. Otherwise someone could intercept the network traffic and pretend to be someone else on my site. I don't see any built-in Amazon method to keep traffic off the internet. (Is peering available here?)
The library mentioned earlier does have Redis support with SSL, but it does not support a Redis cluster. Someone put in a pull request for this but it has not been incorporated and this library is a complex build. I may talk myself into living without the cluster, but that puts us back at a single point of failure.
Tomcat is running on EC2 in your VPC, and ElastiCache is in your VPC. Your AWS VPC is an isolated network. Nobody can intercept the traffic between the EC2 and Elasticache servers unless your VPC network becomes compromised in some way.
If you want to use Redis instead, with SSL connections, then I believe at this time you would need a Tomcat Session Manager implementation that uses Jedis. This one uses Jedis, but you would need to upgrade the version of Jedis it uses in order to use SSL connections.
I'm getting ready to setup HashiCorp Vault with my web application, and while the examples HashiCorp provides make sense, I'm a little unclear of what the intended production setup should be.
In my case, I have:
a small handful of AWS EC2 instances serving my web application
a couple EC2 instances serving Jenkins for continuous deployment
and I need:
My configuration software (Ansible) and Jenkins to be able to read secrets during deployment
to allow employees in the company to read secrets as needed, and potentially, generate temporary ones for certain types of access.
I'll probably be using S3 as a storage backend for Vault.
The types of questions I have are:
Should vault be running on all my EC2 instances, and listening at 127.0.0.1:8200?
Or do I create an instance (maybe 2 for availability) that just run Vault and have other instances / services connect to those as needed for secret access?
If i needed employees to be able to access secrets from their local machines, how does that work? Do they setup vault locally against the S3 storage, or should they be hitting the REST API of the remote servers from step 2 to access their secrets?
And to be clear, any machine that's running vault, if it's restarted, then vault would need to be unsealed again, which seems to be a manual process involving x number of key holders?
Vault runs in a client-server architecture, so you should have a dedicated cluster of Vault servers (usually 3 is suitable for small-medium installations) running in availability mode.
The Vault servers should probably bind to the internal private IP, not 127.0.0.1, since they they won't be accessible within your VPC. You definitely do not want to bind 0.0.0.0, since that could make Vault publicly accessible if your instance has a public IP.
You'll want to bind to the IP that is advertised on the certificate, whether that's the IP or the DNS name. You should only communicate with Vault over TLS in a production-grade infrastructure.
Any and all requests go through those Vault servers. If other users need to communicate with Vault, they should connect to the VPC via a VPN or bastion host and issue requests against it.
When a machine that is running Vault is restarted, Vault does need to be unsealed. This is why you should run Vault in HA mode, so another server can accept requests. You can setup monitoring and alerting to notify you when a server needs to be unsealed (Vault returns a special status code).
You can also read the production hardening guide for more tips.
Specifically for point 3 & 4:
They would talk to the Vault API which is running on one/more machines in your cluster (If you do run it in HA mode with multiple machines, only one node will be active at anytime). You would provision them with some kind of authentication, using one of the available authentication backends like LDAP.
Yes, by default and in it's recommended configuration if any of your Vault nodes in your cluster get restarted or stopped, you will need to unseal it with however many keys are required; depending on how many key shards were generated when you stood up Vault.