Is it possible to connect from an AWS Lambda function to a Redis ElastiCache cluster?
I can't figure out if it's a configuration problem or it's simply not possible.
PS: I made a test from an EC2 instance and I can connect to the Redis node. Also the Lambda function and the Redis node are in the same region.
UPDATE (09 Oct 2015):
Amazon announced VPC for AWS Lambda functions. Details here
This means we can now access any resource in AWS behind VPC security group, including ElastiCache and RDS machines.
UPDATE (11 Feb 2016):
Amazon launched VPC for AWS Lambda.
https://aws.amazon.com/about-aws/whats-new/2016/02/access-resources-within-a-vpc-using-aws-lambda/
As of Feb 2016, AWS allows using lambda functions to connect to Elasticache. Refer to Access Resources within a VPC using AWS Lambda. Here is a link how it works - Tutorial: Configuring a Lambda Function to Access Amazon ElastiCache in an Amazon VPC
Setting up an HTTP Proxy or iptables wouldn't work for the following reasons:
Redis calls are not HTTP and will not be handled by HTTP proxies. iptables (or any port forwarding for that matter) will either won't accept a domain name as destination or is highly inefficient due to DNS resolution required every time.
The best and convenient method is to install twemproxy in an EC2 machine and route your requests through it. As a bonus, you suddenly have deployed a fantastic sharding strategy as well.
I have tried connecting lambda to memcached elasticache and it works fine. Redis should also be doable.
Couple of things to keep in mind:
Lambda and Elasticache has to be in the same VPC.
When lambda is run in VPC, it won't have access to internet (so access to public APIs won't work). NATGateway is required for this.
I was experiencing the same issue. I did not find a direct solution but instead used a Lambda function to connect to an EC2 server using socket.io which was pretty easy and emit an event to that EC2 server.
When the EC2 server received the event it performed the necessary Redis task ( database cleanup after image thumbnail generation ).
Hope this helps! If anyone finds out how to connect to ElastiCache from Lambda directly I'd still love to know!
Related
I am trying to run an AWS Glue Command from AWS CLI to get my job started from an EC2 instance. This is the command
aws glue start-job-run --job-name Connection_Test
But I get the following error:
ConnectTimeoutError: Connect timeout on endpoint URL: "https://glue.us-east-1.amazonaws.com/"
I have added a connection to the Glue Job but I still got the same answer. Do you know what could be? Thanks for your answer!
These errors:
ConnectTimeoutError: Connect timeout on endpoint URL"
occur when your environment (in this case, an EC2 instance) is not able to communicate with the AWS service in question (glue.us-east-1).
You have two options to fix this:
Give your EC2 instance internet connectivity, with either an internet gateway or a NAT gateway. If you do this, traffic to the service will go over the public internet
Create an interface VPC endpoint for the service and deploy it in the same subnet as the EC2 instance. If you do this, traffic to the service will go over AWS private network
The second option is generally the best one, because connectivity to the endpoint will be faster, and because it allows you to keep your instance on a private network.
(The two options are not mutually exclusive).
I have a redis instance on AWS that I want to connect using Redis Desktop Manager from my local machine
I am able to ssh into my ec2 instace and then run redis-cli -h host and connect to it.
But the same is not possible from my local machine.
I am sure there must be a way to monitor my redis using the GUI, and I think if I can connect to the ec2 using pem file and I can connect to redis from insde there, must be a way to combine both? And connect to the redis instance locally via my ec2 instace? Any ideas?
By design AWS EC domain is deployed for use only within AWS. From docs:
Elasticache is a service designed to be used internally to your VPC. External access is discouraged due to the latency of Internet traffic and security concerns. However, if external access to Elasticache is required for test or development purposes, it can be done through a VPN.
Thus, it can't be accessed directly from outside of your VPC. For this, you need to setup a VPN between your local home/work network and your VPC, or what is often easier to do for testing and development, establish a ssh tunnel.
For the ssh tunnel you will need a public proxy/bastion EC2 instance through which the tunnel will be established. There are number tutorials on how to do it for different AWS services. General procedures are same, whether this is ES, EC, Aurora Serverless or RDS Proxy. Some examples:
SSH Tunnels (How to Access AWS RDS Locally Without Exposing it to Internet)
How can I use an SSH tunnel to access Kibana from outside of a VPC with Amazon Cognito authentication?
As #Marcin mentioned, AWS recommends only using Elasticache within your VPC for latency reasons, but you've got to develop on it some how... (Please be sure to read #Marcin's answer)
AWS is a huge mystery, and it's hard to find beginner-intermediate resources, so I'll expand upon #Marcin's answer a little for those that might stumble across this.
It's pretty simple to set up what's often referred to as a "jump box" to connect to all sorts of AWS resources - this is just any EC2 instance that's within the same VPC (network) as the resource you're trying to connect to - in this case the Elasticache redis cluster. (If you're running into trouble, just spin up a new instance - t4g.nano or something super small works just fine.)
You'll want to make sure you're in the directory with your key, but then should be able to run the following command to link whatever port you'd like to use to the remote redis cluster:
ssh -i ${your_ssh_key_name.pem} ${accessible_ec2_host} -L ${port_to_use_locally}:${inaccessable_redis_or_other_host}:${inaccessable_redis_port}
Then you can use localhost and ${port_to_use_locally} to connect to redis
I am using a lambda function in a VPC to connect to an RDS instance in the same VPC. I am considering removing the lambda from the VPC to massively reduce the cold-start time but I want to keep my RDS instance in the VPC.
Can anyone foresee major problems with making the lambda function use an SSH tunnel to connect to a bastion instance within the VPC and subsequently to the RDS instance? Or something similar with a VPN?
There will obviously be some over-head as the traffic has an extra 'jump' so to speak, but would it be significant enough to make this approach non-feasible? Or is the only current approach to keep the Lambda in the same VPC and try to keep and few invocations running?
I also pay for a NAT gateway so my Lambda in a VPC can access the internet. If I can get it out of the VPC by using an SSH tunnel to connect to the RDS instance it will also simplify my architecture here & reduce my operating costs.
Cold starts because of Lambda's in VPC are a big issue, especially when you want to use a relational database. Luckily, AWS has acknowledged this issue and there is hope on the horizon;
Aurora Serverless now supports the Data API that allows to run SQL queries using the AWS SDK over https. This is released on Nov 20 ('18) and is in beta and only in us-east-1, but it's a start.
During re:Invent '18 an improvement on the VPC-cold-start issue was announced (but no release date yet) in which they basically create an ENI for a group of Lambda's and have that ENI ready even if there are no Lambda's warm.
I have a NodeJS server running on an EC2 instance, orchestrated by Elastic Beanstalk.
I would like to create a Lambda function that is triggered upon certain events in AWS Cognito. I want the Lambda function to make a POST call to my NodeJS server.
How can this be done?
As far as I know, EC2 and Lambda are not integrated out of the box. You'll need to expose your EC2 endpoint through a web-server (e.g. Apache) to receive HTTP requests. Then you send a GET or POST request from Lambda to the EC2 server.
You could have both running in a VPC, so that the IP address you use to make requests to your EC2 web-server is only reachable from within your VPC. It won't make your EC2 callable only by the Lambda function but will prevent the external world from calling your EC2 server.
This tutorial might be useful to you: Configuring a Lambda Function to Access Resources in an Amazon VPC.
I have two VPCs:
VPC1 which holds our RDS instance.
VPC2 which holds our cluster of EC2 instances.
We have successfully setup a VPC peering connection, routes and security groups to allow appropriate communication.
In order to resolve the RDS instance AZ-appropriate local IP address from it's hostname, we need to follow these instructions and set --requester-peering-connection-options AllowDnsResolutionFromRemoteVpc=true.
If I do this manually through the AWS Console or the AWS CLI it all works fine, however I'm creating the cluster of EC2 instances through CloudFormation and the option is missing from the CloudFormation documentation.
The effect of this is that my stack starts up and fails because the services themselves cannot connect to the database.
Am I doing something obvious wrong, or is this just Amazon being incomplete?
Thanks!
Due to the frequency of updates, there are many times where an AWS feature isn't available in CloudFormation (ALB targeting Lambda used to be) - you end up having to create a custom resource to manage it. It's not too bad, just make sure that your lambda responds with success or failure in all scenarios, including exceptions, otherwise your stack will be 'in progress' for hours.