I would like to use aws sam to setup my serverless application. I have used it with dynamoDB before. This was very easy to since all I had to do was setup a dynamoDB table as a resource and then link it to the lambda functions. AWS SAM seams to know where the table is located. I was even able ot run the functions on my local machine using the sam-cli.
With RDS its a lot harder. The RDS Aurora Instance I am using sits behind a specific endpoint, in a specific subnet with security groups in my vpc protected by specific roles.
Now from what I understand, its aws sams job to use my template.yml to generate the roles and organize access rules for me.
But I don't think RDS is supported by aws sam by default, which means I would either be unable to test locally or need a vpn access to the aws vpc, which I am not a massive fan of, since it might be a real security risk.
I know RDS proxies exist, which can be created in aws sam, but they would also need vpc access, and so they just kick the problem down the road.
So how can I connect my aws sam project to RDS and if possible, execute the lambda functions on my machine?
Related
I need to retrieve existing data/properties of a given resource by using an AWS Cloudformation template. Is it possible? If it is how can I do it?
Example 1:
Output: Security Group ID which allows traffic on port 22
Example 2:
Output: Instance ID which use default VPC
AWS CloudFormation is used to deploy infrastructure from a template in a repeatable manner. It cannot provide information on any resources created by any methods outside of CloudFormation.
Your requirements seem more relevant to AWS Config:
AWS Config provides a detailed view of the configuration of AWS resources in your AWS account. This includes how the resources are related to one another and how they were configured in the past so that you can see how the configurations and relationships change over time.
An AWS resource is an entity you can work with in AWS, such as an Amazon Elastic Compute Cloud (EC2) instance, an Amazon Elastic Block Store (EBS) volume, a security group, or an Amazon Virtual Private Cloud (VPC).
Using your examples, AWS Config can list EC2 instances and any resources that are connected to the instances, such as Security Groups and VPCs. You can easily click-through these relationship and view the configurations. It is also possible to view how these configurations have changed over time, such as:
When EC2 instance changed state (eg stopped, running)
When rules changed on Security Groups
Alternatively, you can simply make API calls to AWS services to obtain the current configuration of resources, such as calling DescribeInstances to obtain a list of Amazon EC2 instances and their configurations.
What I'm trying to do
I am working on a lambda function which will simply register some metadata about files which are uploaded onto an s3 bucket. This is not about actually processing the data in the files yet. To start with, I just want to register the fact that certain files have been uploaded or not. Then I want to connect that metadata to QuickSight just so that we can have a nice visual about which files have been uploaded.
What I've done so far
This part is fairly easy:
Some simply python code with the pymysql module
Chalice to manage the process of creating and updating the lambda function
I created the database
Where I'm stuck
QuickSight is somehow external to AWS in general. So I had to create the RDS (mysql) in the DMZ of our VPC.
I have configured the security group so that the DB is accessible both from QuickSight and from my own laptop.
But the lambda function can't connect.
I configured the right policy for the role, so that the lambda can connect with IAM
I tested that policy with the simulator
But of course the lambda function is going to have some kind of dynamic IP and that needs to be in the security group
Any Ideas ??
I am even thinking about this right?
Two things.
You shouldn't have to put your RDS in a DMZ. See this article about granting QuickSight access to your RDS: https://docs.aws.amazon.com/quicksight/latest/user/enabling-access-rds.html
In order for a lambda to access something in a VPC (like and RDS instance) the lambda must have a VPC configuration. https://docs.aws.amazon.com/lambda/latest/dg/configuration-vpc.html
I have a lambda connected to an API gateway; it's deployed using sls and works great. However, it's datastore is an Aurora that is in the default VPC and is set to public. This is less than ideal, security-wise.
I have, in the past, set up Auroras in their own VPC on private subnets and had ec2s in that VPC easily access it. However, all of the material I have read about getting a lambda to use a VPC RDS states that the lambda itself should also reside in the VPC.
This concerns me because of the cold start issue. So, my questions are:
Is there a way for my 'no vpc' lambda to access an Aurora RDS that
lives in its own VPC without putting the lambda into the VPC itself?
There has been talk for some time that aws will be addressing the lambda VPC 'cold start' issue soon. Do we know when that is anticipated to happen? Will existing lambdas benefit from this change once it is instituted?
Is there some other method of securing a public RDS to restrict access to only my lambda (besides the obvious user/pass credentials)?
Thanks in advance
1. Is there a way for my 'no vpc' lambda to access an Aurora RDS that
lives in its own VPC without putting the lambda into the VPC itself?
No, if your RDS instance is not publicly accessible then your Lambda must be deployed in your VPC.
2. There has been talk for some time that AWS will be addressing the lambda VPC 'cold start' issue soon. Do we know when that is anticipated to happen? Will existing lambdas benefit from this change once it is instituted?
I don't think that a specific timeline has been officially communicated for this feature. Existing Lambda functions will obviously benefit from this change (after all, Lambda functions are just code that is continuously redeployed on containers). Information about the new architecture: AWS Lambda in a VPC Will Soon be Much Faster
3. Is there some other method of securing a public RDS to restrict access to only my Lambda (besides the obvious user/pass credentials)?
You can use IAM Database Authentication with Aurora. With this method, authentication is managed externally using IAM.
I'm trying to insert records into a Postgres database in RDS from a Lambda function. My Node.js lambda function works correctly when run locally, but the database connection times out when run in AWS.
I've read several articles and tutorials which suggest that AWS Lambda functions cannot access RDS instances that are within a VPC. For example: http://ashiina.github.io/2015/01/amazon-lambda-first-impression/
Unfortunately; it seems I am unable to create an RDS instance that exists outside of a VPC. At this dropdown I would expect to be able to select an option for "No VPC" or something along those lines.
Has this option been removed? Perhaps I have missed a step?
You can create a publicly accessible RDS instance. Then you should be able to access it from anywhere, inside or outside AWS. I believe that would get around your issue with Lambda. You are asked if the instances needs to be publicly accessible when you create a new RDS instance via the web console.
Or you could just wait a few weeks, as Lambda within a VPC is supposed to be enabled "later this year".
Edit: Note that newer Amazon accounts are restricted to VPC only resources. You can't create EC2 or RDS instances outside of a VPC anymore. That's why you don't see the "No VPC" option anymore.
Second Edit: VPC access for Lambda functions is now genearally available.
This question is awhile back, but for those of you who are using MySQL, now you can connect AWS Lambda with Aurora Serverless without VPC, utilizing their new Data API. Take a look at this example for details https://coderecipe.ai/architectures/77374273
I am creating an AWS stack inside a VPC using CloudFormation and need to create ElastiCache clusters on it. I have investigated and there is no support in CloudFormation to create cache clusters in VPCs.
Our "workaround" was to to create the cache cluster when some "fixed" instance (like a bastion for example) bootstrap using CloudInit and AWS AmazonElastiCacheCli tools (elasticache-create-cache-subnet-group, elasticache-create-cache-cluster). Then, when front end machines bootstrap (we are using autoscaling), they use elasticache-describe-cache-clusters to get cache cluster nodes and update configuration.
I would like to know if you have different solutions to this problem.
VPC support has now been added for Elasticache in Cloudformation Templates.
To launch a AWS::ElastiCache::CacheCluster in your VPC, create a AWS::ElastiCache::SubnetGroup that defines which subnet in your VPC you want Elasticache and assign it to the CacheSubnetGroupName property of AWS::ElastiCache::CacheCluster.
You workaround is a reasonable one (and shows that you seem to be in control of your AWS operations already).
You could improve on your custom solution eventually by means of the dedicated CustomResource type, which are special AWS CloudFormation resources that provide a way for a template developer to include resources in an AWS CloudFormation stack that are provided by a source other than Amazon Web Services. - the AWS CloudFormation Custom Resource Walkthrough provides a good overview of what this is all about, how it works and what's required to implement your own.
The benefit of using this facade for a custom resource (i.e. the Amazon ElastiCache cluster in your case) is that its entire lifecycle (create/update/delete) can be handled in a similar and controlled fashion just like any officially supported CloudFormation resource types, e.g. resource creation failures would be handled transparently from the perspective of the entire stack.
However, for the use case at hand you might actually just want to wait for official support becoming available:
AWS has announced VPC support for ElastiCache in the context of the recent major Amazon EC2 Update - Virtual Private Clouds for Everyone!, which boils down to Default VPCs for (Almost) Everyone.
We want every EC2 user to be able to benefit from the advanced networking and other features of Amazon VPC that I outlined above. To enable this, starting soon, instances for new AWS customers (and existing customers launching in new Regions) will be launched into the "EC2-VPC" platform. [...]
You don’t need to create a VPC beforehand - simply launch EC2
instances or provision Elastic Load Balancers, RDS databases, or
ElastiCache clusters like you would in EC2-Classic and we’ll create a
VPC for you at no extra charge. We’ll launch your resources into that
VPC [...] [emphasis mine]
This update sort of implies that any new services will likely be also available in VPC right away going forward (else the new EC2-VPC platform wouldn't work automatically for new customers as envisioned).
Accordingly I'd expect the CloudFormation team to follow suit and complete/amend their support for deployment to VPC going forward as well.
My solution for this has been to have a controller process that polls a message queue, which is subscribed to the SNS topic which I notify CloudFormation events to (click advanced in the console when you create a CloudFormation stack to send notifications to an SNS Topic).
I pass the required parameters as tags to AWS::EC2::Subnet and have the controller pick them up, when the subnet is created. I execute the set up when a AWS::CloudFormation::WaitConditionHandle is created, and use the PhysicalResourceId to cURL with PUT to satisfy a AWS::CloudFormation::WaitCondition.
It works somewhat, but doesn't handle resource deletion in ElastiCache, because there is no AWS::CloudFormation::WaitCondition analogue for stack deletion. That's a manual operation procedure wth my approach.
The CustomResource approach looks more polished, but requires an endpoint, which I don't have. If you can put together an endpoint, that looks like the way to go.