What I'm trying to do
I am working on a lambda function which will simply register some metadata about files which are uploaded onto an s3 bucket. This is not about actually processing the data in the files yet. To start with, I just want to register the fact that certain files have been uploaded or not. Then I want to connect that metadata to QuickSight just so that we can have a nice visual about which files have been uploaded.
What I've done so far
This part is fairly easy:
Some simply python code with the pymysql module
Chalice to manage the process of creating and updating the lambda function
I created the database
Where I'm stuck
QuickSight is somehow external to AWS in general. So I had to create the RDS (mysql) in the DMZ of our VPC.
I have configured the security group so that the DB is accessible both from QuickSight and from my own laptop.
But the lambda function can't connect.
I configured the right policy for the role, so that the lambda can connect with IAM
I tested that policy with the simulator
But of course the lambda function is going to have some kind of dynamic IP and that needs to be in the security group
Any Ideas ??
I am even thinking about this right?
Two things.
You shouldn't have to put your RDS in a DMZ. See this article about granting QuickSight access to your RDS: https://docs.aws.amazon.com/quicksight/latest/user/enabling-access-rds.html
In order for a lambda to access something in a VPC (like and RDS instance) the lambda must have a VPC configuration. https://docs.aws.amazon.com/lambda/latest/dg/configuration-vpc.html
Related
As part of our workflow, we want to upload files to our Amazon EC2 instance automatically.
It's currently only allowing whitelisted IP ranges to connect over SSH. And since we are running Github actions, it seems odd to white list roughly 1500 IP ranges.
Does anyone have an intelligent solution for this?
SCP and/or rsync don't matter for us.
It's merely getting access that I need help with.
I have access to the ssh key, and I can get a hold of an admin to get temporary access to the AWS Console should I need it.
Since the EC2 instance is in a private network, the hurdles to get Github Actions ssh access to it are many.
I would work with a decoupled architecture. Have the GitHub action upload the files to S3.
Then
Lambda can load the file onto the ec2 instance - S3 trigger for Lambda
OR
Have a process running on the ec2 instance poll for new events on the s3 bucket per SNS - S3 polling
I have a simple C# Lambda function that inserts a record into a table using Entity Framework. When I run the test locally (from my desktop machine) I can connect to the remote database just fine and the record gets inserted into the table at AWS just fine. When I upload the lambda to AWS and then send it data the function times out after 15 seconds. Since the code runs on my (external) desktop machine I am assuming that Lambda does not have permissions to connect to the internal RDS database from inside aws.
I have added AmazonRDSFullAccess to the permissions of the Lambda function. The Lambda function still times out.
What am I missing?
The Lambda function needs to be deployed to the same VPC as the RDS server.
It does not need the AmazonRDSFullAccess IAM policy attached.
The security group for the RDS server needs to allow inbound connections from the security group assigned to the Lambda function.
I would like to use aws sam to setup my serverless application. I have used it with dynamoDB before. This was very easy to since all I had to do was setup a dynamoDB table as a resource and then link it to the lambda functions. AWS SAM seams to know where the table is located. I was even able ot run the functions on my local machine using the sam-cli.
With RDS its a lot harder. The RDS Aurora Instance I am using sits behind a specific endpoint, in a specific subnet with security groups in my vpc protected by specific roles.
Now from what I understand, its aws sams job to use my template.yml to generate the roles and organize access rules for me.
But I don't think RDS is supported by aws sam by default, which means I would either be unable to test locally or need a vpn access to the aws vpc, which I am not a massive fan of, since it might be a real security risk.
I know RDS proxies exist, which can be created in aws sam, but they would also need vpc access, and so they just kick the problem down the road.
So how can I connect my aws sam project to RDS and if possible, execute the lambda functions on my machine?
Here's what I've done:
I setup an Aurora Serverless MySql instance.
Created a security group for Cloud9 which allows me to access Aurora Serverless mysql.
Created a Lambda function which queries my db, I added the custom VPC and added the security group which lets me access Aurora Serverless (same one as Cloud9), the Lambda function works fine and can query my DB.
Before setting up Aurora Serverless, I had a RDS MySql instance which my Lambda functions could query, and had a little deploy script on my local machine which I ran to package my Lambda function's changes and uploaded them to their respective Lambda function. To setup CLI I just used this guide https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html.
Now that I have the security group in my Lambda function, AWS CLI doesn't let me upload the Lambda function, it gets stuck during the upload process. Note: I can still upload Lamda functions using the Lambda GUI in the AWS console.
Does anyone know what I can do to upload my Lambda functions using CLI again?
Here's a picture of where it gets stuck
I'm trying to insert records into a Postgres database in RDS from a Lambda function. My Node.js lambda function works correctly when run locally, but the database connection times out when run in AWS.
I've read several articles and tutorials which suggest that AWS Lambda functions cannot access RDS instances that are within a VPC. For example: http://ashiina.github.io/2015/01/amazon-lambda-first-impression/
Unfortunately; it seems I am unable to create an RDS instance that exists outside of a VPC. At this dropdown I would expect to be able to select an option for "No VPC" or something along those lines.
Has this option been removed? Perhaps I have missed a step?
You can create a publicly accessible RDS instance. Then you should be able to access it from anywhere, inside or outside AWS. I believe that would get around your issue with Lambda. You are asked if the instances needs to be publicly accessible when you create a new RDS instance via the web console.
Or you could just wait a few weeks, as Lambda within a VPC is supposed to be enabled "later this year".
Edit: Note that newer Amazon accounts are restricted to VPC only resources. You can't create EC2 or RDS instances outside of a VPC anymore. That's why you don't see the "No VPC" option anymore.
Second Edit: VPC access for Lambda functions is now genearally available.
This question is awhile back, but for those of you who are using MySQL, now you can connect AWS Lambda with Aurora Serverless without VPC, utilizing their new Data API. Take a look at this example for details https://coderecipe.ai/architectures/77374273