I was starting a neptune database from this base stack
https://s3.amazonaws.com/aws-neptune-customer-samples/v2/cloudformation-templates/neptune-base-stack.json
However now i am wondering why a NAT Gateway and also an Internet Gateway are started in this stack? are they required for updates within Neptune? This seems like a huge security risk.
On top of that these gateways are not cheap.
I would be happy for an explanation on this
The answer is no, it's not required, AWS just sneaked some unecessary costly ressources into the template..
Anyways if you want to use the updated template without NAT and IG GWs use this one that i just created https://neptune-stack-custom.s3.eu-central-1.amazonaws.com/base.json
I intend to use Google Cloud Functions to access an API. My goal is to use several functions, each with a different IP address. I would be distributing processing across several functions simultaneously that would then each interact with the target API. As I understand it, there's the possibility that the execution of two separate functions could be take place on the same machine - meaning requests would originate from the IP. In order to respect the rate limits, I need to know how many requests will be going through each IP address and therefore need to ensure that each function is executing with a separate IP.
I'm new to Google Cloud Functions, but I've made some progress. Currently, I have a function function-1. This function is using connector-1 and passing all egress traffic through my default VPC network. I followed the guide provided by Google Cloud for associating a static IP with my function. As a result, I now have router-1 which is connected with my NAT gateway nat-1. Finally, nat-1 has a static IP associated with it.
At this point, any execution of function-1 is using the static IP as expected. However, I'm still trying to understand the proper way of structuring this. I have a few questions on the matter:
Do I have to duplicate every link in the chain for each function that requires its own IP address?
Am I able to re-use some of these items? For example, perhaps all functions can use the same VPC network?
Is there a better way to structure things to meet my requirements assuming I needed 10 or 20 functions using different IPs?
The answers:
I'm not sure what you mean with "duplicate every link in the chain", but if you want to enforce each CF to have a static IP address you will have to follow the steps you shared.
Yes, you could re-use the VPC network and attach a new serverless VPC connector. Even in the same region.
If you want to force a different static IP for each CF, no, you need to follow these steps.
As a tip, you can use gcloud compute networks vpc-access connectors create to kind of automatize the connectors creation. It may be useful if you have to create many because it's faster than using the Console.
If this limitation does not suit your scenario you should wonder whether this is the appropriate product for you.
I am working on moving our business needs into the cloud, namely using AWS Simple Storage Service and Amazon Redshift. Because we are working with sensitive data we have some client concerns to consider. One question we will have to answer is whether or not the data in S3/Redshift is co-mingled with other data and show evidence that it is isolated.
While researching I found information about EC2 instances being shared on the same server unless the instance is specified as a dedicated instance. However, I been totally unable to find anything similar regarding other AWS services. Is the data in S3 and Redshift co-mingled as well?
Everything on cloud is co-mingled but with security boundaries unless you pay more to get dedicated service (like dedicated EC2 hosts) in which case you should stick with on-prem.
Your concerns of co-mingling falls under Shared Responsiblity Model where AWS is responsible to make sure your data is not accessible by other services running on their hosts unless you open up the access.
Read this article on how Shared Responsibility Model works
https://aws.amazon.com/compliance/shared-responsibility-model/
Or this whitepaper
https://d0.awsstatic.com/whitepapers/Security/AWS_Security_Best_Practices.pdf
We have an EC2 instance which for security reasons has no Internet access. But at the same time, the code running on that server needs to call some Lambda functions. It seems to me these two requirements are contradictory since without Internet access, the code can not call Lambda functions.
Does anyone have any suggestion on what are my options without sacrificing the security aspect of the project?
You won't be able to reach the AWS API's generally without internet access. Two exceptions are S3 and DynamoDB where you can create VPC endpoints and keep it completely on a private network. Some services can also be exposed through PrivateLink, but Lambda is not yet one of them.
You can learn more about those here: https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-endpoints.html
Depending on your security requirements, you might be able to use a NAT Gateway (https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-nat-gateway.html) or an Egress-Only Internet Gateway (https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/egress-only-internet-gateway.html)
Those would provide access from the instance to the internet, but without the reverse being true. In many cases, this will provide enough security.
Otherwise, you will have to wait for PrivateLink to support Lambda. You can see more on how to work with PrivateLink here: https://aws.amazon.com/blogs/aws/new-aws-privatelink-endpoints-kinesis-ec2-systems-manager-and-elb-apis-in-your-vpc/
I have a web app which runs behind Amazon AWS Elastic Load Balancer with 3 instances attached. The app has a /refresh endpoint to reload reference data. It need to be run whenever new data is available, which happens several times a week.
What I have been doing is assigning public address to all instances, and do refresh independently (using ec2-url/refresh). I agree with Michael's answer on a different topic, EC2 instances behind ELB shouldn't allow direct public access. Now my problem is how can I make elb-url/refresh call reaching all instances behind the load balancer?
And it would be nice if I can collect HTTP responses from multiple instances. But I don't mind doing the refresh blindly for now.
one of the way I'd solve this problem is by
writing the data to an AWS s3 bucket
triggering a AWS Lambda function automatically from the s3 write
using AWS SDK to to identify the instances attached to the ELB from the Lambda function e.g. using boto3 from python or AWS Java SDK
call /refresh on individual instances from Lambda
ensuring when a new instance is created (due to autoscaling or deployment), it fetches the data from the s3 bucket during startup
ensuring that the private subnets the instances are in allows traffic from the subnets attached to the Lambda
ensuring that the security groups attached to the instances allow traffic from the security group attached to the Lambda
the key wins of this solution are
the process is fully automated from the instant the data is written to s3,
avoids data inconsistency due to autoscaling/deployment,
simple to maintain (you don't have to hardcode instance ip addresses anywhere),
you don't have to expose instances outside the VPC
highly available (AWS ensures the Lambda is invoked on s3 write, you don't worry about running a script in an instance and ensuring the instance is up and running)
hope this is useful.
While this may not be possible given the constraints of your application & circumstances, its worth noting that best practice application architecture for instances running behind an AWS ELB (particularly if they are part of an AutoScalingGroup) is ensure that the instances are not stateful.
The idea is to make it so that you can scale out by adding new instances, or scale-in by removing instances, without compromising data integrity or performance.
One option would be to change the application to store the results of the reference data reload into an off-instance data store, such as a cache or database (e.g. Elasticache or RDS), instead of in-memory.
If the application was able to do that, then you would only need to hit the refresh endpoint on a single server - it would reload the reference data, do whatever analysis and manipulation is required to store it efficiently in a fit-for-purpose way for the application, store it to the data store, and then all instances would have access to the refreshed data via the shared data store.
While there is a latency increase adding a round-trip to a data store, it is often well worth it for the consistency of the application - under your current model, if one server lags behind the others in refreshing the reference data, if the ELB is not using sticky sessions, requests via the ELB will return inconsistent data depending on which server they are allocated to.
You can't make these requests through the load balancer, So you will have to open up the security group of the instances to allow incoming traffic from source other than the ELB. That doesn't mean you need to open it to all direct traffic though. You could simply whitelist an IP address in the security group to allow requests from your specific computer.
If you don't want to add public IP addresses to these servers then you will need to run something like a curl command on an EC2 instance inside the VPC. In that case you would only need to open the security group to allow traffic from some server (or group of servers) that exist in the VPC.
I solved it differently, without opening up new traffic in security groups or resorting to external resources like S3. It's flexible in that it will dynamically notify instances added through ECS or ASG.
The ELB's Target Group offers a feature of periodic health check to ensure instances behind it are live. This is a URL that your server responds on. The endpoint can include a timestamp parameter of the most recent configuration. Every server in the TG will receive the health check ping within the configured Interval threshold. If the parameter to the ping changes it signals a refresh.
A URL may look like:
/is-alive?last-configuration=2019-08-27T23%3A50%3A23Z
Above I passed a UTC timestamp of 2019-08-27T23:50:23Z
A service receiving the request will check if the in-memory state is at least as recent as the timestamp parameter. If not, it will refresh its state and update the timestamp. The next health-check will result in a no-op since your state was refreshed.
Implementation notes
If refreshing the state can take more time than the interval window or the TG health timeout, you need to offload it to another thread to prevent concurrent updates or outright service disruption as the health-checks need to return promptly. Otherwise the node will be considered off-line.
If you are using traffic port for this purpose, make sure the URL is secured by making it impossible to guess. Anything publicly exposed can be subject to a DoS attack.
As you are using S3 you can automate your task by using the ObjectCreated notification for S3.
https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html
https://docs.aws.amazon.com/cli/latest/reference/s3api/put-bucket-notification.html
You can install AWS CLI and write a simple Bash script that will monitor that ObjectCreated notification. Start a Cron job that will look for the S3 notification for creation of new object.
Setup a condition in that script file to curl "http: //127.0.0.1/refresh" when the script file detects new object created in S3 it will curl the 127.0.0.1/refresh and done you don't have to do that manually each time.
I personally like the answer by #redoc, but wanted to give another alternative for anyone that is interested, which is a combination of his and the accepted answer. Using SEE object creation events, you can trigger a lambda, but instead of discovering the instances and calling them, which requires the lambda to be in the vpc, you could have the lambda use SSM (aka Systems Manager) to execute commands via a powershell or bash document on EC2 instances that are targeted via tags. The document would then call 127.0.0.1/reload like the accepted answer has. The benefit of this is that your lambda doesn't have to be in the vpc, and your EC2s don't need inbound rules to allow the traffic from lambda. The downside is that it requires the instances to have the SSM agent installed, which sounds like more work than it really is. There's AWS AMIs already optimized with SSM agent stuff, but installing it yourself in the user data is very simple. Another potential downside, depending on your use case, is that it uses an exponential ramp up for simultaneous executions, which means if you're targeting 20 instances, it runs one 1, then 2 at once, then 4 at once, then 8, until they are all done, or it reaches what you set for the max. This is because of the error recovery stuff it has built in. It doesn't want to destroy all your stuff if something is wrong, like slowly putting your weight on some ice.
You could make the call multiple times in rapid succession to call all the instances behind the Load Balancer. This would work because the AWS Load Balancers use round-robin without sticky sessions by default, meaning that each call handled by the Load Balancer is dispatched to the next EC2 Instance in the list of available instances. So if you're making rapid calls, you're likely to hit all the instances.
Another option is that if your EC2 instances are fairly stable, you can create a Target Group for each EC2 Instance, and then create a listener rule on your Load Balancer to target those single instance groups based on some criteria, such as a query argument, URL or header.