I want to connect to a database running in different cloud provider and it is exposed publicly.
I need to connect to that database from sagemaker notebook instance.
But the public ip of the sagemaker notebook instance needs to be whitelisted on the other side.
Is it possible to attach elastic ip to sagemaker notebook instance as I don't see any option to attach eip to sagemaker notebook instance?
No, it is not possible to assign a SageMaker notebook an Elastic IP, which is a disappointment. This missing feature makes the SageMaker product a lot more difficult to use with many sources of data, limiting its utility.
Official Amazon Answer
From the AWS SageMaker product forums on Dec 12, 2019: Possible to attach Elastic IP to sagemaker notebook instance?
Question> Is it possible to attach elastic ip to sagemaker notebook instance?
Answer> We are always re-evaluating our backlog of features based on customer requests,
so we appreciate the feedback on this feature.
You might want to start a new thread or chime in on that one if you want them to add this feature.
Possible Solutions
A general strategy for using a particular IP to access a resource would be to setup a proxy machine and authorize its IP and use it as a proxy to access your service. How hard this is depends on what you are doing - for S3 it doesn't seem possible - but for web-based requests this shouldn't be too hard. For AWS services you can use a proxy.
Personally I am trying to access Algoseek's requestor-pays S3 buckets directly from SageMaker notebooks and this isn't possible. I looked at setting up a proxy but can't figure out how. Instead I will copy the S3 data each time they add a day into our own S3 bucket.
In my case, I have whitelisted the NAT Gateway's IP in the external database.
EDIT: This works only for private subnets.
Related
I can't figure out how to make them talk using API calls. Previously I used API Gateways which would trigger lambdas and that lambdas would interact with dynamodb and other services and send me back json response. Now I want to shift to EC2 instances and totally skip API gateway usage. And let a server I run in ec2 do the computation for me. Do I need to deploy a web service(DJango RESTFUL) in EC2 instance and then use it to call in my frontend? If yes I need little guidance how
And Suppose I want to access s3 storage from my DJango restufl in EC2. Can I do it without having to enter the access key and ID and use roles instead just like how I would access s3 from the ec2 instance without access key and ID. Traditionally with SDK we have to use access key and secret keys to even get authorized to use services in SDK so I was wondering if there was a way to get over this since the program will be running in EC2 instance itself. One really inefficient way will be to run a batch command that makes the EC2 interact with services I need without SDK and with roles instead but It is really inefficient and too much work as far as I can see.
As you are familiar with API Gateway, you can use the same to connect to your EC2 instance, its private integration, with the use of VPC Links.
You can create an API Gateway API with private integration to provide your customers access to HTTP/HTTPS resources within your Amazon Virtual Private Cloud (Amazon VPC). Such VPC resources are HTTP/HTTPS endpoints on an EC2 instance behind a Network Load Balancer in the VPC.
You can go though this document for step by step integration.
If you do not want to use API gateway any more, then you can simply use Route53 to route traffic to EC2 instance, all you need is the IP address of the EC2 instance and a hosted zone created using Route53.
Here is a tutorial for your reference.
I'm working with AWS and need some support please.
My team provisioned Direct Connect and we can now enjoy private connectivity from our corporate network to VPC on AWS.
Management is asking if it's possible that aws cli commands are executed through Direct Connect and not through the public internet. Indeed, we have a lot of scripts with a lot of commands like aws ec2 describe-instances and so on. I guess these calls the public REST API of EC2 service that AWS exposes.
They're asking if it's possible that these calls do not go through the public internet.
I've seen VPC endpoints? Are they the solution?
See How can I access my Amazon S3 bucket over Direct Connect? for how to do this with S3.
Basically:
After BGP is up and established, the Direct Connect router advertises all global public IP prefixes, including Amazon S3 prefixes. Traffic heading to Amazon S3 is routed through the Direct Connect public virtual interface. The public virtual interface is routed through a private network connection between AWS and your data center or corporate network.
You can extend this to other Amazon services, per the AWS Direct Connect FAQs:
All AWS services, including Amazon Elastic Compute Cloud (EC2), Amazon Virtual Private Cloud (VPC), Amazon Simple Storage Service (S3), and Amazon DynamoDB can be used with Direct Connect.
Refer to #jarmod's answer below for the answer to the question but read on for why I think this sounds like an XY problem.
There is no reason at all why management should be concerned.
Third-party auditors assess the security and compliance of AWS services as part of multiple AWS compliance programs. Using the AWS CLI to access a service does not alter that service's compliance - AWS has compliance programs which pretty much cover every IT compliance framework out there globally.
Compliance aside, the AWS CLI does not store any customer data (there should be no data protection concerns) & transmits data securely (unless you manually override this).
The user guide highlights this:
The AWS CLI does not itself store any customer data other than the credentials it needs to interact with the AWS services on the user's behalf.
By default, all data transmitted from the client computer running the AWS CLI and AWS service endpoints is encrypted by sending everything through a HTTPS/TLS connection.
You don't need to do anything to enable the use of HTTPS/TLS. It is always enabled unless you explicitly disable it for an individual command by using the --no-verify-ssl command line option.
As if that's not enough, you can also add increased security when communicating with AWS services by enforcing a minimum version of TLS 1.2 to be used by the CLI.
There should be targeting of much much bigger attack vectors, like:
The physical accessibility of the device storing the credentials
Permanent access tokens vs. temporary credentials
IAM policies associated with the credentials
The AWS CLI is secure.
I recently realised my NEXT JS project I deployed on AWS Amplify uses Lambda but I need to deploy it on EC2. Is this possible at all?
I'm new to this whole thing so excuse the ignorance but for certain reasons I need to use EC2?
Is that possible?
Thanks
AWS EC2 is a service that provides all the compute, storage, and networking needs you may have for any application you want to develop. From its site:
Amazon EC2 offers the broadest and deepest compute platform with a choice of processor, storage, networking, operating system, and purchase model.
Source
Basically, you can create any number of virtual machines, connected among themselves and to the Internet however you like; and use any data persistence strategy.
There are many things to unpack when using EC2, but to start, I would suggest that you learn how to set up an EC2 instance using the default VPC that comes with your account. Be sure to configure the instance to have a public IP so you can access it through the Internet. Once inside, you can deploy your application however you like and access it through your public IP.
Before moving on, trying to decide why you need your app to run on EC2, Lambda is a SaaS (Software as a Service) product, meaning that all of the service provider's infrastructures are managed. On the other hand, EC2 is an IaaS product (Infrastructure as a Service) which means that you have to handle most of the infrastructure.
This is more of an architectural question rather than coding issue.
Please pardon me if I am in wrong place
I have an Ec2 instance running in private VPC where we in future are
going to deploy PII data and by no mean we can have internet access to it.
However We need to install ETL tool in docker(Airflow, docker, nifi, python etc)
and ofcourse need to ssh into from my local company vpc.
There is two approach as far as I think is
1. To create an another EC2 in public subnet and install all our tool there
and call the VPC EC2 from this one.
So that I can move the PII data to S3 through private Endpoint.
Cons: Does not it still raise the security concern as the EC2(ETL) is still
in internet where from one can access the PII data in second ec2.
Another Option
2. To create the Ec2 in public and install all tools and then
finally change it into private VPC.
Cons: in case if tool crash or there is any change needed then will have to
move it back to public which again does not look proper way of handling it.
I tried to search internet to get any tutorial or training about it.
But cannot find it.
Any suggestion will be highly appreciated.
You don't need to use internet at all if you don't want. I assume that by no internet access you mean that this is two ways - no access from the internet to the instance, nor the instance can connect internet at all (i.e. no NAT or any other proxy).
There are a couple of ways of doing this. One way is as follows:
Prepare custom AMI with per-installed all packages and software that you require.
Create private VPC without any public subnets.
Add VPC interface endpoints to the VPC for S3, AWS System Manager, ECR (to store your private docker images) and other AWS services you may require, e.g. KMS.
Launch your instance from the custom AMI in the private VPC.
Use SSM Session Manager to "ssh" to the instance without any internet access.
I think both approaches are inherently sub-optimal
If all you're trying to do is avoid exposing your compute instances to the internet, and your setup is docker based, simply setup your own docker repository, either using ECS or Sonatype Nexus (on another server), upload your docker images there and have that node use that ECS/Nexus as its docker registry.
That way, your enjoying free access to all resources exposed as docker images while maintaining security compliance.
Is there a way to present onprem storage to AWS EC2 instances without copying to S3 etc. We have a storage array in our DC, I want to carve up LUNs and present them to our apps running on EC2 instances in AWS, using storage gateway. Can someone suggest if this is possible at all? I don't want to pay for AWS EBS volumes, when I have plenty of storage available with me. Thanks.
You will not be able to do this with Storage Gateway - it serves as an interface between your local network and S3.
If you want to expose your local disk to EC2, you will need to run a file share (NFS or Samba), set up a VPN or Direct Connect between your data center and your AWS VPC, and then mount the exported volumes on your EC2 instance.
If you don't want to pay for any EBS volumes, you should look for instance storage based AMIs. Keep in mind that you cannot stop these instances - you can only terminate them.
According to latest AWS cloud architecture and infrastructure developments, AWS cloud resources can access on premise storages. Not only computing instances but also AWS managed SaaS can do it same. Standard protocols such as NFS, SMB and iSCSI available and you can mount AWS computing instances to remote storage through site-to-site VPN.