I have a project in the AWS as an AMI image and its an elasticsearch depended. I have an elasticsearch in my local system. Is there anyway to connect the AMI with the elasticsearch in my local system?
To clarify - are you asking how to connect your self-hosted/on-premise ElasticSearch to an AWS EC2 instance?
If so, you can create a VPN connection between your on-premise Network to your AWS VPC using strategies outlined in the AWS documentation
You can also route the traffic via the Internet if you'd prefer - which depending on how your network looks may involve setting up some NAT port forwards.
Related
I have a redis instance on AWS that I want to connect using Redis Desktop Manager from my local machine
I am able to ssh into my ec2 instace and then run redis-cli -h host and connect to it.
But the same is not possible from my local machine.
I am sure there must be a way to monitor my redis using the GUI, and I think if I can connect to the ec2 using pem file and I can connect to redis from insde there, must be a way to combine both? And connect to the redis instance locally via my ec2 instace? Any ideas?
By design AWS EC domain is deployed for use only within AWS. From docs:
Elasticache is a service designed to be used internally to your VPC. External access is discouraged due to the latency of Internet traffic and security concerns. However, if external access to Elasticache is required for test or development purposes, it can be done through a VPN.
Thus, it can't be accessed directly from outside of your VPC. For this, you need to setup a VPN between your local home/work network and your VPC, or what is often easier to do for testing and development, establish a ssh tunnel.
For the ssh tunnel you will need a public proxy/bastion EC2 instance through which the tunnel will be established. There are number tutorials on how to do it for different AWS services. General procedures are same, whether this is ES, EC, Aurora Serverless or RDS Proxy. Some examples:
SSH Tunnels (How to Access AWS RDS Locally Without Exposing it to Internet)
How can I use an SSH tunnel to access Kibana from outside of a VPC with Amazon Cognito authentication?
As #Marcin mentioned, AWS recommends only using Elasticache within your VPC for latency reasons, but you've got to develop on it some how... (Please be sure to read #Marcin's answer)
AWS is a huge mystery, and it's hard to find beginner-intermediate resources, so I'll expand upon #Marcin's answer a little for those that might stumble across this.
It's pretty simple to set up what's often referred to as a "jump box" to connect to all sorts of AWS resources - this is just any EC2 instance that's within the same VPC (network) as the resource you're trying to connect to - in this case the Elasticache redis cluster. (If you're running into trouble, just spin up a new instance - t4g.nano or something super small works just fine.)
You'll want to make sure you're in the directory with your key, but then should be able to run the following command to link whatever port you'd like to use to the remote redis cluster:
ssh -i ${your_ssh_key_name.pem} ${accessible_ec2_host} -L ${port_to_use_locally}:${inaccessable_redis_or_other_host}:${inaccessable_redis_port}
Then you can use localhost and ${port_to_use_locally} to connect to redis
We are moving from AWS to the GCP. I used Client VPN Endpoint in AWS to get into the VPC network in the AWS. What is the alternative in GCP which I can quickly setup and get my laptop into the VPC network? If there is no exact alternative, what's the closest one and please provide instructions to set it up.
AWS Client VPN is a managed client-based VPN service that enables you to securely access your AWS resources and resources in your on-premises network. With Client VPN, you can access your resources from any location using an OpenVPN-based VPN client.
Currently there is no managed product available on GCP to allow VPN connections from multiple clients to directly access resources within a VPC as Cloud VPN only supports site-to-site connectivity, however there is an existing Feature Request for this.
As an alternative a Compute Engine Instance can be used instead with OpenVPN server manually installed and configured following the OpenVPN documentation, however this would be a self managed solution.
Before moving to Amazon Web Services, I was using Google Cloud Platform to develop my aplication, CloudSQL to be specific, and GCP have something called Cloud SQL Proxy that allows me to connect to my CloudSQL instance using my computer, instead of having to deploy my code to the server and then test it. How can I make the same thing using AWS?
I have a python environment on Elastic Beanstalk, that uses Amazon RDS.
AWS is deny be default so you cannot access an RDS instance outside of the VPC that your application is running in. With that being said... you can connect to the RDS instance via a VPN that can be stood up in EC2 that has rules open to the RDS instance. This would allow you to connect to the VPN on whatever developer machine and then access the RDS instance as if your dev box was in the VPC. This is my preferred method because it is more secure. Only those with access to the VPN have access to the RDS instance. This has worked well for me in a production sense.
The VPN provider that I use is https://aws.amazon.com/marketplace/pp/OpenVPN-Inc-OpenVPN-Access-Server/B00MI40CAE
Alternatively you could open up a hole in your VPC to the RDS instance and make it publicly available. I don't recommend this however because it will leave your RDS instance open to attack as it is publicly exposed.
You can expose your AWS RDS to the internet by proper VPC setting, I did it before.
But it has some risks
So usually you can use those ways to figure it out:
Create a local database server and restore snapshot from your AWS RDS
or use VPN to connect to your private subnet which hold your RDS
A couple people have suggested putting your RDS instance in a public subnet, and allowing access from the internet.
This is generally considered to be a bad idea, and should be the last resort.
So you have a couple of options for getting access to RDS in a private subnet.
The first option is to set up networking between your local network and your AWS VPC. You can do this with Direct Connect, or with a point-point VPN. But based on your question, this isn't something you feel comfortable with.
The second option is to set up a bastion server in the public subnet, and use ssh port forwarding to get local access to the RDS over the SSH tunnel.
You don't say if you on linux or Windows, but this can be accomplished on either OS.
What I did to solve was:
Go to Elastic Beanstalk console
Chose you aplication
Go to Configurations
Click on the endpoint of your database in Databases
Click on the identifier of your DB Instance
In security group rules click in the security groups
Click in the inbound tab
Click edit
Change type to All Traffic and source to Anywhere
Save
This way you can expose the RDS connected to your Elastic Beanstalk aplication to the internet, which is not recommended as people sugested, but it is what I was looking for.
I have a project setup where I can connect to a local resource through AppEngine Flexible instances launching on a VPC network that is setup with a Cloud VPN connection to my local firewall.
With the release of Serverless VPC for the us-east1 region, I wanted to replace my setup to use AppEngine Standard Gen2 instances vs Flexible for the cost savings. I setup a Serverless VPC for the region/network my AppEngine app is hosted on and my Cloud VPN connection is configured for, updated my app.yaml accordingly, and pushed a new version.
I keep getting timeout errors for the new version that is trying to use Serverless VPC to connect to my local resource.
Some context:
The VPC Network is named "portal" and setup to "Auto" mode (auto creation of subnets for each region)
Cloud VPN is setup as a Classic VPN in the "portal" network with Route-based routing in the us-east1 region, connecting to my remote local 192.168.11.0/24 subnet.
A route exists on the VPC network for destinations 192.168.11.0/24 to use the Cloud VPN I have setup as the next hop (automatically created)
With the above, AppEngine Flexible deployments on the "portal" network can connect to my local resource as can any other Compute Engine VM on the "portal" network
I setup the Serverless VPC connector on the us-east1 region with the subnet 10.8.0.0/28
I'm not too clear how Serverless VPC works so I'm not sure how to even begin troubleshooting. When I click on the route rule for the 192.168.11.0/24 destination, I can see the AppEngine Flexible instances listed along with some "serverless-vpc-access" tagged instances that appear to be on a different subnetwork but using 10.8.0.0/28 IPs.
Should this configuration be working? If not, what changes do I need to make in order to support this?
Your problem (most likely) is caused by static routing. Do you have a route for return traffic coming from your VPN going to the VPC connector? Look at the routes defined for the VPN.
The purpose of a Serverless VPC connector is to allow the connection from the App Engine Standard to your VPC Network since the App Engine Standard environment is hosted and managed by Google and is not part of your VPC Network.
More details can be found here: [https://cloud.google.com/vpc/docs/configure-serverless-vpc-access].
That being said, you should verify the following:
Make sure that you’ve added the new subnet (/28) to your local on premise routes, with your VPN Gateway as the next hop. Since you’re using route-based routing, there is nothing to do regarding the Traffic Selectors on the VPN.
Make sure your local firewall is configured to accept the connection back and forth with the new configuration (/28).
While this probably won't apply to you, I just wanted to point out that communication through the Serverless VPC connector to the App Engine Standard environment is not possible unless it’s done on the same original tcp connection that originated from that same App Engine (TCP Established).
Your configuration, as you described is definitely possible to achieve. As mentioned, there are only a few things you need to verify to make sure it works.
I will be using the PythonAnywhere PaaS service. It is using AWS on us-east-1 for all its EC2 instances.
They don't offer Redis and I plan to create a regular AWS Redis instance using Elasticache.
Ideally I would like to connect to the Redis instance from the paas using local internal network of AWS.
Do you know if this kind of setup is possible? That is benefit from local network between paas AWS hosted to regular EC2 instances?
(The paas allows external connections.)
Multiple AWS VPCs can connect to each other via VPC Peering. The VPCs must be in the same region and have non-overlapping IP addresses.
However, this requires changes within both VPCs to connect to each other. It is unlikely that PythonAnywhere would want to modify their configuration for you. (But, by all means, ask them!)
An alternative is to connect to your Redis cluster within your own VPC, but do it via a Public IP address. The traffic will travel between VPCs across the 'edge' of the Internet -- that is, it won't go on the Internet proper, it will just transit within AWS' network if it is in the same Region. So, it is using the "local internal network of AWS", but via a Public IP address.