I am new to redshift. Currently, I am able to create a redshift cluster and connect it through SQL Workbench but I am looking forward to tunnel my redshift cluster doing ssh from my MAC terminal. I did some research and able to create an ec2 instance with same VPC ID and subnet group which I am using to create my Redshift cluster with. I have already installed psql on my ec2 instance as well. I am not able to understand where I am going wrong when I use psql command to connect to redshift :
psql -h my redshift endpoint -p 5439 -d database name -U user -c " my query "
it gives me error psql: could not translate host name "my redshift endpoint" to address: Name or service not known
The first step is to tunnel to the EC2 instance using ssh, with a command that forwards a local port to a remote port:
ssh -i KEYPAIR.pem -L 5439:REDSHIFT-ENDPOINT:5439 ec2-user#EC2-PUBLIC-IP
Where:
KEYPAIR.pem should be the name of the keypair used to access the EC2 instance
REDSHIFT-ENDPOINT is the DNS name of the Redshift endpoint
EC2-PUBLIC-IP is the IP address of the EC2 instance
This command says:
Create an ssh connection using the keypair
Forward any traffic sent to local port 5439 to the remote machine, then have the remote machine send that traffic to REDSHIFT-ENDPOINT:5439 (substitute your endpoint for REDSHIFT-ENDPOINT)
Then, you can connect to Redshift on localhost:5439 as if it were running on your own computer. That traffic will be sent to the remote machine, which will send it to REDSHIFT-ENDPOINT:5439.
For example, if you want to use psql to connect to Redshift, use:
psql -h localhost -p 5439 -U <username>
try psql -h localhost -p 5439 -d -U -c " my query " or psql -p 5439 -d -U -c " my query ", you can't use switch -h without an argument, which is localhost by default if you don't use it
Thanks John Rotenstein on giving me insight, I was actually missing the Inbound rule associated with my security group to only allow traffic coming from redshift private IP. Both of my ec2 instance and redshift were in the same VPC so was supposed to use private instead of public IP. Also forgot to put ssh public key of redshift in the ec2 authorized_key file. Once I did that it worked.
Related
I'm struggling trying to open an ssh tunnel to access an RDS MySql instance through an EC2 bastion host. Using desktop clients (Navicat, MysqlWorkBench) with ssh tunnel set everything works as expected but when I run ssh -i keys.pem user#ec2-instance -L 3307:rds-mysql-instance:3306 -N in my terminal the command hangs indefinitely.
I can access my EC2 instance using ssh -i keys.pem user#ec2-instance and from my EC2 instance I can access the RDS database
Am I missing something in the configuration?
I also tried to open all ports on my Security Group just to be sure that it wasn't a port related issue.
Any help/idea?
Based on the comments.
To identify the issue, more verbose output from ssh can be requested using -v, -vv or even -vvv flags. Thus, the command for debugging can be:
ssh -i keys.pem user#ec2-instance -L 3307:rds-mysql-instance:3306 -N -vv
The detailed output allowed to identify the issue with the connection and fix it.
I'm trying to put use an RDS proxy to pool and share connections established with an RDS database with a PostgreSQL engine. The problem is I'm able to open a connection to the DB, both through an ECS instance or in PgAdmin, however, I'm not able to connect through the proxy. To attempt a connection through the proxy in PgAdmin, I'm using the proxy endpoint as opposed to the DB endpoint but the connection attempt times out.
I've successfully created the proxy and associated with my DB, both proxy and DB status is available. I've followed the example proxy setup and the DB and the proxy are using the same VPC security group.
Any ideas?
It seems to me that you are connecting to the proxy from outside of AWS. If this is the case, then its not possible to do this directly:
Your RDS Proxy must be in the same VPC as the database. The proxy can't be publicly accessible, although the database can be.
Your may be able to connect to RDS since it publicly accessible. RDS proxy on the other hand, can only be access from within the same VPC, e.g., from an instance.
Therefor, the solution is to setup an instance in the same VPC as your RDS and proxy. The instance must be accessible using SSH.
On the instance, you can run pgadmin4, in docker:
docker run --rm -p 8080:80 \
-e 'PGADMIN_DEFAULT_EMAIL=user#domain.com' \
-e 'PGADMIN_DEFAULT_PASSWORD=Fz77T8clJqJ4XQrQunGA' \
-d dpage/pgadmin4
The command, after setting up the docker, will server pgadmin4 on port 8080 on the instance.
You can check on the instance if its working:
curl localhost:8080
which can give the following indicating that its working:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>Redirecting...</title>
<h1>Redirecting...</h1>
<p>You should be redirected automatically to target URL: /login?next=%2F. If not click the link
However, since your instance is only accessible through ssh (port 22), to access it from your local workstation, you have to establish an ssh tunnel:
ssh -i <private-key> -L 8080:localhost:8080 -N ubuntu#<public-instance-ip> -v
In the above, my instance was Ubuntu. For Amazon Linux 2, the user would be ec2-user.
The tunnel will forward port 8080 from the instance (i.e. pgadmin4) to your local workstation on port 8080.
Then you just point your browser to localhost:8080 and you should see the pgadmin4 welcome screen.
P.S. My RDS and proxy settings used for the verification:
I currently have an EC2 instance that I am using to host my mongodb sever on from port 27017. Previously I had just setup the security group to just use my home IP address to authenticate a TCP connection to port 27017, however I no longer have a static IP. I now have one that changes everyday that I cannot control. Is there a way to create a mongo URI like mongolabs has
mongodb://<username><passs>#<my EC2 IP>:27017/db
that I can use to connect from PyMongo.
There are many, many guides available by searching that describe how to enable MongoDB authentication.
Alternatively, you could create a small script that uses the AWS CLI to update the security group with your current IP address. The script could be run when needed or set to run automatically your computer starts or you log in.
Install AWS CLI on your machine. You should have proper IAM permissions to update the security group. Then you can use below bash script to update your security group with your current IP address.
#!/bin/bash
ip = 'curl -s http://whatismijnip.nl |cut -d " " -f 5'
sleep 5
aws ec2 authorize-security-group-ingress --group-name MySecurityGroup --protocol tcp --port 22 --cidr $ip/24
I'm new to AWS and I'm having trouble connecting to my RDS instance from my EC2 instance. I think both are sitting within the same VPC. I can use SSH2 from putty to connect to my EC2 instance and I can use eclipse to connect to my RDS instance. However, I can't see to connect to the RDS from the EC2. When I try the following command:
ec2 command prompt> mysql -h endpoint -P 3306 -u user -p password
I get the following error:
Enter password:
ERROR 1049 (42000): Unknown database 'password'
Any ideas what I'm doing wrong? Also, why is it asking me for my password again? I've already specified it in my connection string.
Any help would be appreciated - thanks!
Okay - I was able to figure out what was wrong. It seems that I should be using the following to connect to the RDS:
mysql -u username -p -h endpoint databaseName
This prompts me to enter in my password, which then connects me to my database above in the RDS instance that I have set up. Evidently, you have to specify the database that you want to connect to and the port number is optional. I was able to connect both with it and without it. The documentation on AWS that I've read has never actually had the database name, which is what was throwing me off. Hope this can help someone else.
I'm able to connect to an ElastiCache Redis instance in a VPC from EC2 instances. But I would like to know if there is a way to connect to an ElastiCache Redis node outside of Amazon EC2 instances, such as from my local dev setup or VPS instances provided by other vendors.
Currently when trying from my local set up:
redis-cli -h my-node-endpoint -p 6379
I only get a timeout after some time.
SSH port forwarding should do the trick. Try running this from you client.
ssh -f -N -L 6379:<your redis node endpoint>:6379 <your EC2 node that you use to connect to redis>
Then from your client
redis-cli -h 127.0.0.1 -p 6379
It works for me.
Please note that default port for redis is 6379 not 6739. And also make sure you allow the security group of the EC2 node that you are using to connect to your redis instance into your Cache security group.
Also, AWS now supports accessing your cluster more info here
Update 2018
The previous answer was accurate when written, however it is now possible with some configuration to access redis cache from outside using the directions according to Accessing ElastiCache Resources from Outside AWS
Old Answer
No, you can't without resorting to 'tricks' such as a tunnel, which maybe OK for testing but will kill any real benefit of using a super-fast cache with the added latency/overhead.
The Old FAQ under How is using Amazon ElastiCache inside a VPC different from using it outside?:
An Amazon ElastiCache Cluster, inside or outside a VPC, is never allowed to be accessed from the Internet
However, this language has been removed in the current faq
These answers are out of date.
You can access elastic-cache outside of AWS by following these steps:
Create a NAT instance in the same VPC as your cache cluster but in a
public subnet.
Create security group rules for the cache cluster and
NAT instance.
Validate the rules.
Add an iptables rule to the NAT
instance.
Confirm that the trusted client is able to connect to the
cluster.
Save the iptables configuration.
For a more detailed description see the aws guide:
https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/accessing-elasticache.html#access-from-outside-aws
Not so old question, I ran to the same issue myself and solved it:
Sometimes, for developing reasons you need to access from outside (to avoid multi-deployments just for a simple bug-fix maybe?)
Amazon have published a new guide that uses the EC2 as proxies for the outside world:
https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/accessing-elasticache.html#access-from-outside-aws
Good luck!
BTW if anyone wants a windows EC2 solution, try these at the DOS prompt (on said windows EC2 machine):
To Add port-forwarding
C:\Users\Administrator>netsh interface portproxy add v4tov4 listenport=6379 listenaddress=10.xxx.64.xxx connectport=6379 connectaddress=xxx.xxxxxx.ng.0001.use1.cache.amazonaws.com
To list port-forwarded ports
C:\Users\Administrator>netsh interface portproxy show all
Listen on ipv4: Connect to ipv4:
Address Port Address Port
10.xxx.128.xxx 6379 xxx.xxxxx.ng.0001.use1.cache.amazonaws.com 6379
To remove port-forwarding
C:\Users\Administrator>netsh interface portproxy delete v4tov4 listenport=6379 listenaddress=10.xxx.128.xxx
We are using HAProxy as a reserved proxy server.
Your system outside AWS ---> Internet --> HAProxy with public IP --> Amazon Redis (Elasticache)
Notice that there is another good reason to do that (at that time)
As we use node.js client, which don't support Amazon DNS fail over, the client driver don't support dns look up again.
If the redis fail, the client driver will keep connect to the old master, which is slave after failed over.
By using HAProxy, it solved that problem.
Now using the latest ioredis driver, it support amazon dns failover.
This is a solid node script that will do all the dirty work for you. Tested and verified it worked.
https://www.npmjs.com/package/uzys-elasticache-tunnel
How to use
Usage: uzys-elasticache-tunnel [options] [command]
Commands:
start [filename] start tunneling with configuration file (default: config.json)
stop stop tunneling
status show tunneling status
Options:
-h, --help output usage information
-V, --version output the version number
Usage Example
start - uzys-elasticache-tunnel start ./config.json
stop - uzys-elasticache-tunnel stop
status - uzys-elasticache-tunnel status
Its is not possible to directly access the classic-cluster from a VPC instance. The workaround would be configuring NAT on the classic instance.
NAT need to have a simple tcp proxy
YourIP=1.2.3.4
YourPort=80
TargetIP=2.3.4.5
TargetPort=22
iptables -t nat -A PREROUTING --dst $YourIP -p tcp --dport $YourPort -j DNAT \
--to-destination $TargetIP:$TargetPort
iptables -t nat -A POSTROUTING -p tcp --dst $TargetIP --dport $TargetPort -j SNAT \
--to-source $YourIP
iptables -t nat -A OUTPUT --dst $YourIP -p tcp --dport $YourPort -j DNAT \
--to-destination $TargetIP:$TargetPort
I resolved using this amazon docs it says you ll have to install stunnel in your another ec2 machine.
https://aws.amazon.com/premiumsupport/knowledge-center/elasticache-connect-redis-node/