Network Newbie here.
Background
I am trying to use the AWS Command Line utility to connect to my instances on the AWS cloud.
Problem
When I try a simple command (from either local machine or EC2) such as...
aws ec2 describe-instances
I get...
Could not connect to the endpoint URL: "https://ec2.sa-east-1a.amazonaws.com/"
Which is fine and dandy because according to the AWS Command Line User Manual I should take note of:
The AWS CLI makes API calls to services over HTTPS. Outbound connections
on TCP port 443 must be enabled in order to perform calls.
But what that is suppose to mean in the first place? I find the above statement as clear as a glass of muddy water.
1) Should I go to the 'Security Group' of my instance and set inbound rules to 'HTTPS TCP 443 0.0.0.0/0'? (Doesn't work)
2) Or set outbound rules and set 'HTTPS TCP 443 0.0.0.0/0' ? (Doesn't work)
3) Does the above statement has anything to do with the 'Security Groups'?
4) If it doesn't has to then, what should I do in my machine in order to 'enable the port 443. That being said...
5) Should I RUN the AWS Command Line utility from my local machine or from my EC2 remote server?
I'd appreciate some guidance so I can start solving my problem.
Thanks in advance!
You have your region configured as "sa-east-1a", but the region is actually "sa-east-1". Your availability zones are sa-east-1a, sa-east-1b, etc.
Related
I installed dataiku on GCP instance.
It works perfectly fine on http protocol.
Now I am trying to allow https traffic.
I followed this official doc: https://doc.dataiku.com/dss/latest/installation/custom/advanced-customization.html#configuring-https
But after doing all the mentioned steps, I have an ERR_CONNECTION_REFUSED on my browser when doing: https://MyDataikuInstanceIp:1024
Is something more required on GCP in order to allow https traffic on the instance ?
Thanks.
In order to allow https and http traffic on a GCP instance, you should follow the next steps:
Navigate to Compute Engine > VM Instances.
Select your instance. Go to Edit.
Scroll down and locate Firewalls. Check both "Allow HTTP traffic" and "Allow HTTPS traffic".
Save your instance.
To review whether the firewall rule is enabled or disabled, you can use the following command:
gcloud compute firewall-rules describe [FIREWALL-NAME]
Also, if you have created a network tag in your instance, you will need to associate it to your firewall rule to match the target.
In the following link, you will find more information about how the firewall rules work in GCP
In addition as a best practice check if the firewall on your client is allowing connections through the TCP ports 443,1024 and the certificate from your server domain was allowed in your client.
I have a docker image running on EC2 instance that tries to issue an AWS command when I get this error
botocore.exceptions.ConnectTimeoutError: Connect timeout on endpoint URL: "https://ec2.us-east-2.amazonaws.com/"
If I give the EC2 instance Security Group that can access everything it works. But I look for a more limited set of rules. I already tried to give it only ports 443 and 80 (both ingress and outgress) but it doesn't work.
Any other configuration I should try?
Thanks,
I am trying to connect to Neptune DB in AWS Instance from my local machine in office, like connecting to RDS from office. Is it possible to connect Neptune db from local machine? Is Neptune db publicly available? Is there any way a developer can connect Neptune db from office?
Neptune does not support public endpoints (endpoints that are accessible from outside the VPC). However, there are few architectural options using which you can access your Neptune instance outside your VPC. All of them have the same theme: setup a proxy (EC2 machine, or ALB, or something similar, or a combination of these) that resides inside your VPC, and make that proxy accessible from outside your VPC.
It seems like you want to talk to your instance purely for development purposes. The easiest option for that would be to spin up an ALB, and create a target group that points to your instance's IP.
Brief Steps (These are intentionally not in detail, please refer to AWS Docs for detailed instructions):
dig +short <your cluster endpoint>
This would give you the current master's IP address.
Create an ALB (See AWS Docs on how to do this).
Make your ALB's target group point to the IP Address obtained for step #1. By the end of this step, you should have an ALB listening on PORT-A, that would forward requests to IP:PORT, where IP is your database IP (from Step 1) and PORT is your database port (default is 8182).
Create a security group that allows inbound traffic from everywhere. i.e. Inbound TCP rule for 0.0.0.0 on PORT-A.
Attach the security group to your ALB
Now from your developer boxes, you can connect to your ALB endpoint at PORT-A, which would internally forward the request to your Neptune instance.
Do checkout ALB docs for details around how you can create it and the concepts around it. If you need me to elaborate any of the steps, feel free to ask.
NOTE: This is not a recommended solution for a production setup. IP's used by Neptune instances are bound to change with failovers and host replacements. Use this solution only for testing purposes. If you want a similar setup for production, feel free to ask a question and we can discuss options.
As already mentioned you can't access directly outside your VPC.
The following link describes another solution using a SSH tunnel: connecting-to-aws-neptune-from-local-environment.
I find it much easier for testing and development purpose.
You can create the SSH tunnel with Putty as well.
Reference: https://github.com/M-Thirumal/aws-cloud-tutorial/blob/main/neptune/connect_from_local.md
Connect to AWS Neptune from the local system
There are many ways to connect to Amazon Neptune from outside of the VPC, such as setting up a load balancer or VPC peering.
Amazon Neptune DB clusters can only be created in an Amazon Virtual Private Cloud (VPC). One way to connect to Amazon Neptune from outside of the VPC is to set up an Amazon EC2 instance as a proxy server within the same VPC. With this approach, you will also want to set up an SSH tunnel to securely forward traffic to the VPC.
Part 1: Set up a EC2 proxy server.
Launch an Amazon EC2 instance located in the same region as your Neptune cluster. In terms of configuration, Ubuntu can be used. Since this is a proxy server, you can choose the lowest resource settings.
Make sure the EC2 instance is in the same VPC group as your Neptune cluster. To find the VPC group for your Neptune cluster, check the console under Neptune > Subnet groups. The instance's security group needs to be able to send and receive on port 22 for SSH and port 8182 for Neptune. See below for an example security group setup.
Lastly, make sure you save the key-pair file (.pem) and note the directory for use in the next step.
Part 2: Set up an SSH tunnel.
This step can vary depending on if you are running Windows or MacOS.
Modify your hosts file to map localhost to your Neptune endpoint.
Windows: Open the hosts file as an Administrator (C:\Windows\System32\drivers\etc\hosts)
MacOS: Open Terminal and type in the command: sudo nano /etc/hosts
Add the following line to the hosts file, replacing the text with your Neptune endpoint address.
127.0.0.1 localhost YourNeptuneEndpoint
Open Command Prompt as an Administrator for Windows or Terminal for MacOS and run the following command. For Windows, you may need to run SSH from C:\Users\YourUsername\
ssh -i path/to/keypairfilename.pem ec2-user#yourec2instanceendpoint -N -L 8182:YourNeptuneEndpoint:8182
The -N flag is set to prevent an interactive bash session with EC2 and to forward ports only. An initial successful connection will ask you if you want to continue connecting? Type yes and enter.
To test the success of your local graph-notebook connection to Amazon Neptune, open a browser and navigate to:
https://YourNeptuneEndpoint:8182/status
You should see a report, similar to the one below, indicating the status and details of your specific cluster:
{
"status": "healthy",
"startTime": "Wed Nov 04 23:24:44 UTC 2020",
"dbEngineVersion": "1.0.3.0.R1",
"role": "writer",
"gremlin": {
"version": "tinkerpop-3.4.3"
},
"sparql": {
"version": "sparql-1.1"
},
"labMode": {
"ObjectIndex": "disabled",
"DFEQueryEngine": "disabled",
"ReadWriteConflictDetection": "enabled"
}
}
Close Connection
When you're ready to close the connection, use Ctrl+D to exit.
Hi you can connect NeptuneDB by using gremlin console at your local machine.
USE THIS LINK to setup your local gremlin server, it works for me gremlin 3.3.2 version
Only you have to update the remote.yaml as per your url and port
I am supposed to install Google cloud SDK on a secured windows server where even port for http(80) and https(443) is not enabled.
What are the ports to be opened to work with gcloud, gsutil and bq commands?
I tested the behaviour in my machine, I expected to need merely port 443 because Google Cloud SDK is based on HTTPS Rest API calls.
For example you can check what is going on behind the scenes with the flag --log-http
gcloud compute instances list --log-http
Therefore you need an egress rule allowing TCP:443 egress traffic.
With respect to the ingress traffic:
if your firewall is smart enough to recognise that since you opened the connection it should let the traffic pass (most common scenario) and therefore you do not need any rule for the incoming.
Otherwise you will need as well to allow TCP:443 incoming traffic.
Update
Therefore you will need to be able to open connection toward:
accounts.google.com:443
*.googleapis.com:443
*:9000 for serialport in case you need this feature
Below error shows it is 443
app> gcloud storage cp C:\Test-file6.txt gs://dl-bugcket-dev/
ERROR: (gcloud.storage.cp) There was a problem refreshing your current auth tokens: HTTPSConnectionPool(host='sts.googleapis.com', port=443): Max retries exceeded with url: /v1/token (Caused by NewConnectionError...
If you run netstat -anb at same time you run any gcloud command which need remote connection, you will also see below entry for the app you are using. In my case PowerShell
[PowerShell.exe]
TCP 142.174.184.157:63546 40.126.29.14:443 SYN_SENT
Do not use any proxy to see above entry else gcloud will connect to proxy and you can't see actual port. you can do this by creating new config.
gcloud config configurations create no-proxy-config
I'm trying to access Tensorboard on AWS. Here is my setting :
Tensorboard : tensorboard --host 0.0.0.0 --logdir=train :
Starting TensorBoard b'39' on port 6006 (You can navigate to
http://172.31.18.170:6006)
AWS Security groups (in):
HTTPS TCP 443 0.0.0.0/0
Custom_TCP TCP 6006 0.0.0.0/0
However connecting to ec2-blabla.us-west-1.compute.amazonaws.com:6006 I can't see anything, I basically can't connect.
Do you have any idea?
You can use ssh tunneling technique.
In your terminal:
ssh -i /path/to/your/AWS/key/file -NL 6006:localhost:6006 user#host
where:
user and host: your aws ec2 user and instance specific.
-N: don't execute a remote command (just forward ports)
-L: [bind_address:]port:host:hostport
After that, browse to http://localhost:6006/
Run tensorboard in your ec2 terminal (you can custom logdir and port)
tensorboard --logdir=data/model --port=8080
Find your workstations public ip (a.b.c.d) address by visiting http://ip4.me/
Access the security group configuration assigned to your EC2 and add a custom TCP rule to your inbound traffic.
Outbound should be set to allow traffic from tensorboard port. (In this case 8080). Or you just allow all outgoing traffic from your EC2 instance
Protocol Port Range Destination Description
All traffic All All 0.0.0.0/0
Use your public DNS to access tensorboard from your workstation
http://ec2-xx-xxx-xx-xx.compute-1.amazonaws.com:8080/
Fast (but unsecure) solution:
Run:
tensorboard --logdir=/training --host=0.0.0.0 --port=8080
on your AWS instance.
Make sure that both your inbound and outbound rules on AWS console (control center) are as unrestricted as possible (allow all types, all ports etc.). However, keep in mind that this solution is not recommendable for environments requiring security (in our case, we didn't consider security for training an NN).
An attempt to explain why this works: when the policy is set as described, AWS still seems to prohibit inbound/outbound connections on the standard tensorboard port 6006. This does not seem to apply to the port 8080.
Long (but more secure) solution:
See: https://blog.altoros.com/getting-started-with-a-cpu-enabled-tensorflow-instance-on-aws.html
(provides explanations for setting ports correctly on AWS)
I managed to set it up like this:
Go to security groups in your ec2 console:
Choose the relevant security group in the table, click edit.
Add a rule like this:
Start tensorboard: tensorboard --logdir tf_summary/ --port 8080
Find out the URL of your instance and visit http://yourURL:8080
Simply run the tensorboard without the host parameter (which poses restrictions)
tensorboard --logdir XXX --port 6006
I suffered from the same problem for several days.
Fortunately I solved this issue by adding rule on "AWS Outbound rule" as if I had added "AWS Inbound rule".
Regardless of this setting, it works at home.
The same error is still happening only in the company.