I am trying to use Airflow for workflow management on my development machine on aws. I have multiple virtual environments setup and have installed airflow.
I am listening to port 8080 in my nginx conf as:
listen private.ip:8080;
I have allowed inbound connection to port 8080 on my AWS machine.
I am unable to access my airflow console as well as admin page from my public ip / website address.
You can just create a tunnel for viewing UI locally.
ssh -N -L 8080:ec2-machineip-compute-x.amazonaws.com:8080 YOUR_USERNAME_FOR_MACHINE#ec2-machineip-compute-x.amazonaws.com:8080
ssh -N -L 8080:ec2-machineip-compute-x.amazonaws.com:8080 YOUR_USERNAME_FOR_MACHINE#ec2-machineip-compute-x.amazonaws.com:8080
localhost:8080 for viewing airflow 8080 UI
Related
Problem:
When I try to connect from my local machine to a Redis VM hosted on Google Cloud, the connection is refused.
QUESTION: How can I connect to Redis installed on a private VM in GCP after successfully connecting to the VPC via a VPN?
Setup:
VM hosted in Google Cloud without public IP
Redis installed on VM by deploying the VM using the Bitnami package from GCP Marketplace
Firewall rule added to GCP targeting my VM, allowing TCP ingress traffic on port 6379 for all IP ranges
VPN setup using OpenVPN to tunnel into GCP VPC from local machine (Windows)
What I know:
Redis is running... if I SSH into my VM and run redis-cli, everything works as expected
VPN is working... from my local machine, I can successfully ping my VM when connected to the VPN
Redis config comments out the binding to 127.0.0.0 to (theoretically) open it up to all bindings, after which I restarted Redis on my VM (I think)
The password I'm using is correct
What doesn't work:
StackExchange.Redis:
var redis = ConnectionMultiplexer.Connect("my.ip.to.vm", config =>
{
config.Password = "my-redis-password";
});
Command Line from Local Machine (using redis-cli npm package):
rdcli -h my.ip.to.vm -a my-redis-password
Recap
What am I missing?
How can I connect to Redis installed on a private VM in GCP after successfully connecting to the VPC via a VPN?
Solution (this is embarrassing):
Stop VM
Start VM
I am trying to run my flask app on GCP instance. However the app gets published at local host of that instance. I want to access that instances localhost.
I saw couple of videos and article but almost all were about deploying app on GCP. Is there no simple way to just forward whatever is published on localhost of VM instance to my PC browser and If I submit some information in the app then it goes to VM instance and gives back the result to my local machine's browser via VM instances localhost.
You can use Local Port Forwarding when you ssh into the target instance hosted in GCP.
Local port forwarding lets you connect from your local machine to another server. To use local port forwarding, you need to know your destination server, source port and target port.
You should already know your destination server. The target port must be the one on which your flask app is listening. The source port can be any port that is not in use on your local computer.
Assuming flask app is listening on port 8080 on the GCP instance and you want to make the app available in your local computer on port 9876, ssh into your GCP instance using the following command:
ssh -L 9876:127.0.0.1:8080 <username>#<gcpInstanceIP>
Same result can be achieved using gcloud compute ssh if you don't have the ssh key on the target instance.
The -- argument must be specified between gcloud specific args on the left and SSH_ARGS on the right:
gcloud compute ssh <gcp-instance-name> --zone=<instance-zone> -- -L <source-port>:localhost:<target-port>
You can also use the Google Cloud Shell:
Activate Cloud Shell located at the top-right corner in the GCP Web Interface
ssh into your instance with Local Port Forwarding
gcloud compute ssh <gcp-instance-name> --zone=<instance-zone> -- -L 8080:localhost:<target-port>
Click the Web Preview in the Google Cloud Shell, the Preview on port 8080.
I installed Apache Spark and Zeppelin on an Ubuntu instance running on AWS. Zeppelin is starting fine and when I check the status it is OK:
sudo ./bin/zeppelin-daemon.sh status
Zeppelin is running [ OK ]
But I can not use the "ip address":8090 to see the zeppelin webpage and create notebooks. The IP address I am using is the public one AWS give me (and I changed the port to 8090 in zeppelin-site.xml).
Should I change the server address in the zeppelin-site.xml?
Zeppelin is indeed running on port 8090 on that server, but the port is not accessible externally -- that is, it can only be accessed from the server itself.
No fear! You can use port forwarding to connect to it.
Linux
ssh -i keypair.pem -L 8090:localhost:8090 user#<IP-ADDRESS>
This tells SSH to forward any requests sent to port 8090 on the local computer to the remote machine's localhost:8090. Therefore, you can access Zeppelin via localhost:8090 on your computer.
Here's an example of it in use: Big Data: Amazon EMR, Apache Spark and Apache Zeppelin – Part 2 of 2
PuTTy
If you are using PuTTY to connect to the host, there is a similar command in the Tunnels configuration screen. Redirect 8090 to localhost:8090 on the remote machine.
I can't access Cloud9 on port 8081 running on Google cloud platform.
I am sure the application is running on that port and applications on the same machine on other ports (e.g. http://xxx.xxx.xxx.xxx:3000) are accessible correctly, so this doesn't seem to be an issue with the iptables settings.
I receive no response from the server http://xxx.xxx.xxx.xxx:8081/.
Google Cloud Platform configuration:
Allowed protocols and ports include tcp:8000-8089
IP Address set up as static and external
Command used to run Cloud9:
node server.js -w /home/workspace -l 0.0.0.0 -p 8081 -a username:password
The problem was that I have accidentally added some Target tags in the Google cloud firewall settings (Networking > Firewall rules).
Removing those tags solved my problem, I just use the default Apply to all targets setting.
I'm a total newbie when it comes to servers and EC2 but I have an EC2 instance running which was stopped and restarted. Since that I haven't been able to connect via a web-browser to the site but can connect via SSH. Are there any ideas why this could be the case?
Here is the debug checklist:
Check your server, make sure it is up and running in AWS management console.
Check port security setting, from your local machine. If it is close open it in security group.
telnet your_ip 80 # Check http port
telnet your_ip 22 # Check SSH port
Check your web server, Apache, Nginx, etc. If you cannot find your process, start it.
ps -ef | grep -e nginx -e httpd -e apache
Check your virtual host config, make sure you web server is listening to port 80 (http) or 443 (https).
Run chkconfig and make sure your web server will start as a service.
Check the security group in your instance property, it is probably allowing only SSH (default), you may want to add a inbound rule allowing HTTP traffic.