I have setup a standard Debian Linux VM via Compute Engine on GCP. The VM does not have an external IP address. I can connect to it via ssh by using the browser. I allowed incomming ssh (port 22) traffic and all outgoing traffic. I have tested BigQuery by executing queries via the browser interface and it works. I have configured BigQuery to be enabled for the VM via settings -> Cloud API access scopes. Now I would like to do a simple thing as the following:
bq show bigquery-public-data:samples.shakespeare
But nothing happens. I tried to do the following to get more info:
bq --apilog=stdout show bigquery-public-data:samples.shakespeare
Output is the following:
I0106 15:29:47.271125 140258687915840 bigquery_client.py:1205] Requesting discovery document from https://www.googleapis.com/discovery/v1/apis/bigquery/v2/rest
I0106 15:29:47.271456 140258687915840 transport.py:158] Attempting refresh to obtain initial access_token
Nothing more happens. Any ideas what the issue could be?
After reading the documentation it seems to me that the connection via the BigQuery command line tool should work by itself.
Firstly, why it doesn't work. In fact, when you use bq CLI, it's only a wrapper that call the BigQuery apis: https://bigquery.googleapis.com. The domain name is public. The Compute Engine try to resolve it on the public internet. But the compute engine doesn't have public IP and can't go on internet (internet server doesn't know how to route back the answer, because the VM is not reachable!)
Then, how to solve. 2 solutions:
You can set a Cloud NAT on your Compute Engine and thus, grant it a shareable public IP, only used to initiate outgoing traffic
You can use a not well known trick: activate the Google private API access in your subnet. For this, note the subnet of your Compute engine. Then go to VPC and select this subnet. Edit it and set to ON the private Google access.
Related
I have a server running at home but I haven't a fixed ip address. So, I use DDNS to update my domains DNS when IP changed, and it is working fine. My problem comes trying to access a MySQL instance, because currently it is using a VPC, so I need to update manually adding new IP as Authorized Network. I wonder if it is possible to do that with a API REST call, in that way I can add a crontab in my server to check changes each n minutes and update Authorized Networks.
I read Google documentation, but in my understanding (I am not an english speaker) it is possible just from an authorized network. Somebody can give me a clue?.
Thanks in advance.
Take a look at installing and using Cloud SQL Auth Proxy on your local server. This will remove the need to keep updating Authorized Networks when your IP changes.
I wonder if it is possible to do that with a API REST call, in that
way I can add a crontab in my server to check changes each n minutes
and update Authorized Networks.
Google Cloud provides the Cloud SQL Admin API. To modify the authorized networks, use the instances.patch API.
Google Cloud SQL Method: instances.patch
Modify this data structure to change the authorized networks:
Google Cloud SQL IP Configuration
You might find it easier to use the CLI to modify the authorized networks:
gcloud sql instances patch <INSTANCENAME> --authorized-networks=x.x.x.x/32
gcloud sql instances patch
I do not recommend constantly updating the authorized networks when not required. Use an external service to fetch your public IP and compare with the last saved value. Only update Cloud SQL if your public IP address changed.
Common public services to determine your public IP address. Note you should randomly select one as these services can rate limit you. Some of the endpoints require query parameters to only return your IP address and not a web page. Consult their documentation.
https://checkip.amazonaws.com/
https://ifconfig.me/
https://icanhazip.com/
https://ipecho.net/plain
https://api.ipify.org
https://ipinfo.io/ip
Note: I recommend that you use the Google Cloud SQL Auth Proxy. This provides several benefits including network traffic encryption. The auth proxy does not require that you whitelist your network.
Refer to my other answer for more details
I'm running a cloud run service with a working Cloud-SQL connection using the proxy to connect to the Cloud-SQL instance. The Cloud-SQL instance does not have a private IP configured.
Now there is a new requirement that this service needs to connect to a DB outside of GCP, for which it needs a static egress-IP that can be whitelisted. I attempted to achieve this via a serverless-VPC-connector (https://cloud.google.com/run/docs/configuring/static-outbound-ip).
Problem: When I add the VPC-connector to the service, and configure it to route all traffic through the vpc-connector, the service fails to deploy because it cannot connect to Cloud-SQL via the proxy anymore:
CloudSQL connection failed. Please see https://cloud.google.com/sql/docs/mysql/connect-run for additional details: Post "https://sqladmin.googleapis.com/sql/v1beta4/projects/<>/instances/<>/createEphemeral?alt=json&prettyPrint=false": context deadline exceeded
I was able to get this exact setup to work for a cloud function (identical external DB, CloudSQL, and vpc connector), and I'm at a loss as to why this wouldn't work for Cloud Run, and I'm wondering if there is additional configuration required which I'm missing?
Is it possible to connect to Cloud-SQL with the proxy, while at the same time using a VPC-connector to achieve a static egress IP?
If you need a static egress public IP, you only did the half of the path. Now, you need to add a Cloud NAT, that gets the VPC connector range and NAT the traffic to the internet. In this NAT, you can define a public static IP which will be used for all outgoing communications.
In any cases, if you want to reach Cloud SQL, it's better to use Cloud SQL proxy because the connection is secured (encrypted channel). If you use directly the public IP, add a SSL certificat to encrypt the communication over the public internet. (same thing if your instance is located outside GCP)
Based on my current understanding, when I enable a service connection to my Cloud SQL instance in one of my revisions, the path /cloudsql/[instance name]/.s.PGSQL.5432 becomes populated. This is a UNIX socket connection.
Unfortunately, a 3rd party application I'm using doesn't support UNIX socket connections and as such I'm required to connect via TCP.
Does the Google Cloud SQL Proxy also configure any way I can connect to Cloud SQL via something like localhost:5432, or other equivalent? Some of the documentation I'm reading suggests that I have to do elaborate networking configuration with private IPs just to enable TCP based Cloud SQL for my Cloud Run revisions, but I feel like the Cloud Proxy is already capable of giving me a TCP connection instead of a UNIX socket.
What is the right and most minimal way forward here, obviously assuming I do not have the ability to modify the code I'm running.
I've also cross posted this question to the Google Cloud SQL Proxy repo.
The most secure and easiest way is to use the private IP. It's not so long and so hard, you have 3 steps
Create a serverless VPC connector. Create it in the same region as your Cloud Run service. Note the VPC Network that you use (by default it's "default")
Add the serverless VPC Connector to Cloud Run service. Route only the private IPs through this connector
Add a private connection to your Cloud SQL database. Attached it in the same VPC Network as your serverless VPC Connector.
The Cloud configuration is over. Now you have to get the Cloud SQL private IP of your instance and to add it in parameters of your Cloud Run service to open a connection to this IP.
Google Cloud Platform Compute Engine
When I created the VM within the VPC network, I set up a static internal IP.
The firewall allowed the personal computer IP.
In this situation, the VM's SSH could not be reached.
Once logged in to the Google Cloud platform, metadata I generated using the keigen command to access the VM in a terminal in local environment.
e.g) ssh email ID#IP
Why can't I access that VM's SSH from the console on the Google Cloud platform?
Thank you for reading it. :)
If we read the following GCP documentation ... https://cloud.google.com/solutions/connecting-securely#external
we find the following statement:
When instances do not have external IP addresses (including VMs that are backends for HTTPS and SSL proxy load balancers) they can only be reached by other instances on the network, Identity-Aware Proxy's TCP forwarding feature, or by using managed VPN gateway.
The key phrase to me is that they can only be reached .... It would thus appear that you can only reach a VM without a public IP using the recipes described in the article. If it were me, I'd consider actually giving the machine a public IP address and then following the recipes described in the article to only allow access to the machine through secured means. Google is promoting a story it called Beyond Corp which basically summarizes down to trust nothing. Every attempt to access any resource will be validated on a request by request basis and don't assume that lack of network access is enough to secure a resource. Instead, consider the notion that the VM can be publicly reachable but only authorized requests should be satisfied.
I have some Cloud SQL instances that currently have public IP's. It would make certain security-minded people happy if I changed them to have private IP's.
I am following the instructions documented here: https://cloud.google.com/sql/docs/mysql/private-ip
A summary of those instructions:
Ensure your shared VPC host has servicenetworking.googleapis.com enabled
Ensure your project has servicenetworking.googleapis.com enabled
Allocate an IP address range for your new private IP's
Configure VPC network peering (https://cloud.google.com/sql/docs/mysql/configure-private-services-access)
Create cloud sql instance without public IP
Expect new instance's private IP to be in allocated range
I've completed these through step 4, and I'm seeing this:
My interpretation of that page is that I've done my part and now it's google's turn--but that was several days ago. Do I have do do something to prompt google to create the connection?
I think I'm focusing in the right place because if I try to use I private IP, gcloud tells me to go create the network that I'm waiting on:
❯ gcloud --project=my-project-name beta \
sql instances patch foo \
--network=my-network-name --no-assign-ip
The following message will be used for the patch API method.
{"name": "foo", "project": "my-project-name", "settings": {"ipConfiguration": {"ipv4Enabled": false, "privateNetwork": "https://compute.googleapis.com/compute/v1/projects/my-project-name/global/networks/my-network-name"}}}
Patching Cloud SQL instance...failed.
ERROR: (gcloud.beta.sql.instances.patch) [INTERNAL_ERROR] Failed to create subnetwork. Please create Service Networking connection with service 'servicenetworking.googleapis.com' from consumer project '11111111111' network 'my-network-name' again.
In general private services access is implemented as a VPC peering connection between your VPC network and the Google services VPC network where your Cloud SQL instance resides. As #JohnHanley pointed out, the VPC peering should be created within minutes so it’s not expected you have to wait more than that.
To check the peering creation on Stackdriver you can use the following Advanced Filter:
jsonPayload.event_subtype="compute.networks.addPeering"
That said, it makes sense the error you are observing when trying to patch your SQL Instance as the Peering hasn’t been created. Instead of ‘Inactive’ it should be ‘Peer VPC network is connected’
To sum up, in your scenario the Cloud SQL instance cannot get an IP on the aforementioned network as it cannot reach it.
At this specific point I would suggest you focus on the Peering creation. As you mentioned you tried recreating it and the status remains the same, it’s possible that there’s something in your project preventing the peering to be established.
I would also suggest you check the peering limits quota in case it has been reached:
gcloud compute networks peerings list --network='your network'
Also it would be good to review the VPC Peering Restrictions.
All that being said, if you still experience the same issue when creating the VPC Peering, an internal investigation may be required and I would suggest you to you to report this using this link
I hope this helps.