I have a project with Cloud Composer and Cloud SQL.
I am able to connect to Cloud SQL because i edited the yaml of airflow-sqlproxy-service and added my Cloud SQL instance on cloud proxy used for the airflow-db, mapping to port 3307.
The workers can connect to airflow-sqlproxy-service on port 3307 but i think the webserver can't connect to this.
Do i need to add some firewall rule to map the 3307 port so the webserver or the UI can connect to airflow-sqlproxy-service?
https://i.stack.imgur.com/LwKQK.png
https://i.stack.imgur.com/CJf7Q.png
https://i.stack.imgur.com/oC2dJ.png
Best regards.
Composer does not currently support configuring additional sql proxies from the webserver. One workaround for cases like this is to have a separate DAG which loads Airflow Variables with the information needed from the other database (via the workers which do have access), then generate a DAG based on the Variable which the webserver can access.
https://github.com/apache/incubator-airflow/pull/4170 recently got merged (not currently available in Composer), which defines a CloudSQL connection type. This might work for these use cases in the future.
Related
I am trying to reach our Postgres SQL server running as a GCP Cloud SQL instance from the PgAdmin 4 tool on desktop. For that you have to whitelist your IP address. Until this point it was working fine, the whole team used IPV4 addresses, but I moved to a new country where my new ISP assigned an IPV6 address to me, and it seems like GCP doesn't allow IPV6 addesses to be whitelisted, thus you can't use them to connect.
Here's a picture of the Connections/Allowed Networks tab:
Is there any kind of solution to this?
Or do they expect me to only have ISPs who assign IPV4 addesses to me?
Thank you.
Alternatively, you could just use the Cloud SQL Auth Proxy directly. You'll have to run an instance of it on your local machine and then PgAdmin can connect to the proxy on localhost.
Yes, there is a workaround for this but not using the UI, instead you need to use the Cloud SDK.
This has recently been added to cloud SDK beta. So as you can see in the documentation for gcloud sql connect:
If you're connecting from an IPv6 address, or are constrained by certain organization policies (restrictPublicIP, restrictAuthorizedNetworks), consider running the beta version of this command to avoid error by connecting through the Cloud SQL proxy: gcloud beta sql connect.
I couldn't find any Feature Requests for this being implemented into the UI at the moment, so if you'd like to have this changed I would suggest you to open one in Google's Issue Tracker system.
Based on my current understanding, when I enable a service connection to my Cloud SQL instance in one of my revisions, the path /cloudsql/[instance name]/.s.PGSQL.5432 becomes populated. This is a UNIX socket connection.
Unfortunately, a 3rd party application I'm using doesn't support UNIX socket connections and as such I'm required to connect via TCP.
Does the Google Cloud SQL Proxy also configure any way I can connect to Cloud SQL via something like localhost:5432, or other equivalent? Some of the documentation I'm reading suggests that I have to do elaborate networking configuration with private IPs just to enable TCP based Cloud SQL for my Cloud Run revisions, but I feel like the Cloud Proxy is already capable of giving me a TCP connection instead of a UNIX socket.
What is the right and most minimal way forward here, obviously assuming I do not have the ability to modify the code I'm running.
I've also cross posted this question to the Google Cloud SQL Proxy repo.
The most secure and easiest way is to use the private IP. It's not so long and so hard, you have 3 steps
Create a serverless VPC connector. Create it in the same region as your Cloud Run service. Note the VPC Network that you use (by default it's "default")
Add the serverless VPC Connector to Cloud Run service. Route only the private IPs through this connector
Add a private connection to your Cloud SQL database. Attached it in the same VPC Network as your serverless VPC Connector.
The Cloud configuration is over. Now you have to get the Cloud SQL private IP of your instance and to add it in parameters of your Cloud Run service to open a connection to this IP.
I have a postgresql database on the google cloud platform (cloud SQL). I'm currently managing this database through pgadmin, installed on my laptop. I've added the IP address of my laptop to the whitelist on the cloud sql settings page. This all works.
The problem is: when I go somewhere else and I connect to a different network, the IP address changes and I cannot connect to the postgresql database (through pgadmin) from my laptop.
Is there someone who knows a (secure) solution, involving a proxy server (or something else), to connect from my laptop (and only my laptop) to my postgresql database, even if I'm not on a whitelisted network (IP address)? Maybe I can set up a VM instance and install a proxy server and use this? But I have no clue where to start (or search for).
You have many options for connecting to a Cloud SQL instance from an external applications such a Public IP address with SSL, Public IP address without SSL, Cloud SQL proxy, etc. You can see all of them here.
Between all connection options there exists Cloud SQL Proxy, it basically provides secure access to your instances without the need for Authorized networks or configuring SSL on your part.
You only need to follow the steps listed here and you will be able to connect your Cloud SQL instance using the proxy.
Enable Cloud SQL Admin API on your console.
Install the proxy client on your local machine (Linux):
wget https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64 -O cloud_sql_proxy
chmod +x cloud_sql_proxy
Determine how you will authenticate the proxy. You can use use a service account or let Cloud SDK take care of the authentication.
However, if required by your authentication method, create a service account.
Determine how you will specify your instances for the proxy. Your options for instance specification depend on your operating system and environment
Start the proxy using either TCP sockets or Unix sockets.
Take note that as of this writing, Cloud SQL Proxy does not support Unix sockets on Windows.
Update your application to connect to Cloud SQL using the proxy.
We have a GCE VM with MySQL server. Firewall rules deny incoming connections from external IP. Our cloud build process requires to perform DB migrations so it needs to connect to MySQL from Cloud Build. I want to add a Firewall rule to allow only cloud builder to connect through 3306 from the external IP address.
Cloud Build does not run on internal network so there is no way to connect from the internal IP.
I tried adding a rule for service account scope but I can't see cloud build service account in the list.
Currently, as mentioned by #guillaume blaquiere, there is a Feature Request. I recommend you to follow it (star) to receive all the updates there. Seems that the FR has been sent to the Cloud Build engineering team and they will evaluate it. Also note that there is not yet an ETA of the implementation.
Looking for a way to JDBC from managed Cloud Run to a Cloud SQL instance that doesn't require opening up a public IP on the database.
Per the documentation, managed Cloud Run only supports UNIX socket access, which JDBC doesn't really support. I tried junixsockets (https://kohlschutter.github.io/junixsocket/dependency.html) and couldn't get it to work.
I'd prefer to not be reduced to having to run a SOCKS proxy :).
There is now documentation on connecting Cloud SQL to Cloud Run via JDBC: https://cloud.google.com/sql/docs/mysql/connect-run#connecting_to (click on the Java tab)