Google Cloud Run allows for using Cloud SQL. But what if you need Cloud SQL when building your container in Google Cloud Build? Is that possible?
Background
I have a Next.js project, that runs in a Container on Google Cloud Run. Pushing my code to Cloud Build (installing the stuff, generating static pages and putting everything in a Container) and deploying to Cloud Run works perfectly. 👌
Cloud SQL
But, I just added some functionality in which it also needs to some data from my PostgreSQL instance that runs on Google Cloud SQL. This data is used when building the project (generating the static pages).
Locally, on my machine, this works fine as the project can connect to my CloudSQL proxy. While running in CloudRun this should also work, as Cloud Run allows for connecting to my Postgres instance on Cloud SQL.
My problem
When building my project with Cloud Build, I need access to my database to be able to generate my static pages. I am looking for a way to connect my Docker cloud builder to Cloud SQL, perhaps just like Cloud Run (fully managed) provides a mechanism that connects using the Cloud SQL Proxy.
That way I could be connecting to /cloudsql/INSTANCE_CONNECTION_NAME while building my project!
Question
So my question is: How do I connect to my PostgreSQL instance on Google Cloud SQL via the Cloud SQL Proxy while building my project on Google Cloud Build?
Things like my database credentials, etc. already live in Secrets Manager, so I should be able to use those details I guess 🤔
You can use the container that you want (and you need) to generate your static pages, and download cloud sql proxy to open a tunnel with the database
- name: '<YOUR CONTAINER>'
entrypoint: 'sh'
args:
- -c
- |
wget https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64 -O cloud_sql_proxy
chmod +x cloud_sql_proxy
./cloud_sql_proxy -instances=<my-project-id:us-central1:myPostgresInstance>=tcp:5432 &
<YOUR SCRIPT>
App engine has an exec wrapper which has the benefit of proxying your Cloud SQL in for you, so I use that to connect to the DB in cloud build (so do some google tutorials).
However, be warned of trouble ahead: Cloud Build runs exclusively* in us-central1 which means it'll be pathologically slow to connect from anywhere else. For one or two operations, I don't care but if you're running a whole suite of integration tests that simply will not work.
Also, you'll need to grant permission for GCB to access GCSQL.
steps:
- id: 'Connect to DB using appengine wrapper to help'
name: gcr.io/google-appengine/exec-wrapper
args:
[
'-i', # The image you want to connect to the db from
'$_GCR_HOSTNAME/$PROJECT_ID/$REPO_NAME:$SHORT_SHA',
'-s', # The postgres instance
'${PROJECT_ID}:${_POSTGRES_REGION}:${_POSTGRES_INSTANCE_NAME}',
'-e', # Get your secrets here...
'GCLOUD_ENV_SECRET_NAME=${_GCLOUD_ENV_SECRET_NAME}',
'--', # And then the command you want to run, in my case a database migration
'python',
'manage.py',
'migrate',
]
substitutions:
_GCLOUD_ENV_SECRET_NAME: mysecret
_GCR_HOSTNAME: eu.gcr.io
_POSTGRES_INSTANCE_NAME: my-instance
_POSTGRES_REGION: europe-west1
* unless you're willing to pay more and get very stung by Beta software, in which case you can use cloud build workers (at the time of writing are in Beta, anyway... I'll come back and update if they make it into production and fix the issues)
The ENV VARS (including DB connections) are not available during build steps.
However, you can use ENTRYPOINT (of Docker) to run commands when the container runs (after completing the build steps).
I was having the need to run DB migrations when a new build was deployed (i.e. when the container starts running) and using ENTRYPOINT (to a file/command) was able to run migrations (which require DB connection details, not available during the build-process).
"How to" part is pretty brief and is located here : https://stackoverflow.com/a/69088911/867451
Related
This question is about cloud run job (not services).
InterfaceError - (pg8000.exceptions.InterfaceError) Can't create a connection to host 127.0.0.1 and port 5432 (timeout is None and source_address is None).
I have python code that connects to cloud sql and runs a simple select * on sql db.
My cloud sql instance is public, in same account & region as cloud run
I had added cloud sql connection to cloud RUN job through console:
Recreating this error on local machine using docker:
When I run the container locally along with cloud sql proxy as shown below it works successfully:
docker run --rm --network=host job1
If I remove --network=host then I can recreate the exact error (shown in cloud RUN) locally:
docker run --rm job1
Am I using wrong host?
On local machine I set host - 127.0.0.1 as shown in official example - gcp github
On cloud RUN I tried setting host to 127.0.0.1 and /cloudsql/project:region:instance . Both did not work
My python code that runs on cloud RUN:
import os
import pandas
import sqlalchemy
def execute_sql(query, engine):
with engine.connect() as connection:
df = pandas.read_sql(
con=connection,
sql=query
)
return df
def connect_tcp_socket() -> sqlalchemy.engine.base.Engine:
db_user = 'postgres' # e.g. 'my-database-user'
db_pass = 'abcxyz123' # e.g. 'my-database-password'
db_name = 'development' # e.g. 'my-database'
db_host = os.getenv('host', '127.0.0.1')
db_port = os.getenv('port', 5432)
connect_args = {}
pool = sqlalchemy.create_engine(
sqlalchemy.engine.url.URL.create(
drivername="postgresql+pg8000",
username=db_user,
password=db_pass,
host=db_host,
port=db_port,
database=db_name,
),
connect_args=connect_args,
pool_size=5,
max_overflow=2,
pool_timeout=30, # 30 seconds
pool_recycle=1800, # 30 minutes
)
return pool
def func1():
engine = connect_tcp_socket()
query = 'select * from public.administrator;'
df = execute_sql(query, engine)
print(f'df={df}')
if __name__ == '__main__':
func1()
How is your Cloud SQL instance configured? is it using private or public ip? Is the Cloud SQL instance in the same project, region and net? usually when you are connecting to 127.0.0.1 you are actually connecting to the Cloud SQL via Auth Proxy locally, however this doesn't apply for Cloud Run, depending on your cloud sql configuration you want to make sure that you configured the Cloud SQL connectivity at the deployement moment using the following flags if your Cloud SQL uses public ip
gcloud run deploy
--image=IMAGE
--add-cloudsql-instances=INSTANCE_CONNECTION_NAME
If your Cloud SQL is using private ip you want to use the instance private ip and not 127.0.0.1
I was unable to connect to proxy from container running on cloud run jobs. So instead I started proxy manually from inside the Dockerfile. This way I know exact port and host to map.
To run the python script job1.py shown in the question use following files
Dockerfile:
FROM python:buster
ADD . /code
RUN pip install --upgrade pip
RUN pip install pandas sqlalchemy
RUN pip install pg8000 cloud-sql-python-connector
# download the cloudsql proxy binary
RUN mkdir "/workspace"
RUN wget https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64 -O /workspace/cloud_sql_proxy
RUN chmod +x /workspace/cloud_sql_proxy
RUN chmod +x /code/start.sh
CMD ["/code/start.sh"]
start.sh
#!/bin/sh
/workspace/cloud_sql_proxy -instances=pangea-dev-314501:us-central1:pangea-dev=tcp:5432 -credential_file=/key/sv_account_key.json &
sleep 6
python /code/job1.py
I'd recommend using the Cloud SQL Python Connector package as it offers a consistent way of connecting across all environments (Local machine, Cloud Run, App Engine, Cloud Functions etc.) and provides the following benefits (one of which is not having to worry about IP addresses or needing the Cloud SQL proxy):
IAM Authorization: uses IAM permissions to control who/what can connect to your Cloud SQL instances
Improved Security: uses robust, updated TLS 1.3 encryption and identity verification between the client connector and the server-side proxy, independent of the database protocol.
Convenience: removes the requirement to use and distribute SSL certificates, as well as manage firewalls or source/destination IP addresses.
(optionally) IAM DB Authentication: provides support for Cloud SQL’s automatic IAM DB AuthN feature.
You can find a Flask App example using the Python Connector in the same Github repo you linked in your question.
Basic usage example:
from google.cloud.sql.connector import Connector, IPTypes
import sqlalchemy
# build connection (for creator argument of connection pool)
def getconn():
# Cloud SQL Python Connector object
with Connector() as connector:
conn = connector.connect(
"project:region:instance", # Cloud SQL instance connection name
"pg8000",
user="my-user",
password="my-password",
db="my-db-name",
ip_type=IPTypes.PUBLIC # IPTypes.PRIVATE for private IP
)
return conn
# create connection pool
pool = sqlalchemy.create_engine(
"postgresql+pg8000://",
creator=getconn,
)
I've set up a Google Cloud Run with continuous deployment to a github, and it redeploys every time there's a push to the main (what I what), but when I go to check the site, it hasn't updated the HTML I've been testing with. I've tested it on my local machine, and it's updating the code when I run the Django server, so I'm guessing it's something with my cloudbuild.yml? There was another post I tried to mimic, but it didn't take.
Any advice would be very helpful! Thank you!
cloudbuild.yml:
steps:
# Build the container image
- name: 'gcr.io/cloud-builders/docker'
args: ['build', '-t', 'gcr.io/${PROJECT_ID}/exeplore', './ExePlore']
# Push the image to Container Registry
- name: 'gcr.io/cloud-builders/docker'
args: ['push', 'gcr.io/${PROJECT_ID}/exeplore']
# Deploy image to Cloud Run
- name: 'gcr.io/cloud-builders/gcloud'
args:
- 'run'
- 'deploy'
- 'exeplore'
- '--image'
- 'gcr.io/${PROJECT_ID}/exeplore'
- '--region'
- 'europe-west2'
- '--platform'
- 'managed'
images:
- gcr.io/${PROJECT_ID}/exeplore
Here are the variables for GCR
Edit 1: I've now updated my cloudbuild, so the SHORT_SHA is all gone, but now google cloud run is saying it can't find my manage.py at /Exeplore/manage.py. I might have to trial and error it, as running the container locally is fine, and same with running the server locally. I have yet to try what Ezekias suggested, as I've tried rolled back to when it was correctly running the server and it doesn't like that.
Edit 2: I've checked the services, it is at 100% Latest
Check your Cloud Run service, either on the Cloud Console or by running gcloud run services describe. It may be set to serve traffic to a specific revision instead of having 100% of traffic serving LATEST.
If that's the case, it won't automatically move traffic to the new revision when you deploy. If you want it to automatically switch to the new update, you can run gcloud run services update-traffic --to-latest or use the "Manage Traffic" button on the revisions tab of the Cloud Console to set 100% of traffic to the latest healthy revision.
It looks like you're building gcr.io/${PROJECT_ID}/exeplore:$SHORT_SHA, but pushing and deploying gcr.io/${PROJECT_ID}/exeplore. These are essentially different images.
Update any image variables to include the SHORT_SHA to ensure all references are the same.
To avoid duplication you may also want to use dynamic substitution variables
I've deployed my web app on google cloud today but when I open the link to my website only the HTML/almost no JS/flask functions, only some elements of bootstrap work (mostly the responsiveness that cols/rows provide, not e.g. the navbar collapse etc.) and most of the pictures I've stored in my static files aren't getting displayed as well. I suppose my Javascript doesn't work because most HTML/CSS properties provided by me / bootstrap are still working. When I run the container without cloud run everything works as it should. I run the container with gunicorn as my WSGI Server.
This is my file tree of the webapp
I created the image with docker:
-docker build -t styleit .
Then I used this https://cloud.google.com/container-registry/docs/pushing-and-pulling to push my container to container registry:
-docker tag styleit eu.gcr.io/styleit/styleit (my project-id is styleit as well)
-docker push eu.gcr.io/styleit/styleit
Then I used these (1) (2) configurations for the cloud run setup.
Please don't be easy on me if I did something wrong, I want to learn how the cloud infrastructure works ^^
I see that all of the examples per the documentation use some form of a simple web application (For example, Flask in Python). Is it possible to use cloud run as a non web application? For example, deploy cloud run to use a python script and then use GCP Scheduler to invoke cloud run every hour to run that script? Basically my thinking for this is to avoid having to deploy and pay for Compute Engine, and only pay for when the cloud run container is invoked via the scheduler.
It's mandatory to answer to HTTP request. It's the contract of Cloud Run
Stateless (no volume attached to the container)
Answer to HTTP request
However, if you already have a python script, it's easy to wrap it in a flask webserver. Let's say, you have something like this (I assume that the file name is main.py -> important for the Dockerfile at the end)
import ....
var = todo(...)
connect = connect(...)
connect(var)
Firstly, wrap it in a function like this
import ....
def my_function(request):
var = todo(...)
connect = connect(...)
connect(var)
return 'ok',200
Secondly, add a flask server
from flask import Flask, request
import os
import ....
app = Flask(__name__)
#app.route('/')
def my_function(request):
var = todo(...)
connect = connect(...)
connect(var)
return 'ok',200
if __name__ == "__main__":
app.run(host='0.0.0.0',port=int(os.environ.get('PORT',8080)))
Add flask in your requirements.txt
Build a standard container, here an example of Dockerfile
FROM python:3-alpine
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
ENV PORT 8080
CMD [ "python", "main.py" ]
Build (with Cloud Build for example) and deploy the service on Cloud Run
Now you have an URL, that you can call with Cloud Scheduler.
Be careful, the max request duration is, for now, limited to 15 minutes (soon 4x more) and limited to 2vCPU and 2Gb of memory (again, soon more).
It depends what is being installed in the container image, as there is no requirement that one would have to install a web-server. For example, with such an image I can build Android applications, triggered whenever a repository changes (file excludes recommend) ...and likely could even run a head-less Android emulator for Gradle test tasks and publish test results to Pub/Sub (at least while the test-suite wouldn't run for too long). I mean, one has to understand the possibilities of Cloud Build to understand what Cloud Run can do.
I've struggled with deployment my function which no need to handle any request in Cloud Run by putting functions in Flask app and found out that Cloud Run provides us 2 kinds of jobs, services and jobs.
Illustration from codelabs
From cloud run jobs documentation,
This page describes how to create and update Cloud Run jobs from an existing container image. Unlike services, which listen for requests, a job does not serve requests but only runs its tasks and exits when finished.
After you create or update a job, you can execute the job as a one-off, on a schedule or as part of a workflow. You can manage individual job executions and view the execution logs.
You may see that there are two tabs in Cloud Run console. I am not sure when Cloud Run jobs started.
See cloud run console
After my app is successfully pushed via cf I usually need do manually ssh-log into the container and execute a couple of PHP scripts to clear and warmup my cache, potentially execute some DB schema updates etc.
Today I found out about Cloudfoundry Tasks which seems to offer a pretty way to do exactly this kind of things and I wanted to test it whether I can integrate it into my build&deploy script.
So used cf login, got successfully connected to the right org and space, app has been pushed and is running and I tried this command:
cf run-task MYAPP "bin/console doctrine:schema:update --dump-sql --env=prod" --name dumpsql
(tried it with a couple of folder changes like app/bin/console etc.)
and this was the output:
Creating task for app MYAPP in org MYORG / space MYSPACE as me#myemail...
Unexpected Response
Response Code: 404
FAILED
Uses CF CLI: 6.32.0
cf logs ArcticTenTestBackend --recent does not output anything (this might be the case because I have enabled an ELK instance for logging - as I wanted to service-connect to ELK to look up the logs I found out that the service-connector cf plugin is gone for which I will open a new ticket).
Created new Issue for that: https://github.com/cloudfoundry/cli/issues/1242
This is not a CF CLI issue. Swisscom Application Cloud does not yet support the Cloud Foundry tasks. This explains the 404 you are currently receiving. We will expose this feature of Cloud Foundry in an upcoming release of Swisscom Application Cloud.
In the meantime, maybe you can find a way to execute your one-off tasks (cache warming, DB migrations) at application startup.
As mentioned by #Mathis Kretz Swisscom has gotten around to enable cf run-task since this question was posted. They send out e-mails on 22. November 2018 to announce the feature.
As discussed on your linked documentation you use the following commands to manage tasks:
cf tasks [APP_NAME]
cf run-task [APP_NAME] [COMMAND]
cf terminate-task [APP_NAME] [TASK_ID]