I'm trying to load Tensorboard from within my google cloud VM terminal.
tensorboard --logdir logs --port 6006
Serving TensorBoard on localhost; to expose to the network, use a proxy or pass --bind_all
TensorBoard 2.2.1 at http://localhost:6006/ (Press CTRL+C to quit)
When I click on the link:
Chrome I get error 400
Firefox Error: Could not connect to Cloud Shell on port 6006. Ensure your server is listening on port 6006 and try again.
I've added a new firewall rule to allow port 6006 for ip 0.0.0.0/0 , but still can't get this to work. I've tried using --bind_all too but this doesn't work.
From Training a Keras Model on Google Cloud ML GPU:
... To train this model now on google cloud ML engine run the below command on cloud sdk terminal
gcloud ml-engine jobs submit training JOB1
--module-name=trainer.cnn_with_keras
--package-path=./trainer
--job-dir=gs://keras-on-cloud
--region=us-central1
--config=trainer/cloudml-gpu.yaml
Once you have started the training you can watch the logs from google console. Training would take around 5 minutes and the logs should look like below. Also you would be able to view the tensorboard logs in the bucket that we had created earlier named ‘keras-on-cloud’
To visualize the training and changes graphically open the cloud shell by clicking the icon on top right for the same. Once started type the below command to start Tensorboard on port 8080.
tensorboard --logdir=gs://keras-on-cloud --port=8080
For anyone else struggling with this, I decided to output my logs to an S3 bucket, and then rather than trying to run tensorboard from within the GCP instance, I just ran it locally, tested with the below script.
I needed to put this into a script rather than calling directly from the command line as I needed my AWS credentials to be loaded. I then use subprocess to run the command line function as normal.
Credentials contained within an env file, found using python-dotenv
from dotenv import find_dotenv, load_dotenv
import subprocess
load_dotenv(find_dotenv())
if __name__=='__main__':
cmd = 'tensorboard --logdir s3://path-to-s3-bucket/Logs/'
p = subprocess.Popen(cmd, shell=True)
p.wait()
Serving TensorBoard on localhost; to expose to the network, use a proxy or pass --bind_all
TensorBoard 2.1.1 at http://localhost:6006/ (Press CTRL+C to quit)
Related
This question is about cloud run job (not services).
InterfaceError - (pg8000.exceptions.InterfaceError) Can't create a connection to host 127.0.0.1 and port 5432 (timeout is None and source_address is None).
I have python code that connects to cloud sql and runs a simple select * on sql db.
My cloud sql instance is public, in same account & region as cloud run
I had added cloud sql connection to cloud RUN job through console:
Recreating this error on local machine using docker:
When I run the container locally along with cloud sql proxy as shown below it works successfully:
docker run --rm --network=host job1
If I remove --network=host then I can recreate the exact error (shown in cloud RUN) locally:
docker run --rm job1
Am I using wrong host?
On local machine I set host - 127.0.0.1 as shown in official example - gcp github
On cloud RUN I tried setting host to 127.0.0.1 and /cloudsql/project:region:instance . Both did not work
My python code that runs on cloud RUN:
import os
import pandas
import sqlalchemy
def execute_sql(query, engine):
with engine.connect() as connection:
df = pandas.read_sql(
con=connection,
sql=query
)
return df
def connect_tcp_socket() -> sqlalchemy.engine.base.Engine:
db_user = 'postgres' # e.g. 'my-database-user'
db_pass = 'abcxyz123' # e.g. 'my-database-password'
db_name = 'development' # e.g. 'my-database'
db_host = os.getenv('host', '127.0.0.1')
db_port = os.getenv('port', 5432)
connect_args = {}
pool = sqlalchemy.create_engine(
sqlalchemy.engine.url.URL.create(
drivername="postgresql+pg8000",
username=db_user,
password=db_pass,
host=db_host,
port=db_port,
database=db_name,
),
connect_args=connect_args,
pool_size=5,
max_overflow=2,
pool_timeout=30, # 30 seconds
pool_recycle=1800, # 30 minutes
)
return pool
def func1():
engine = connect_tcp_socket()
query = 'select * from public.administrator;'
df = execute_sql(query, engine)
print(f'df={df}')
if __name__ == '__main__':
func1()
How is your Cloud SQL instance configured? is it using private or public ip? Is the Cloud SQL instance in the same project, region and net? usually when you are connecting to 127.0.0.1 you are actually connecting to the Cloud SQL via Auth Proxy locally, however this doesn't apply for Cloud Run, depending on your cloud sql configuration you want to make sure that you configured the Cloud SQL connectivity at the deployement moment using the following flags if your Cloud SQL uses public ip
gcloud run deploy
--image=IMAGE
--add-cloudsql-instances=INSTANCE_CONNECTION_NAME
If your Cloud SQL is using private ip you want to use the instance private ip and not 127.0.0.1
I was unable to connect to proxy from container running on cloud run jobs. So instead I started proxy manually from inside the Dockerfile. This way I know exact port and host to map.
To run the python script job1.py shown in the question use following files
Dockerfile:
FROM python:buster
ADD . /code
RUN pip install --upgrade pip
RUN pip install pandas sqlalchemy
RUN pip install pg8000 cloud-sql-python-connector
# download the cloudsql proxy binary
RUN mkdir "/workspace"
RUN wget https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64 -O /workspace/cloud_sql_proxy
RUN chmod +x /workspace/cloud_sql_proxy
RUN chmod +x /code/start.sh
CMD ["/code/start.sh"]
start.sh
#!/bin/sh
/workspace/cloud_sql_proxy -instances=pangea-dev-314501:us-central1:pangea-dev=tcp:5432 -credential_file=/key/sv_account_key.json &
sleep 6
python /code/job1.py
I'd recommend using the Cloud SQL Python Connector package as it offers a consistent way of connecting across all environments (Local machine, Cloud Run, App Engine, Cloud Functions etc.) and provides the following benefits (one of which is not having to worry about IP addresses or needing the Cloud SQL proxy):
IAM Authorization: uses IAM permissions to control who/what can connect to your Cloud SQL instances
Improved Security: uses robust, updated TLS 1.3 encryption and identity verification between the client connector and the server-side proxy, independent of the database protocol.
Convenience: removes the requirement to use and distribute SSL certificates, as well as manage firewalls or source/destination IP addresses.
(optionally) IAM DB Authentication: provides support for Cloud SQL’s automatic IAM DB AuthN feature.
You can find a Flask App example using the Python Connector in the same Github repo you linked in your question.
Basic usage example:
from google.cloud.sql.connector import Connector, IPTypes
import sqlalchemy
# build connection (for creator argument of connection pool)
def getconn():
# Cloud SQL Python Connector object
with Connector() as connector:
conn = connector.connect(
"project:region:instance", # Cloud SQL instance connection name
"pg8000",
user="my-user",
password="my-password",
db="my-db-name",
ip_type=IPTypes.PUBLIC # IPTypes.PRIVATE for private IP
)
return conn
# create connection pool
pool = sqlalchemy.create_engine(
"postgresql+pg8000://",
creator=getconn,
)
So I uploaded my Django API with Rest framework on AWS EC2 instance. However, I have to manually go to Putty and connect to my EC2 instance and turn API on whenever I want to use it by inputting python manage.py runserver 0.0.0.0:8000.
When I turn off my PC, putty closes and API cannot be accessed anymore on the ip address.
How do I keep my API on forever? Does turning it into https help? Or what can be done?
You can make it live always by following means,
connect your ec2 instance using ssh.
Then deploy your backend (django) on that instance and run it at any port.
Once run on your desired port, you can close the terminal, please don't press ctrl+c so that your django server does not stop. you can just cross the terminal. it will be now running.
You can also run the django server on tmux (its terminal inside terminal). here is tutorial on tmux.
https://linuxize.com/post/getting-started-with-tmux/
One other approach is, you can deploy the django using docker container.
I hope you will come over your problem.
Thanks.
Ok I finally solved this. So when you close putty or a ssh client session, the session goes offline. However, if you run the session via daemon, the session continues in the background even when you close your clients. The code is
$ nohup python ./manage.py runserver 0.0.0.0:8000 &
Of course you can use tmux or docker, as suggested by madi, but I think running this one code is much simpler.
You can use pm2.
Please install pm2.
And make a server.json file in the root directory of your django app to run your app.
{
apps:
[{
name: "appname",
script: "manage.py",
args: ["runserver", "0.0.0.0:8888"],
exec_mode: "fork",
instances: "1",
wait_ready: true,
autorestart: false,
max_restarts: 5,
interpreter : "python3"
}]
}
Then you can run this app with pm2 start server.json.
Your app will run on port 8888 .
Google Cloud Run allows for using Cloud SQL. But what if you need Cloud SQL when building your container in Google Cloud Build? Is that possible?
Background
I have a Next.js project, that runs in a Container on Google Cloud Run. Pushing my code to Cloud Build (installing the stuff, generating static pages and putting everything in a Container) and deploying to Cloud Run works perfectly. 👌
Cloud SQL
But, I just added some functionality in which it also needs to some data from my PostgreSQL instance that runs on Google Cloud SQL. This data is used when building the project (generating the static pages).
Locally, on my machine, this works fine as the project can connect to my CloudSQL proxy. While running in CloudRun this should also work, as Cloud Run allows for connecting to my Postgres instance on Cloud SQL.
My problem
When building my project with Cloud Build, I need access to my database to be able to generate my static pages. I am looking for a way to connect my Docker cloud builder to Cloud SQL, perhaps just like Cloud Run (fully managed) provides a mechanism that connects using the Cloud SQL Proxy.
That way I could be connecting to /cloudsql/INSTANCE_CONNECTION_NAME while building my project!
Question
So my question is: How do I connect to my PostgreSQL instance on Google Cloud SQL via the Cloud SQL Proxy while building my project on Google Cloud Build?
Things like my database credentials, etc. already live in Secrets Manager, so I should be able to use those details I guess 🤔
You can use the container that you want (and you need) to generate your static pages, and download cloud sql proxy to open a tunnel with the database
- name: '<YOUR CONTAINER>'
entrypoint: 'sh'
args:
- -c
- |
wget https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64 -O cloud_sql_proxy
chmod +x cloud_sql_proxy
./cloud_sql_proxy -instances=<my-project-id:us-central1:myPostgresInstance>=tcp:5432 &
<YOUR SCRIPT>
App engine has an exec wrapper which has the benefit of proxying your Cloud SQL in for you, so I use that to connect to the DB in cloud build (so do some google tutorials).
However, be warned of trouble ahead: Cloud Build runs exclusively* in us-central1 which means it'll be pathologically slow to connect from anywhere else. For one or two operations, I don't care but if you're running a whole suite of integration tests that simply will not work.
Also, you'll need to grant permission for GCB to access GCSQL.
steps:
- id: 'Connect to DB using appengine wrapper to help'
name: gcr.io/google-appengine/exec-wrapper
args:
[
'-i', # The image you want to connect to the db from
'$_GCR_HOSTNAME/$PROJECT_ID/$REPO_NAME:$SHORT_SHA',
'-s', # The postgres instance
'${PROJECT_ID}:${_POSTGRES_REGION}:${_POSTGRES_INSTANCE_NAME}',
'-e', # Get your secrets here...
'GCLOUD_ENV_SECRET_NAME=${_GCLOUD_ENV_SECRET_NAME}',
'--', # And then the command you want to run, in my case a database migration
'python',
'manage.py',
'migrate',
]
substitutions:
_GCLOUD_ENV_SECRET_NAME: mysecret
_GCR_HOSTNAME: eu.gcr.io
_POSTGRES_INSTANCE_NAME: my-instance
_POSTGRES_REGION: europe-west1
* unless you're willing to pay more and get very stung by Beta software, in which case you can use cloud build workers (at the time of writing are in Beta, anyway... I'll come back and update if they make it into production and fix the issues)
The ENV VARS (including DB connections) are not available during build steps.
However, you can use ENTRYPOINT (of Docker) to run commands when the container runs (after completing the build steps).
I was having the need to run DB migrations when a new build was deployed (i.e. when the container starts running) and using ENTRYPOINT (to a file/command) was able to run migrations (which require DB connection details, not available during the build-process).
"How to" part is pretty brief and is located here : https://stackoverflow.com/a/69088911/867451
I see that all of the examples per the documentation use some form of a simple web application (For example, Flask in Python). Is it possible to use cloud run as a non web application? For example, deploy cloud run to use a python script and then use GCP Scheduler to invoke cloud run every hour to run that script? Basically my thinking for this is to avoid having to deploy and pay for Compute Engine, and only pay for when the cloud run container is invoked via the scheduler.
It's mandatory to answer to HTTP request. It's the contract of Cloud Run
Stateless (no volume attached to the container)
Answer to HTTP request
However, if you already have a python script, it's easy to wrap it in a flask webserver. Let's say, you have something like this (I assume that the file name is main.py -> important for the Dockerfile at the end)
import ....
var = todo(...)
connect = connect(...)
connect(var)
Firstly, wrap it in a function like this
import ....
def my_function(request):
var = todo(...)
connect = connect(...)
connect(var)
return 'ok',200
Secondly, add a flask server
from flask import Flask, request
import os
import ....
app = Flask(__name__)
#app.route('/')
def my_function(request):
var = todo(...)
connect = connect(...)
connect(var)
return 'ok',200
if __name__ == "__main__":
app.run(host='0.0.0.0',port=int(os.environ.get('PORT',8080)))
Add flask in your requirements.txt
Build a standard container, here an example of Dockerfile
FROM python:3-alpine
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
ENV PORT 8080
CMD [ "python", "main.py" ]
Build (with Cloud Build for example) and deploy the service on Cloud Run
Now you have an URL, that you can call with Cloud Scheduler.
Be careful, the max request duration is, for now, limited to 15 minutes (soon 4x more) and limited to 2vCPU and 2Gb of memory (again, soon more).
It depends what is being installed in the container image, as there is no requirement that one would have to install a web-server. For example, with such an image I can build Android applications, triggered whenever a repository changes (file excludes recommend) ...and likely could even run a head-less Android emulator for Gradle test tasks and publish test results to Pub/Sub (at least while the test-suite wouldn't run for too long). I mean, one has to understand the possibilities of Cloud Build to understand what Cloud Run can do.
I've struggled with deployment my function which no need to handle any request in Cloud Run by putting functions in Flask app and found out that Cloud Run provides us 2 kinds of jobs, services and jobs.
Illustration from codelabs
From cloud run jobs documentation,
This page describes how to create and update Cloud Run jobs from an existing container image. Unlike services, which listen for requests, a job does not serve requests but only runs its tasks and exits when finished.
After you create or update a job, you can execute the job as a one-off, on a schedule or as part of a workflow. You can manage individual job executions and view the execution logs.
You may see that there are two tabs in Cloud Run console. I am not sure when Cloud Run jobs started.
See cloud run console
I have set a crontab file to run a Python script that creates an JSON file and writes it to an S3 bucket. It runs as expected when executed from the command line, but when I run it as a cron job, I get the following error:
botocore.exceptions.ConnectTimeoutError: Connect timeout on endpoint URL
This results from the following lines of code in the script:
import boto3
def main():
# Create EC2 client and get EC2 response
ec2 = boto3.client('ec2')
response = ec2.describe_instances()
My guess is that some permission is not set in the cron job, denying me access to the URL.
It turns out that I had to set the proxy settings so I access AWS as myself rather than root. I ran the cron job as a Linux shell script rather then a Python script, and exported my http_proxy, https_proxy, and no_proxy settings found in ~/.bash_profile in the first lines of the shell script
`export http_proxy=<http_proxy from ~/.bash_profile>
export https_proxy=<https_proxy from ~/.bash_profile>
export no_proxy=<no_proxy from ~./bash_profile>
python <python script>`