How to alert if cloud composer airflow UI is throwing server error?
Note: Airflow UI is hosted in Google tenant. I tried with uptime check but how to configure authentication for it. We are having multi factor authentication enabled in our accounts.
Can we use log based metrics? Please share steps for both ways.
Yes, log based metric can be used to track the health of the web server.
Actually, Cloud Composer creates one automatically for each Composer environment. Its name is logging.googleapis.com/user/$ENVIRONMENT_NAME-$ENVIRONMENT_LOCATION-web-health. Naturally, you can setup an alert for it. Its expected value is >= 0.1 entries/s.
Related
I would like to connect to a Cloud SQL instance from Cloud Run, using a service account. The connection used to be created within the VPC and we would just provide a connection string with a user and a password to our PostgreSQL client. But now we want the authentication to be managed by Google Cloud IAM, with the service account associated with the Cloud Run service.
On my machine, I can use the enable_iam_login argument to use my own service account. The command to run the Cloud SQL proxy would look like this:
./cloud_sql_proxy -dir=/cloudsql -instances=[PROJECT-ID]:[REGION]:[INSTANCE] \
-enable_iam_login -credential_file=${HOME}/.config/gcloud/application_default_credentials.json
The problem is that I can't seem to find a way to use the IAM authentication method to run the Cloud SQL Proxy from Cloud Run, I can just provide an instance name. Has anyone face this problem before?
Unfortunately, there isn't a way to configure Cloud Run's use of the Cloud SQL proxy to do this for you.
If you are using Java, Python, or Go, there are language specific connectors you can use from Cloud Run. These all have the option to use IAM DB AuthN as part of them.
For anyone using NodeJS on Cloud Run :
you can't enable the -enable_iam_login option in Cloud Run (cf accepted answer), which means you have to use a connector.
AFAIK, there is no NodeJS connector right now.
Which means you have 2 options :
write a connector yourself (good luck) or wait for Google to do it
use built-in database authentication for now.
We are getting this error on-and-off since 01/22 on Composer UI, the airflow dags seems to be running fine, the web UI however having the problem. Google cloud status page indicates there is no issue, changing browser or connection through other wifi / mobile hotspot doesn't help either. Not sure what's going on.
An internal server error occurred while authorizing your request. Error code 1
location: us-central1-c
version: composer-1.7.3-airflow-1.10.2
Cloud Composer is a managed workflow automation tool, so I don't think that you can address this issue directly.
For more context on what is happening, Cloud Composer architecture states that the Airflow Web Server is inside of a Tenant project, which is a isolated environment used for deploying managed services, this is for unified Cloud Identity and Access Management (IAM) access control and an additional layer of data security. Given that the error states that something is failing when authorizing, this issue may be related to the Cloud IAM service.
I recommend you to open a new case from the GCP console in the Support section, and ask for further assistance in this issue.
I receive an error message while attempting to deploy anything from the marketplace into a specific GCP project.
You must have a valid default service account in order to create a
deployment, but this account could not be detected. Contact support
for help restoring the account.
Things I've Tried:
Every VM from the marketplace shows the same error message
I can deploy regular VM instance
I can see there is an enabled service account for the project with the name "Compute Engine default service account".
I am able to deploy VM's from the marketplace into other projects under the same organization
I've contacted GCP Billing support and they cannot find anything wrong from a billing perspective
Researching online shows that others that have had this issue have just rebuilt the project. It appears that service account is created by default when the project is spun up.
I'm hoping there is another way around it as this project is a host for a shared VPC deployment. There are already other projects with deployed VM's that are utilizing the host projects networks.
Thank you!
Looks like you deleted a default service account.
As mentioned in one comment some can be recreated by disable/enable the corresponding API
Below are the default service accounts I have in my project, hope it helps you to find the root cause. (these service accounts let me deploy a wordpress solution depending on what you are trying to deploy you might need more service accounts)
PROJECT-NUMBER-compute#developer.gserviceaccount.com Compute Engine
default service account
PROJECT-NUMBER#cloudservices.gserviceaccount.com Google APIs Service
Agent
PROJECT-ID#appspot.gserviceaccount.com App Engine default service
account
service-ORG-ID3#gcp-sa-cloudasset.iam.gserviceaccount.com Cloud Asset
Service Agent
service-PROJECT-NUMBER#cloud-ml.google.com.iam.gserviceaccount.com Google
Cloud ML Engine Service Agent
service-PROJECT-NUMBER#compute-system.iam.gserviceaccount.com Compute
Engine Service Agent
service-PROJECT-NUMBER#container-engine-robot.iam.gserviceaccount.com Kubernetes
Engine Service Agent
service-PROJECT-NUMBER#containerregistry.iam.gserviceaccount.com Google
Container Registry Service Agent
service-PROJECT-NUMBER#dataflow-service-producer-prod.iam.gserviceaccount.com Cloud
Dataflow Service Account
service-PROJECT-NUMBER#service-networking.iam.gserviceaccount.com Service
Networking Service Agent
The service account was intact and had the same permissions as other service accounts for working projects.
We purchased and opened a case with GCP technical support. After a little more than a week of them troubleshooting the issues, they determined there was no way to correct the problem. Their root cause was that something happened during the initial project deployment that caused some backend configuration issues. For what its worth, the project was deployed using Terraform, but its uncertain if that was a factor.
After recreating the host project, we were able to deploy from the marketplace again successfully.
If you run into this problem, save yourself the hassle and time and just recreate the project.
Istio has been recently activated on the Google Kubernetes Engine cluster we are using. However we don't know who of our team did it.
Is there a way to find out who activated which features on our Google Kuberentes Engine respectively on the Google Cloud Platform in general?
You might want to check the "Activity" tab within GCP Console. It register all the actions that have been made in the project. Also, you can take a look at Stackdriver Logging and check the Cluster activity logs, it will show you the user that executed the Istio activation.
I'm trying to connect a GCE instance to a Cloud SQL instance (2nd gen) with Cloud SQL Proxy. When i ssh to the GCE instance and
sudo ./cloud_sql_proxy -dir=/cloudsql
I get
2016/07/13 18:59:28 Using gcloud's active project: xxxx-0000
2016/07/13 18:59:29 Error listing instances in xxxx-0000:
googleapi: Error 403: Access Not Configured.
Cloud SQL Administration API has not been used in project 000000000000
before or it is disabled. Enable it by visiting
https://console.developers.google.com/apis/api/sqladmin/overview?project=000000000000
then retry.
If you enabled this API recently, wait a few minutes for the action to
propagate to our systems and retry., accessNotConfigured
2016/07/13 18:59:29 no Cloud SQL Instances found in these projects: [xxxx-0000]
I have followed the documentations at
https://cloud.google.com/sql/docs/compute-engine-access#gce-connect-proxy and
https://cloud.google.com/sql/docs/sql-proxy
Both instances are within the same project and Cloud SQL API is enabled on the GCE instance (plus the default ones.) By my understanding, the default GCE service account should be able to authenticate the proxy.
Using cloud_sql_proxy with -instances seems to work but errors with accessNotConfigured too while trying to connect.
As per the error message, you need make sure that the "Cloud SQL Admin" API is enabled by visiting the provided link.
There's also a GUI way to connect your Compute Engine instance if you're using a 2nd gen GCloud SQL instance. You first get the external IP of your compute engine and include it in the Authorizations tab of your GCloud SQL dashboard. Here's a quick video
Enabling Cloud SQL Admin API will solve your problem: