I have a custom gRPC backend deployed behind an Endpoints Service Proxy (ESP) connected to Google Cloud Endpoints.
When sending a request with the X-Cloud-Trace-Context header set, I can see the spans recorded by ESP show up in my Stackdriver Trace dashboard.
However, my service is also sending requests to Google Cloud KMS as part of handling that request. I'd like Google Cloud to create trace spans for those sub-requests automatically for me as well; however, attaching the X-Cloud-Trace-Context header that ESP forwarded to me to the sub-requests sent to Cloud KMS does not cause any spans for those sub-requests to show up in Stackdriver Trace. The service account used to connect to Cloud KMS does have the "Stackdriver Trace Agent" role enabled.
Is it possible to tell Google Cloud services (such as Cloud KMS) to automatically generate trace spans for the current request's trace context, or do I need to manually generate traces for these requests in my backend code?
Cloud Trace doesn't currently generate service-side traces for requests to most GCP services, although we're aware of it as a valuable feature. To track how much of your latency is being consumed by KMS (or other services) you can create a client-side trace record using OpenCensus (Github) or similar.
Cloud KMS (as of this writing) doesn't support gRPC, but we are working on it.
Related
I installed managed Anthos on a GKE cluster. Anthos Service Mesh is working and is displaying my API. Thanks to that Services that are in Monitoring automatically detect my API. This is great as it enables me to easily set SLOs and Error Budget for my API.
However I would like to be able to easily set SLOs for individual endpoints in my api. Services(in Monitoring) detect only my API and not the endpoints within my API(my API is one pod/container + sidecar). I tried to add endpoints to Services in Monitoring but it looks like it is only possible to add Kubernetes Objects there.
Is there a way to use Services in Monitoring with endpoints? Is the only way to do so to break endpoints to separate microservices?
You can monitor your endpoints using Cloud Endpoints with OpenAPI, which allows you to monitor the health of APIs you own by using the logs and metrics Cloud Endpoints maintains for you automatically. When users make requests to your API, Endpoints logs information about the requests and responses and also tracks three of the four golden signals of monitoring: latency, traffic, and errors. These usage and performance metrics help you monitor your API.
The following URL Configuring Cloud Endpoints has the configuration process for Cloud Endpoints. Use this URL Monitoring your API as a reference on the monitoring process for your API, and this last URL for the Cloud Endpoint’s overview.
I'm having some trouble setting uptime checks for some Cloud Run services that don't allow unauthenticated invocations.
For context, I'm using Cloud Endpoints + ESPv2 as an API gateway that's connected to a few Cloud Run services.
The ESPv2 container/API gateway allows unauthenticated invocations, but the underlying Cloud Run services do not (since requests to these backends flow via the API gateway).
Each Cloud Run service has an internal health check endpoint that I'd like to hit periodically via Cloud Monitoring uptime checks.
This serves the purpose of ensuring that my Cloud Run services are healthy, but also gives the added benefit of reduced cold boot times as the containers are kept 'warm'
However, since the protected Cloud Run services expect a valid authorisation header all of the requests from Cloud Monitoring fail with a 403.
From the Cloud Monitoring UI, it looks like you can only configure a static auth header, which won't work in this case. I need to be able to dynamically create an auth header per request sent from Cloud Monitoring.
I can see that Cloud Scheduler supports this already. I have a few internal endpoints on the Cloud Run services (that aren't exposed via the API gateway) that are hit via Cloud Scheduler, and I am able to configure an OIDC auth header on each request. Ideally, I'd be able to do the same with Cloud Monitoring.
I can see a few workarounds for this, but all of them are less than ideal:
Allow unauthenticated invocations for the underlying Cloud Run services. This will make my internal services publicly accessible and then I will have to worry about handling auth within each service.
Expose the internal endpoints via the API gateway/ESPv2. This is effectively the same as the previous workaround.
Expose the internal endpoints via the API gateway/ESPv2 AND configure some sort of auth. This sort of works but at the time of writing the only auth methods supported by ESPv2 are API Keys and JWT. JWT is already out of the question but I guess an API key would work. Again, this requires a bit of set up which I'd rather avoid if possible.
Would appreciate any thought/advice on this.
Thanks!
This simple solution may work on your use case as it is easier to just use a TCP uptime check on port 443:
Create your own Cloud Run service using https://cloud.google.com/run/docs/quickstarts/prebuilt-deploy.
Create a new uptime check on TCP port 443 Cloud Run URL.
Wait a couple of minutes.
Location results: All locations passed
Virginia OK
Oregon OK
Iowa OK
Belgium OK
Singapore OK
Sao Paulo OK
I would also like to advise that Cloud Run is a Google fully managed product and it has a 99.95 % monthly up time SLA, with no recent incidents in the past few months, but proactively monitoring this on your end is a very good thing too.
I am preparing for AWS exam and I found some documentation about AWS CloudTrail and AWS X-RAY where it creates confusion on their usage requirement.
I have came across following question where requirement was to trace and analyse the user request as it travels through Amazon API Gateway APIs to underlying services.
As per my understanding, we can use CludTrail to trace and analyse the user request. But the correct answer was AWS XRAY.
The documents which have referred mentions that, we can use AWS CloudTrail logs for tracing,Security Analysis, Resource Change Tracking and Compliance/Auditing. On the other hand, we can use AWS X-RAY to analyse and debug applications running on distributed micro service architecture.
XRAY and CloudTrail usage both have the term Analyse and trace. So it is quite confusing to which service should we choose under such requirement to trace and analyse the user request
X-Ray is more detailed in the information it provides for the request's flow and state. It scans the request all the way through its lifetime from when it is received in the api gateway to whatever services are called and executed after that. So I imagine that is why it is the preferred option.
I am using google cloud storage and bigquery services of google cloud platform. I'm using service account to authenticate my application and cloud sdk to perform any actions on them.As of now, I'm able to connect and also able to perform any actions. I am curious to know that let suppose if my service account file get modified or somehow the path of that file get lost. Is there is anyway to verify my service account whether it is valid account or not and based on that I can print my code(error codes) and take action on that?
Using Google Cloud SDK commands, you can request additional debugging information:
Google Cloud gcloud has two flags that give user control over information that are displayed:
--log-http Logs all HTTP server requests
--verbosity Can display error or critical`
Cloud Storage gsutil has two options:
-D requests additional debug information
-DD requests full HTTP upstream payload
However, to have greater control over processes and errors, use Google Cloud Client Libraries, to authenticate and access Cloud Storage.
Is Pub/Sub significantly faster way of communicating between, say, Kubernetes Engine (GKE) api server and a Cloud Function (GCF)?
Is it possible to use Pub/Sub to have such communication between GKE from one Google Cloud Project and GCF from another Google Cloud Project?
What is the way to communicate with Cloud Functions from another Google Cloud Project with low latency?
I think a global answer will clarify your questions. For this particular case, there are two ways to trigger a Google Cloud Function (GCF). You can directly make an HTTP request or you can subscribe the GCF to a topic by using Pub/Sub [https://cloud.google.com/functions/docs/calling ].
If your requests are occasional, an HTTP request will be faster because you don't need an intermediary. If that's not the case, then the Pub/Sub subscription queues the messages and ensures the delivery by retrying them until it receives confirmation.
To communicate between Google Kubernetes Engine (GKE) from one Google Cloud Project and Google Cloud Function (GCF) to another Google Cloud Project you can use either option. Trigger the GCF by the HTTP request directly or do it by publishing the message. When publishing, specify the project where you are sending it and the desirable topic in that project.
Also you need to give the proper permission to the service account to access from one project to the other:
For Pub/Sub https://cloud.google.com/pubsub/docs/authentication
For HTTP request
https://cloud.google.com/solutions/authentication-in-http-cloud-functions.
Google Cloud Function HTTP triggers documentation here: https://cloud.google.com/functions/docs/calling/http
Pub/Sub documentation here:
https://cloud.google.com/pubsub/docs/reference/libraries (you can
access to GitHub by the links in the code and see functions examples
for each language)