my google cloud instance(compute engine) was shut down automatically with unknown reason
there is the event log for the instance
{"shutdownEvent":{},"bootCounter":"6","#type":"type.googleapis.com/cloud_integrity.IntegrityEvent"}
{
insertId: "0"
jsonPayload: {
#type: "type.googleapis.com/cloud_integrity.IntegrityEvent"
bootCounter: "6"
shutdownEvent: {
}
}
logName: "projects/<xxxx>/logs/compute.googleapis.com%2Fshielded_vm_integrity"
receiveTimestamp: "2019-05-09T10:14:29.112673979Z"
resource: {
labels: {
instance_id: "<xxxx>"
project_id: "<xxxx>"
zone: "us-west1-b"
}
type: "gce_instance"
}
severity: "NOTICE"
timestamp: "2019-05-09T10:14:26.928978844Z"
}
gcloud compute instances update INSTANCE_NAME --shielded-learn-integrity-policy
Instance must be running and have Integrity Monitoring enabled.
I don't know if this is secure though. Read more:
https://cloud.google.com/compute/docs/instances/integrity-monitoring#updating-baseline
This seems to be a shielded VM with "integrity monitoring" enabled
... and most likely this is caused by an integrity validation failure.
You might use Stackdriver Advanced Logs Filters and try to get more details about the reason behind the shutdown, for example, sometimes instances are migrated (Live Migration) and cause the behavior reported.
Some filters that may help you to determine what caused the shutdown on instances are:
Checking for migrations, in Stackdriver Advanced Logging filter, use following entries:
compute.instances.migrationOnHostMaintenance
OR
compute.instances.migrateOnHostMaintenance
For hostError, you can use following filter:
"compute.instances.hostError"
For VM Instance Stop:
resource.type="gce_instance"
protoPayload.methodName="v1.compute.instances.stop"
had the some problem and confirmed from gcp that my vm had moved from one physics host to another..
pls feel free to open case and google customer support will help you diagnostics
Related
I am trying to setup a cloud replication for my master/slave db. The master resides on an external vpc and I want to set up a slave in google cloud sql. I have followed the steps as here to setup the databases.
They are setup fine and I can see initial replication taking place from my master. The data is synchronized. However shortly after, it becomes disabled for replication. I cannot seem to start it again to replicate and each time gives the following error
The instance or operation is not in an appropriate state to handle the request.
I checked the suggestions from here but that didnt work.
Running gcloud sql instances describe replica-instance1 gives me the following (excerpt):
state: RUNNABLE
replicaConfiguration:
failoverTarget: false
kind: sql#replicaConfiguration
I can update if you need more of the results but that all looks fine. Can anyone help?
Edit:
This is in the postgresql logs
resource: {
labels: {3}
type: "cloudsql_database"
}
severity: "ERROR"
textPayload: "2023-01-20 22:10:36.354 UTC [282]: [2-1] db=postgres,user=[unknown] ERROR: data stream ended"
timestamp: "2023-01-20T22:10:36.354863Z"
}
I am using gcp functions, and I think the function crush log is not informative like below, which only mention finished with status: 'crash'.
Wondering how we can get a crush stack log to find out the bug easier?
Thank you.
{
insertId: "000000-e410caa4-0287-4e3f-b80d-c215e06a5430"
labels: {
execution_id: "pbc4oxnzl1ji"
}
logName: "projects/sturdy-hangar-2/logs/cloudfunctions.googleapis.com%2Fcloud-functions"
receiveTimestamp: "2020-07-04T07:01:52.837476035Z"
resource: {
labels: {
function_name: "gen_thumbnail"
project_id: "sturdy-hangar-2"
region: "us-central1"
}
type: "cloud_function"
}
severity: "DEBUG"
textPayload: "Function execution took 91 ms, finished with status: 'crash'"
timestamp: "2020-07-04T07:01:51.834276757Z"
trace: "projects/sturdy-hangar-2/traces/1bbc73e71f5057d38ecb65be525b233b"
}
I understand that ambiguous crash logs without stack traces can be quite frustrating to troubleshoot and test. This is a known issue with Cloud Functions as of now and is actively being investigated by Cloud Functions specialists. I would suggest that you take a look at the Public Tracker and click the star icon to subscribe to the issue for further updates and click the bell icon to be notified of updates via email.
As all issues require a dedicated team of specialists to investigate them accordingly, we do not have an ETA on the resolution. Please rest assured that we are doing our best to get this issue resolved.
I have created a GCP Cloud Scheduler job to run every 15 minutes. It is supposed to call an API from my Node js application.
In the console the job definition looks like this:
Description: A job
Frequency: */15 * * * *
Timezone: Central Standard Time
Target: HTTP
URL: https://<company url>/api/email-reminder/
HTTP method: GET
Auth header: Add OIDC token
Service account: xxxxxxxxxxx-compute#developr.gserviceaccount.com
When it runs it returns the following in the logs:
httpRequest: {
}
insertId: "15wxxxxxxge1lv"
jsonPayload: {
#type: "type.googleapis.com/google.cloud.scheduler.logging.AttemptFinished"
jobName: "projects/<project name>/locations/us-central1/jobs/xxxxxxxxx-scheduler-emailreminders-1"
status: "UNKNOWN"
targetType: "HTTP"
url: "https://<company url>/api/email-reminder/"
}
logName: "projects/<project name>/logs/cloudscheduler.googleapis.com%2Fexecutions"
receiveTimestamp: "2019-11-14T04:45:50.280446452Z"
resource: {
labels: {…}
type: "cloud_scheduler_job"
}
severity: "ERROR"
timestamp: "2019-11-14T04:45:50.280446452Z"
}
How do I find out more information about the error?
The default value for timeout scheduler jobs process is 180s, you can change vía gcloud command
gcloud scheduler jobs update http my-super-job --attempt-deadline 540s
As well you can see the complete info of your jobs with this commands...
gcloud scheduler jobs list
gcloud scheduler jobs describe my-super-job
I've seen similar problems recently using Cloud Scheduler for HTTPS targets. Every so often, the scheduler will fail, and all I get is a log message like yours.
Viewing in the Logs Viewer, the key parts of the log are, in the log header:
"status":"RESOURCE_EXHAUSTED",
"#type":"type.googleapis.com/google.cloud.scheduler.logging.AttemptFinished"
and in the log data:
httpRequest: {
status: 429
}
jsonPayload: {
#type: "type.googleapis.com/google.cloud.scheduler.logging.AttemptFinished"
jobName: "projects/joburlhere"
status: "RESOURCE_EXHAUSTED"
targetType: "HTTP"
url: "https://urlgoeshere"
}
severity: "ERROR"
"Resource exhausted" is the description of the 429 error code.
There's a description of this code here:
https://cloud.google.com/apis/design/errors
429 RESOURCE_EXHAUSTED Either out of resource quota or reaching rate limiting. The client should look for google.rpc.QuotaFailure error detail for more information.
Given that I'm running this Job once an hour, and the receiver is a very modest Cloud Function, I don't I'm doing anything to cause resource exhaustion. So I think it's a recurring transient problem with Google's cloud infrastructure. I'm guessing that the cloud functions were unavailable for that particular request, and because I set up the Cloud Function with default settings, the Scheduler did not retry.
Additionally, it is possible to configure a Scheduler Job to retry in case of failure. This functionality is not shown in the web console, but you can control it using the gcloud command.
The default setting is not to retry.
Look at the --max-retry-attempts flag.
https://cloud.google.com/sdk/gcloud/reference/scheduler/jobs/update/http
There are similar controls for pubsub Jobs
https://cloud.google.com/sdk/gcloud/reference/scheduler/jobs/update/pubsub
It's because the http endpoint you specified doesn't return the response within the default attempt deadline.
Refer the link
Purpose
Use Google Cloud Diagnostics on a .net core 2.2 REST API, for Logging, Tracing and Error Reporting in two possible scenarios:
Local execution on Visual Studio 2017
Deployed on a Docker Container and Running on a GCP Kubernetes Engine
Environment details
.NET version: 2.2.0
Package name and version: Google.Cloud.Diagnostics.AspNetCore 3.0.0-beta13
Description
For configuring Google Cloud Diagnostics, two documentation sources were used:
Google.Cloud.Diagnostics.AspNetCore
Enable configuration support for Google.Cloud.Diagnostics.AspNetCore #2435
Based on the above documentation the UseGoogleDiagnostics extension method on IWebHostBuilder was used, as this configures Logging, Tracing and Error Reporting middleware.
According to the 2) link, the following table presents the information needed when using the UseGoogleDiagnostics method:
For local execution => project_id, module_id, and version_id are needed,
For GKE => module_id, and version_id
The .net core configuration files were used to provide the above information for each deployment:
appsettings.json
{
"GCP": {
"ServiceID": "my-service",
"VersionID": "v1"
}
}
appsettings.Development.json
{
"GCP": {
"ID": "my-id"
}
}
Basically, the above will render the following configuration:
On Local execution
return WebHost.CreateDefaultBuilder(args)
.UseGoogleDiagnostics("my-id", "my-service", "v1")
.UseStartup<Startup>();
On GKE
return WebHost.CreateDefaultBuilder(args)
.UseGoogleDiagnostics(null, "my-service", "v1")
.UseStartup<Startup>();
To guarantee i'm using the correct information, i used two places on the GCP UI to verify:
On Endpoints Listing, checked the service details:
Service name: my-service Active version: v1
Checked the Endpoint logs, for a specific API POST Endpoint
{
insertId: "e6a63a28-1451-4132-ad44-a4447c33a4ac#a1"
jsonPayload: {…}
logName: "projects/xxx%2Fendpoints_log"
receiveTimestamp: "2019-07-11T21:03:34.851569606Z"
resource: {
labels: {
location: "us-central1-a"
method: "v1.xxx.ApiOCRPost"
project_id: "my-id"
service: "my-service"
version: "v1"
}
type: "api"
}
severity: "INFO"
timestamp: "2019-07-11T21:03:27.397632588Z"
}
Am i doing anything wrong or is it a bug on Google.Cloud.Diagnostics.AspNetCore 3.0.0-beta13?
When executing the service Endpoints, for each specific deployment, Google Cloud Diagnostics behaves differently:
On Local execution (VS2017) => Logging, Tracing and Error Reporting work as expected, everything showing in GCP UI
On GKE Deployment => Logging, Tracing and Error Reporting DO NOT Work, nothing shows in GCP UI
I've tried several variations, hardcoding the values directly in the code, etc, but no matter what i do, Google Cloud Diagnostics is not working when deployed in GKE:
Hardcoding the values directly
return WebHost.CreateDefaultBuilder(args)
.UseGoogleDiagnostics(null, "my-service", "v1")
.UseStartup<Startup>();
Without the v in Version
return WebHost.CreateDefaultBuilder(args)
.UseGoogleDiagnostics(null, "my-service", "1")
.UseStartup<Startup>();
I'm kind of new on IOT world and trying to learn a bit about GCP products.
Ive made a simples python app that uses PAHO to send a message to an IOT topic (IOT Core in GCP).
Everything, apparently, works just fine. But I was wondering if I could see, on stackdriver, the content of a message that the device had sent.
I already have enable debuging log for it, but the message didnt show up.
Publish Log in stackdriver:
{
insertId: "78178yfwnl"
jsonPayload: {
eventType: "PUBLISH"
protocol: "MQTT"
publishFromDeviceTopicType: "EVENTS"
resourceName: "projects/demoiot/locations/us-central1/registries/iotchicago/devices/2753540639583"
serviceName: "cloudiot.googleapis.com"
status: {
code: 0
}
}
labels: {
device_id: "us_chi"
}
logName: "projects/demoiot/logs/cloudiot.googleapis.com%2Fdevice_activity"
receiveTimestamp: "2018-11-20T11:10:01.123928203Z"
resource: {
labels: {
device_num_id: "2753540639583"
device_registry_id: "iotchicago"
location: "us-central1"
project_id: "demoiot-223010"
}
type: "cloudiot_device"
}
severity: "DEBUG"
timestamp: "2018-11-20T11:10:01.104415969Z"
}
No telemetry data will be logged by our system. The potential for privacy concerns because of permissions to logs vs. permission on the telemetry itself is such that we didn't want to touch that.
You CAN write to Stackdriver explicitly though, so a way to do that would be to have a Cloud Function tied to the Pub/Sub topic where the telemetry is being written, and then have that function write messages out to Stackdriver with the payload data. Could also do it with DataFlow if Java is more your thing.
A teammate also pointed out to me, using the /state/ MQTT topic to write out the device's state and checking it in the GCP console is also a good way to quickly test/check. In the device details, there's a tab for "Configuration and State History" that will show you.