A few days ago we started to receive a huge amount of logs generated by the the external load balancer with statusDetails: "internal_error"
This is an example:
{
"insertId": "h7bxsuflmbgkx",
"jsonPayload": {
"#type": "type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry",
"remoteIp": "remote_ip_replaced",
"statusDetails": "internal_error"
},
"httpRequest": {
"remoteIp": "remote_ip_replaced",
"latency": "0s"
},
"resource": {
"type": "http_load_balancer",
"labels": {
"backend_service_name": "",
"url_map_name": "",
"zone": "global",
"project_id": "project_id_replaced",
"target_proxy_name": "",
"forwarding_rule_name": "rule_name_replaced"
}
},
"timestamp": "2022-10-06T19:04:48.807346Z",
"severity": "INFO",
"logName": "projects/project_id_replaced/logs/requests",
"receiveTimestamp": "2022-10-06T19:04:49.013874339Z"
}
The amount of logs that we see is approximately 2 million / hour.
The LB is configured with 4 backend services with serverless NEG to Cloud Run.
No configuration changes in the last 15 days.
The traffic of our application/backend services is the same and it does not seem to be affected by this situation.
Any idea what could be the cause of these error logs?
Related
I'm struggling to deploy my cloud function. I'm unsure what information to provide. My set up:
# main.py
def callRequest():
print("bla")
return(1)
Entry point for the function is callRequest.
After failing to deploy I see this red highlighted message under details:
Deployment failure:
Build failed: {"metrics":{},"error":{"buildpackId":"","buildpackVersion":"","errorType":"OK","canonicalCode":"OK","errorId":"","errorMessage":""},"stats":[{"buildpackId":"google.utils.archive-source","buildpackVersion":"0.0.1","totalDurationMs":47,"userDurationMs":46},{"buildpackId":"google.python.runtime","buildpackVersion":"0.9.1","totalDurationMs":9487,"userDurationMs":6307},{"buildpackId":"google.python.functions-framework","buildpackVersion":"0.9.6","totalDurationMs":53,"userDurationMs":52},{"buildpackId":"google.python.pip","buildpackVersion":"0.9.2","totalDurationMs":5832,"userDurationMs":5822},{"buildpackId":"google.utils.label","buildpackVersion":"0.0.2","totalDurationMs":0,"userDurationMs":0}],"warnings":null,"customImage":false}
In the logs I see a notice related to the attempted deploy:
{
"protoPayload": {
"#type": "type.googleapis.com/google.cloud.audit.AuditLog",
"authenticationInfo": {
"principalEmail": "myname#bla.com"
},
"requestMetadata": {
"callerIp": "152.170.106.184",
"callerSuppliedUserAgent": "Mozilla/5.0 (X11; Linux x86_64; rv:108.0) Gecko/20100101 Firefox/108.0,gzip(gfe),gzip(gfe)",
"requestAttributes": {
"time": "2023-01-11T13:15:30.667011Z",
"auth": {}
},
"destinationAttributes": {}
},
"serviceName": "cloudfunctions.googleapis.com",
"methodName": "google.cloud.functions.v1.CloudFunctionsService.UpdateFunction",
"authorizationInfo": [
{
"resource": "projects/my-project/locations/us-central1/functions/ga4-to-s3-1",
"permission": "cloudfunctions.functions.update",
"granted": true,
"resourceAttributes": {}
}
],
"resourceName": "projects/my-project/locations/us-central1/functions/ga4-to-s3-1",
"request": {
"updateMask": "entryPoint,sourceUploadUrl",
"#type": "type.googleapis.com/google.cloud.functions.v1.UpdateFunctionRequest",
"function": {
"name": "projects/my-project/locations/us-central1/functions/ga4-to-s3-1",
"runtime": "python39",
"serviceAccountEmail": "my-project#appspot.gserviceaccount.com",
"availableMemoryMb": 256,
"maxInstances": 3000,
"timeout": "60s",
"eventTrigger": {
"eventType": "google.pubsub.topic.publish",
"resource": "projects/my-project/topics/ga4-daily-extract-complete"
},
"secretEnvironmentVariables": [
{
"version": "latest",
"key": "PAT",
"secret": "PAT-GA4-S3-Extract",
"projectId": "1234567"
}
],
"sourceUploadUrl": "https://storage.googleapis.com/uploads-1234567.us-central1.cloudfunctions.appspot.com/123-456-789-abc-def.zip?GoogleAccessId=service-123456789#gcf-admin-robot.iam.gserviceaccount.com&Expires=12345&Signature=kjhgfghjkjhg%iuytfrghj8765467uhgfdfghj",
"entryPoint": "callRequest",
"ingressSettings": "ALLOW_ALL"
}
},
"resourceLocation": {
"currentLocations": [
"us-central1"
]
}
},
"insertId": "nlbq4xd9dhq",
"resource": {
"type": "cloud_function",
"labels": {
"project_id": "my-project",
"function_name": "ga4-to-s3-1",
"region": "us-central1"
}
},
"timestamp": "2023-01-11T13:15:30.423213Z",
"severity": "NOTICE",
"logName": "projects/my-project/logs/cloudaudit.googleapis.com%2Factivity",
"operation": {
"id": "operations/Z2E0LWV4dHJhY3QvdXMtY2VudHJhbDEvZ2E0LXRvLXMzLTEvbHA2QlowNzBTekk",
"producer": "cloudfunctions.googleapis.com",
"first": true
},
"receiveTimestamp": "2023-01-11T13:15:31.626931279Z"
}
I'm unsure where else to look? Any pointers or advice most welcome.
Found the similar issue discussed here and the issue resolved.
cloud build service account was missing the Cloud Build Service Account role.
I tried removing the Cloud Build Service Account role and deployed the function, I also got the same deployment errors
Try adding the Cloud Build Service Account role for the Google Cloud Build Service Account (project-number#cloudbuild.gserviceaccount.com) in the Google Cloud IAM console . This fixed symptom of a cloud function deploy with the message:
message=Build failed: {
"metrics":{},
"error":{
"buildpackId":"",
"buildpackVersion":"",
"errorType":"OK",
"canonicalCode":"OK",
"errorId":"",
"errorMessage":""
}
}
Also have a look at this github link1 & link2 which might help
We're trying to deploy firebase functions and we continuously have this error: The service has encountered an error during container import. Please try again late
Here's a video overview: https://www.loom.com/share/9afb2facb5e3461ebef74e7e802a2761
{
"protoPayload": {
"#type": "type.googleapis.com/google.cloud.audit.AuditLog",
"status": {
"code": 14,
"message": "The service has encountered an error during container import. Please try again later"
},
"authenticationInfo": {},
"serviceName": "cloudfunctions.googleapis.com",
"methodName": "google.cloud.functions.v1.CloudFunctionsService.UpdateFunction",
"resourceName": "projects/voypost-matching-prod/locations/europe-west3/functions/createFromJobPubSub"
},
"insertId": "-wq3kwnb2c",
"resource": {
"type": "cloud_function",
"labels": {
"project_id": "voypost-matching-prod",
"function_name": "createFromJobPubSub",
"region": "europe-west3"
}
},
"timestamp": "2023-01-07T16:39:01.688761Z",
"severity": "ERROR",
"logName": "projects/voypost-matching-prod/logs/cloudaudit.googleapis.com%2Factivity",
"operation": {
"id": "operations/dm95cG9zdC1tYXRjaGluZy1wcm9kL2V1cm9wZS13ZXN0My9jcmVhdGVGcm9tSm9iUHViU3ViL0pOaGkxYW9zMWxj",
"producer": "cloudfunctions.googleapis.com",
"last": true
},
"receiveTimestamp": "2023-01-07T16:39:02.021123649Z"
}
As per this doc, it may be due to non-utf8 characters, but there are none (checked using https://stackoverflow.com/a/41741313 grep -axv '.*' ./lib/**/*.js)
It failed 3 times in a row and continues failing. And every time it fails on the same functions.
There is always only one deployment ongoing - we don't run multiple firebase functions deploy at the same time.
The original discussion is on their github, but we were refered here.
Facing this issue with my Google Cloud Functions where from the very first function that I deployed to the ones I'm to upgrade today, are all saying the same thing on their status.
"Function is active, but the last deploy failed"
What may this be?
Here's the log visible for updating the function on the log explorer.
{
"protoPayload": {
"#type": "type.googleapis.com/google.cloud.audit.AuditLog",
"status": {},
"authenticationInfo": {
"principalEmail": "start#pyme.team"
},
"serviceName": "cloudfunctions.googleapis.com",
"methodName": "google.cloud.functions.v1.CloudFunctionsService.UpdateFunction",
"resourceName": "projects/pyme-webapp/locations/us-central1/functions/applicationSubmitted"
},
"insertId": "d1k3hyd3jfe",
"resource": {
"type": "cloud_function",
"labels": {
"region": "us-central1",
"function_name": "applicationSubmitted",
"project_id": "pyme-webapp"
}
},
"timestamp": "2022-02-02T20:23:05.726462Z",
"severity": "NOTICE",
"logName": "projects/pyme-webapp/logs/cloudaudit.googleapis.com%2Factivity",
"operation": {
"id": "operations/cHltZS13ZWJhcHAvdXMtY2VudHJhbDEvYXBwbGljYXRpb25TdWJtaXR0ZWQvaWdGS2o4bXpjbDA",
"producer": "cloudfunctions.googleapis.com",
"last": true
},
"receiveTimestamp": "2022-02-02T20:23:06.263576440Z"
}
Similarly, all I see on the log in the function itself is:
Image of the Function Log itself available
The exact error that I am seeing and am concerned about and with is this: Function Error with ORANGE HAZARD on update
Attaching another, even more detailed update log as well.
{
"protoPayload": {
"#type": "type.googleapis.com/google.cloud.audit.AuditLog",
"authenticationInfo": {
"principalEmail": "start#pyme.team"
},
"requestMetadata": {
"callerIp": "80.83.136.68",
"callerSuppliedUserAgent": "FirebaseCLI/10.0.1,gzip(gfe),gzip(gfe)",
"requestAttributes": {
"time": "2022-02-02T20:21:00.491300Z",
"auth": {}
},
"destinationAttributes": {}
},
"serviceName": "cloudfunctions.googleapis.com",
"methodName": "google.cloud.functions.v1.CloudFunctionsService.UpdateFunction",
"authorizationInfo": [
{
"resource": "projects/pyme-webapp/locations/us-central1/functions/workContracts",
"permission": "cloudfunctions.functions.update",
"granted": true,
"resourceAttributes": {}
}
],
"resourceName": "projects/pyme-webapp/locations/us-central1/functions/workContracts",
"request": {
"updateMask": "name,sourceUploadUrl,entryPoint,runtime,labels,httpsTrigger,availableMemoryMb,environmentVariables,sourceToken",
"function": {
"runtime": "nodejs16",
"availableMemoryMb": 512,
"entryPoint": "workContracts",
"name": "projects/pyme-webapp/locations/us-central1/functions/workContracts",
"sourceUploadUrl": "https://storage.googleapis.com/gcf-upload-us-central1-d393f99f-6b88-4b68-8202-d75b734aa7a1/64b2646f-35b6-4919-8e89-c662fc29f01f.zip?GoogleAccessId=service-748321615979#gcf-admin-robot.iam.gserviceaccount.com&Expires=1643835053&Signature=McjqD9mmo%2F1wLbvO6SklkHi%2B34nQEwcpz7cLOLNAF4RwG8bpHh8RThxFJwnGZo1F92iQnquRQyGYbJFuihP%2FUGrgW7cG6GmhVq2gkugDywngZXT9d7UTBG0wgKF29XcbZkwV3IX7oKKiUwf6Q6mzCOOoCrjc5LBxqJo9WvWDZynv8R75nVZTZ5IhekMdqAw%2BRvIBvooXa%2BuA3Sezhh%2Bz2BR1XtIyS21CY%2FkoPDaKPwvftr3%2Fjcyuzb2V39%2BSajQg3t0U7Gt6oSch9qUhl6gnknr6wphFGmC7t7h9l0LUbjHUDuaMNNoB1LXxI30CRNkRupf9XBKTKpKMf%2F0nAAMltA%3D%3D",
"httpsTrigger": {},
"labels": {
"deployment-tool": "cli-firebase"
}
},
"#type": "type.googleapis.com/google.cloud.functions.v1.UpdateFunctionRequest"
},
"resourceLocation": {
"currentLocations": [
"us-central1"
]
}
},
"insertId": "1g6c2gwd46lm",
"resource": {
"type": "cloud_function",
"labels": {
"region": "us-central1",
"function_name": "workContracts",
"project_id": "pyme-webapp"
}
},
"timestamp": "2022-02-02T20:21:00.307699Z",
"severity": "NOTICE",
"logName": "projects/pyme-webapp/logs/cloudaudit.googleapis.com%2Factivity",
"operation": {
"id": "operations/cHltZS13ZWJhcHAvdXMtY2VudHJhbDEvd29ya0NvbnRyYWN0cy96bHlTLUtwbzI2VQ",
"producer": "cloudfunctions.googleapis.com",
"first": true
},
"receiveTimestamp": "2022-02-02T20:21:00.985842395Z"
}
If this isn't the log to look for, just let me know what to find but I'd appreciate the help.
So turns out today morning, I login and check and everything is fine. I still have no logs stating the exact cause of the error but the same functions, the same code and the exact same deployment methods have worked and the function seems to be working fine.
This is concerning as separate cloud functions should never ever be changing on deployments.
A cloud function which takes in a POST METHOD and send data to SendGrid for example has nothing to do with a cloud function triggered by updates to the Firestore Database and if they're both deployed since the 5th of January and never touched again (in terms of edits), they should not be showing the same deployment error message across the board.
my temporal solution is to delete the function then deploy. It seems like it cannot be deployed while in use, i'm sorry i couldn't provide a better solution i will edit it as soon as possible.
i have few lambdas that use different other services like SSM, athena, dynamodb, s3, SQS, SNS for my process. i am almost done with all my development and would love to monitor it visually. I use X-ray and cloud watch as my regular log monitoring and analysis. I feel cloud watch dashboards is not so efficient way to visualize my stuff with multiple services. So i did a lambda that pulls trace data from my X-ray traces and outputs a nested json file something like below.
[
{
"id": "4707a33e472",
"name": "test-lambda",
"start_time": 1524714634.098,
"end_time": 1524714672.046,
"parent_id": "1b9122bc",
"aws": {
"function_arn": "arn:aws:lambda:us-east-1:9684596:function:test-lambda",
"resource_names": [
"test-lambda"
],
"account_id": "9684596"
},
"trace_id": "1-5ae14c88-41dca52ccec8c7d",
"origin": "AWS::Lambda::Function",
"subsegments": [
{
"id": "ab6420197c",
"name": "S3",
"start_time": 1524714671.7148032,
"end_time": 1524714671.8333395,
"http": {
"response": {
"status": 200
}
},
"aws": {
"id_2": "No9Gemg5b9Y2XREorBG+6a1KLXX7S6O3HtPZ3f6vUuU5F1dQE0nIE1WmwmRRHIqCjI=",
"operation": "DeleteObjects",
"region": "us-east-1",
"request_id": "E2709BB91B8"
},
"namespace": "aws"
},
{
"id": "370e11d6d",
"name": "SSM",
"start_time": 1524714634.0991564,
"end_time": 1524714634.194922,
"http": {
"response": {
"status": 200
}
},
"aws": {
"operation": "GetParameter",
"region": "us-east-1",
"request_id": "f901ed67-4904-bde0-f9ad15cc558b"
},
"namespace": "aws"
},
{
"id": "8423bf21354",
"name": "DynamoDB",
"start_time": 1524714671.9744427,
"end_time": 1524714671.981935,
"http": {
"response": {
"status": 200
}
},
"aws": {
"operation": "UpdateItem",
"region": "us-east-1",
"request_id": "3AHBI44JRJ2UJ72V88CJPV5L4JVV4K6Q9ASUAAJG",
"table_name": "test-dynamodb",
"resource_names": [
"test-dynamodb"
]
},
I only posted the first few line of x-ray trace json output, but it's pretty large to post here. AWS quicksight doesn't support nested json, my question is, is there a way to visualize all my lambdas in a better way using quicksight. I am not allowed to use other third party monitoring systems. Need help with this
I have created a simple ShellCommandActivity which echos some text. It runs on a plain ec2 (vpc) instance. I see that the host has spinned up but it never executes the tasks and the task remains in WAITING_FOR_RUNNER status. After all the retries I get this error
Resource is stalled. Associated tasks not able to make progress.
I followed this troubleshoot-link but it didn't resolve my problem.
Here is the json description of the pipeline:
{
"objects": [
{
"resourceRole": "DataPipelineDefaultResourceRole",
"role": "DataPipelineDefaultRole",
"name": "ec2-compute",
"id": "ResourceId_viWO9",
"type": "Ec2Resource"
},
{
"failureAndRerunMode": "CASCADE",
"resourceRole": "DataPipelineDefaultResourceRole",
"role": "DataPipelineDefaultRole",
"pipelineLogUri": "s3://xyz-logs/",
"scheduleType": "ONDEMAND",
"name": "Default",
"id": "Default"
},
{
"name": "EchoActivity",
"id": "ShellCommandActivityId_kc8xz",
"runsOn": {
"ref": "ResourceId_viWO9"
},
"type": "ShellCommandActivity",
"command": "echo HelloWorld"
}
],
"parameters": []
}
What could be the problem here?
Thanks in advance.
I figured this out. The routing table in the VPC subnets was not properly configured.
To be specific, in my case the routing table didn't have 0.0.0.0/0 mapped to an internet-gateway. When I added this mapping, everything started working.