Setup
AWS Lambda (3s timeout)
NodeJS 12.x
mssql 2.6.1
tedious (dependency of mssql and so it installs tedious 6.7.0)
SQL Server DB in RDS (db.t3.small)
I'm also dealing with a fair bit of traffic, with roughly 1k invocations per minute.
Problem
Most of the time the Lambda executes just fine. Roughly 0.35% of the time the Lambda throws an error. The logs look like this:
In the screenshot you can see that the function STARTs, prints some debug info, then throws an error, ENDs, and REPORTs.
While this is a "timeout" error, the error message says,
RequestError: Timeout: Request failed to complete in 15000ms
This confuses me because as you see in the REPORT log, the invocation time was just 255.33ms total.
Question
The obvious question is how does something timeout after 15 seconds in just 255ms? Is this an issue with tedious, mssql, my code, or something else? If my code is relevant to the question please let me know and I can add it. I assume the code is basically functional because it works > 99% of the time.
Failed Theories:
The logs are interleaved and the errors are not from the 225ms invocation. That's wrong because in the screenshot the request ID's match up.
There's an intermittent error connecting to the DB, possibly a DNS issue. I have seen very rare EAIAGAIN DNS errors when resolving the DB host, but that doesn't seem to match with the steadiness of this error.
I didn't spot anything super helpful in the GitHub issues for Tedious.
Full Error
{
"errorType": "Runtime.UnhandledPromiseRejection",
"errorMessage": "RequestError: Timeout: Request failed to complete in 15000ms",
"reason": {
"errorType": "RequestError",
"errorMessage": "Timeout: Request failed to complete in 15000ms",
"code": "ETIMEOUT",
"originalError": {
"errorType": "RequestError",
"errorMessage": "Timeout: Request failed to complete in 15000ms",
"code": "ETIMEOUT",
"message": "Timeout: Request failed to complete in 15000ms",
"stack": [
"RequestError: Timeout: Request failed to complete in 15000ms",
" at RequestError (/var/task/node_modules/mssql/node_modules/tedious/lib/errors.js:32:12)",
" at Connection.requestTimeout (/var/task/node_modules/mssql/node_modules/tedious/lib/connection.js:1212:46)",
" at Timeout._onTimeout (/var/task/node_modules/mssql/node_modules/tedious/lib/connection.js:1180:14)",
" at listOnTimeout (internal/timers.js:549:17)",
" at processTimers (internal/timers.js:492:7)"
]
},
"name": "RequestError",
"number": "ETIMEOUT",
"precedingErrors": [],
"stack": [
"RequestError: Timeout: Request failed to complete in 15000ms",
" at Request.userCallback (/var/task/node_modules/mssql/lib/tedious/request.js:429:19)",
" at Request.callback (/var/task/node_modules/mssql/node_modules/tedious/lib/request.js:56:14)",
" at Connection.endOfMessageMarkerReceived (/var/task/node_modules/mssql/node_modules/tedious/lib/connection.js:2407:20)",
" at Connection.dispatchEvent (/var/task/node_modules/mssql/node_modules/tedious/lib/connection.js:1279:15)",
" at Parser.<anonymous> (/var/task/node_modules/mssql/node_modules/tedious/lib/connection.js:1072:14)",
" at Parser.emit (events.js:315:20)",
" at Parser.EventEmitter.emit (domain.js:482:12)",
" at Parser.<anonymous> (/var/task/node_modules/mssql/node_modules/tedious/lib/token/token-stream-parser.js:37:14)",
" at Parser.emit (events.js:315:20)",
" at Parser.EventEmitter.emit (domain.js:482:12)"
]
},
"promise": {},
"stack": [
"Runtime.UnhandledPromiseRejection: RequestError: Timeout: Request failed to complete in 15000ms",
" at process.<anonymous> (/var/runtime/index.js:35:15)",
" at process.emit (events.js:315:20)",
" at process.EventEmitter.emit (domain.js:482:12)",
" at processPromiseRejections (internal/process/promises.js:209:33)",
" at processTicksAndRejections (internal/process/task_queues.js:98:32)"
]
}
Related
i exported my arm template from develop environment related to my Azure Data Explorer.
Now i'am trying to import it in test environment but the process fails:
New-AzResourceGroupDeployment : 12:18:24 - The deployment 'template' failed with error(s). Showing 3 out of 10 error(s). Status Message: [BadRequest] Validation Errors found: mapping does not exist (Code:EventHubValidationErrorFound) Status Message: [BadRequest] Validation Errors found: mapping does not exist (Code:EventHubValidationErrorFound) Status Message: [BadRequest] Validation Errors found: mapping does not exist (Code:EventHubValidationErrorFound) CorrelationId: b27cdf8e-c583-4dee-8dbc-2b0e4876b8ca
I have different Data Connections from my Azure DATA Explorer to an Event HUB:
`
{
"type": "Microsoft.Kusto/Clusters/Databases/EventHubConnections",
"apiVersion": "2018-09-07-preview",
"name": "[concat(parameters('Clusters_xyzazne_name'), '/asd/asd-fondi')]",
"location": "North Europe",
"dependsOn": [
"[resourceId('Microsoft.Kusto/Clusters/Databases', parameters('Clusters__name'), 'DNA_R_NRT')]",
"[resourceId('Microsoft.Kusto/Clusters', parameters('Clusters__name'))]"
],
"kind": "EventHub",
"properties": {
"eventHubResourceId": "[concat(parameters('namespaces_ehub_externalid'), '/eventhubs/fondi')]",
"consumerGroup": "fondi_consumer",
"tableName": "fondi",
"mappingRuleName": "fondi_mapping",
"dataFormat": "multijson"
}
}
`
I'm trying to import an Azure Data Explorer Arm Template to another environment but fails
I was able to create and publish APIs on my dev WSO2 installation based on H2 DB. Unfortunately started from the fifth APIs (the first four still working very well) I have this error.
If I'm calling the API from Postman I receive
{
"code": 404,
"type": "Status report",
"message": "Not Found",
"description": "The requested resource is not available."
}
Into the logs I see
INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = XXXXXXXXXXXXXXX, HEALTH CHECK URL = XXXXXXXXXXXXXXX
Any help will appriciate.
API-M version 4.1.0
Regards,
I am using fluent-ffmpeg nodejs package to run ffmpeg for audio conversion on AWS Lambda. I am using this FFmpeg layer for lambda.
Here is my code
const bitrate64 = ffmpeg("file.mp3").audioBitrate('64k');
bitrate64.outputOptions([
'-preset slow',
'-g 48',
"-map", "0:0",
'-hls_time 6',
'-master_pl_name master.m3u8',
'-hls_segment_filename 64k/fileSequence%d.ts'
])
.output('./64k/prog_index.m3u8')
.on('progress', function(progress) {
console.log('Processing 64k bitrate: ' + progress.percent + '% done')
})
.on('end', function(err, stdout, stderr) {
console.log('Finished processing 64k bitrate!')
})
.run()
after running it via AWS lambda I get following error message
ERROR Uncaught Exception
{
"errorType": "Error",
"errorMessage": "ffmpeg exited with code 1: Conversion failed!\n",
"stack": [
"Error: ffmpeg exited with code 1: Conversion failed!",
"",
" at ChildProcess.<anonymous> (/var/task/node_modules/fluent-ffmpeg/lib/processor.js:182:22)",
" at ChildProcess.emit (events.js:198:13)",
" at ChildProcess.EventEmitter.emit (domain.js:448:20)",
" at Process.ChildProcess._handle.onexit (internal/child_process.js:248:12)"
]
}
I don't get any more info so I am not sure what's going on. Can anyone tell me what's wrong here and how can I enable more detailed logs?
Added on error callback to get a detailed error and found that there are permissions issue on lambda
.on('error', function(err, stdout, stderr) {
if (err) {
console.log(err.message);
console.log("stdout:\n" + stdout);
console.log("stderr:\n" + stderr);
reject("Error");
}
})
I tried to create a wrapper around Google Text-to-Speech copied verbatim from your help pages. From the logs it seems it tried to create a Jetty Instance(to forward the call to another service?) but fails as it can't assign it a network address by reading from the default. I will include the log output for your reference. Can you suggest changes to the code or a different strategy to use when using Cloudfunctions?
Log Output follows:
"textPayload": "java.io.IOException: com.google.api.gax.rpc.DeadlineExceededException: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 19.821560546s. [buffered_nanos=2713815445, buffered_nanos=17108132243, waiting_for_connection]\n\tat com.gcp.cloudfunctions.Test.service(Test.java:91)\n\tat com.google.cloud.functions.invoker.NewHttpFunctionExecutor.service(NewHttpFunctionExecutor.java:67)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:790)\n\tat org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:755)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:547)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\tat com.google.cloud.functions.invoker.runner.Invoker$NotFoundHandler.handle(Invoker.java:379)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:500)\n\tat org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)\n\tat org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)\n\tat org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)\n\tat java.base/java.lang.Thread.run(Thread.java:834)\nCaused by: com.google.api.gax.rpc.DeadlineExceededException: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 19.821560546s. [buffered_nanos=2713815445, buffered_nanos=17108132243, waiting_for_connection]\n\tat com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:51)\n\tat com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:72)\n\tat com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:60)\n\tat com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97)\n\tat com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68)\n\tat com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:982)\n\tat com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)\n\tat com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1138)\n\tat com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:957)\n\tat com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:748)\n\tat io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:522)\n\tat io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:497)\n\tat io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:426)\n\tat io.grpc.internal.ClientCallImpl.access$500(ClientCallImpl.java:66)\n\tat io.grpc.internal.ClientCallImpl$1CloseInContext.runInContext(ClientCallImpl.java:416)\n\tat io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)\n\tat io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\t... 1 more\n\tSuppressed: com.google.api.gax.rpc.AsyncTaskException: Asynchronous task failed\n\t\tat com.google.api.gax.rpc.ApiExceptions.callAndTranslateApiException(ApiExceptions.java:57)\n\t\tat com.google.api.gax.rpc.UnaryCallable.call(UnaryCallable.java:112)\n\t\tat com.google.cloud.texttospeech.v1beta1.TextToSpeechClient.synthesizeSpeech(TextToSpeechClient.java:268)\n\t\tat com.google.cloud.texttospeech.v1beta1.TextToSpeechClient.synthesizeSpeech(TextToSpeechClient.java:241)\n\t\tat com.gcp.cloudfunctions.CloudTextToSpeech.getSpeech(CloudTextToSpeech.java:97)\n\t\tat com.gcp.cloudfunctions.Test.service(Test.java:53)\n\t\tat com.google.cloud.functions.invoker.NewHttpFunctionExecutor.service(NewHttpFunctionExecutor.java:67)\n\t\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:790)\n\t\tat org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:755)\n\t\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:547)\n\t\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)\n\t\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)\n\t\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)\n\t\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)\n\t\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)\n\t\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)\n\t\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\t\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\t\tat com.google.cloud.functions.invoker.runner.Invoker$NotFoundHandler.handle(Invoker.java:379)\n\t\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\t\tat org.eclipse.jetty.server.Server.handle(Server.java:500)\n\t\tat org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)\n\t\tat org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)\n\t\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)\n\t\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)\n\t\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)\n\t\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)\n\t\tat org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)\n\t\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)\n\t\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)\n\t\t... 1 more\nCaused by: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 19.821560546s. [buffered_nanos=2713815445, buffered_nanos=17108132243, waiting_for_connection]\n\tat io.grpc.Status.asRuntimeException(Status.java:533)\n\t... 12 more",
"insertId": "000000-a16ab144-e2f2-4fe6-ae1a-fc939a327991",
"resource": {
"type": "cloud_function",
"labels": {
"region": "us-central1",
"function_name": "function-1",
"project_id": "nice-theater-281908"
}
},....
Update:
It turns out that the line causing a DEADLINE_EXCEEDED Error is the culprit. Googling the error showed that it may be because the client does not receive a response from the server in time (before timeout). Here is the line causing the error.
SynthesizeSpeechResponse response = textToSpeechClient.synthesizeSpeech(input, voice, audioConfig);
Any advice on how to work around it or how to increase the timeout for the process? Willing to give anything a shot.
SynthesizeSpeech is a synchronous method so you have to wait for the response. Cloud Functions have a default timeout, but you can extend the timeout. You could also try breaking up the text so that it is smaller and thus more likely to finish within the Cloud Function time limit.
I am trying to setup cloudwatch alarm for Lambda execution. I am able to setup ALARM and OK for Errors. But whenever there is any syntax error in my code I get INSUFFICIENT_DATA alarm.
I added my code below:
import json
import sys
print "Buckle your seat belt even if you are in back seat"
def lambda_handler(event, context):
try:
print( "value 1 = " + event['key'])
print( "value 2 = " + event['key2'])
print( "value 3 = " + event['key3'])
return event['key1']
except Exception as e:
print sys.exc_info()[0]
raise
Test Data:
{ "key3": "value3", "key2": "value2", "key1": "value1" }
Here is the error I am generating:
{
"stackTrace": [
[
"/var/task/lambda_function.py",
6,
"lambda_handler",
"print( \"value 1 = \" + event['key'])"
]
],
"errorType": "KeyError",
"errorMessage": "'key'"
}
I can create a metric filter for KeyError and set my alarm. But I want to create one single alarm for all errors whether system level like lambda execution or code level like KeyError etc.
Can anyone please help me how to capture the syntax error or data error in one single alarm of cloudwatch?
Thanks
if you want to catch the error in the logs, then you can use logger feature and do logger.debug(e) or logger.error(e), where e is the exception raised and caught. exception will be available in logs and then you will be able to setup alarm on top of that.