I am using Postman v5.5.3 and trying to run a collection of requests with tests and report the results.
I would like a TIMEOUT to be considered a test failure. This is not happening right now.
I have 2 asserts as:
pm.test("Status code is 200", function () {
pm.response.to.have.status(200);
});
pm.test("Response time is less than 1000ms", function () {
pm.expect(pm.response.responseTime).to.be.below(1000);
});
If I set the Postman XHR timeout to 10000 ms and the Collection Test Runner encounters a timeout, it does not consider this a failed test.
It seems that timeouts short-circuit the test stack and immediately drop out.
If there a way to "test for timeouts" so that timeouts show as test failures in the Collection Test Runner Test Results?
Request timeout is to ensure requests returns back with the response within provided ms.
Since the timeout occurs waiting for the request to respond within that time no response is generated, and so nothing to be asserted on response.
You can achieve this programatically using Newman and its associated 'request' event handler Newman events
Related
I want to run the Postman collection with timeout enabled at the collection level.
Is this setting possible at the collection level in Postman?
I know that the timeout can be set per request using setTimeout(() => {}, 15000), but I could not find something similar to run the collections.
Request timeout for single call
Postman has a setting that you set manually in Settings > General > Request timeout in ms that you can set if you want to set an explicit timeout. However, you won’t be able to do what you’re trying to do from a script.
You can, however, write request tests to make sure your response times
are under a certain threshold. That way when you run the request, the
test will fail if it’s slower than say, 800ms:
pm.test("Response time is less than 800ms", function () {
pm.expect(pm.response.responseTime).to.be.below(800);
});
According to https://community.postman.com/t/request-timeout-for-single-call/6881/2
I'm running tests with a delay set to 2 seconds on the Test Runner on Postman.
I'm just wondering if a response took longer than this, let says 5 seconds, does the next request get sent straight away after this response (because it's gone past the 2 seconds) ?
The delay is between each request. Once the response is received, then the delay starts.
https://learning.postman.com/docs/running-collections/intro-to-collection-runs/#starting-a-collection-run
This isn't the same as the request timeout setting.
https://learning.postman.com/docs/getting-started/settings/#request
I’m searching for a solution for a following issue:
There is a user-flow (folder with a list of requests with a relevant tests).
User-flow running under Newman + htmlextra ci/cd pipeline
User-flow running on test environments which is has 3 party dependencies (services, networking etc)
The issue is the network timeouts, socket hang up.
Which is why i added a kind of retry mechanism in postman
that allows me to repeat the requests once test got status code 5xx
Print screen: User-Flow report
So eventually the user-flow itself pass as you see at the print screen
(the second retry succeeded) but Newman return fail on such user-flow
once we got at least one timeout.
The question is: How i can overcome such issue
Once the retry succeeded i would like Newman will return Success (exit code 0)
Please see the image :
Just imagine : You are getting 5xx series code on "Code Generation" request and you want to retry until you get 2xx series. use the following code for that.
if(responseCode.code!=200){
postman.setNextRequest("Code Generation");
}
pass the request name in setNextRequest
Our API Gateway and Lambdas are regularly used and work just fine most of the time, however we see spikes in 5XX errors now and then which causes a spike in customer complaints and other issues. When I look at the logs during this time I see a flood of the following error:
Execution failed due to configuration error: Malformed Lambda proxy response
There are no other details beyond this. After 10 or 15 minutes it will go away along with customer complaints. I've read that it may happen if you exceed your concurrency limit, but looking at the dashboard and it doesn't look like we ever break above 150 concurrent executions.
The calls themselves being hit work consistently as well, aside from these random spikes in 5XXs.
What else might be causing this inconsistency?
Looking through logs to try and get this figured out. I have made the logs as verbose as possible and there is nothing there. We'll have a normal call with a success response then a few minutes later this error comes up with no other logging, just the error alone. Then a few minutes after that we have logs starting for the next successful call.
10:25:42 Successfully completed execution
10:25:42 Method completed with status: 200
10:42:01 Execution failed due to configuration error: Malformed Lambda
proxy response
12:21:21 Successfully completed execution
12:21:21 Method completed with status: 200
Logging can't go further because the lambdas are never even executed. So we have no details on the payload sent to it, or any internal logging for the call, etc. It just immediately fails at the API Gateway level.
Edit: We still get these spikes but we are working on splitting the lambdas out more. We have an ExpressJS app that handles the lion's share of all requests. So we are breaking more off, especially high traffic requests, into their own lambdas to see if this helps. In-case there is an issue where a container gets too backlogged or times-out because it was handling long running requests (that takes upwards of 20s) as well as being hammered by requests that finish < 500ms.
Other theory is that maybe there is an error that gets triggered somewhere that kills the process or something else and that container is bad until it gets destroyed and respawned. As these spike and then go away in a few minutes. So breaking the lambdas up more should reduce the odds of errors from one cascading and impacting all other requests.
We also increase the resources of the lambda to see if that would help with it handling so many requests.
This usually happens when there is a timeout with your call and if there is a delay with your lambda execution.
If you are accessing an external resource such as RDS or an external network call, wrap that with a promise and handle with a timeout. This way you can identify which resource is having a bottleneck or taking a long time to execute.
exports.handler = function(event, context, callback) {
var response = {}; // set the response object
var err = "An error occured";
setTimeout(function () {
callback(err, response);
}, 3000); // 3000 ms is the timeout
}
// Actual code here
};
Also, check for any missing callbacks. That will also cause this issue.
Hope this helps.
Since Api Gateway time limit is 10 seconds to execute any request I'm trying to deal with timeout errors, but a haven't found a way to catch and respond a custom message.
Context of the problem: I have a function that takes less than 2 seconds to execute, but when the function performs a cold start sometimes it takes more than 10 seconds creating a connection with DynamoDB in Java. I've already optimize my function using threads but I still cannot keep between the 10-seconds limit for the initial call.
I need to find a way to deliver a response model like this:
{
"error": "timeout"
}
To find a solution I created a function in Lambda that intentionally responds something after 10 seconds of execution. Doing the integration with Api Gateway I'm getting this response:
Request: /example/lazy
Status:
Latency: ms
Response Body
{
"logref": "********-****-****-****-1d49e75b73de",
"message": "Timeout waiting for endpoint response"
}
In documentation I found that you can catch this errors using HTTP status regex in Integration Response. But I haven't find a way to do so, and it seems that nobody on the Internet is having my same problem, as I haven't find this specific message in any forum.
I have tried with these regex:
.*"message".*
Timeout.*
.*"status":400.*
.*"status":404.*
.*"status":504.*
.*"status":500.*
Anybody knows witch regex I should use to capture this "message": "Timeout... ?
You are using Test Invoke feature from console which has a timeout limit of 10 seconds. But, the deployed API's timeout is 30 seconds as mentioned here. So, that should be good enough to handle Lambda cold start case. Please deploy and then test using the api link. If that times out because your endpoint takes more than 30 seconds, the response would be:
{"message": "Endpoint request timed out"}
To clarify, you can configure your method response based on the HTTP status code of integration response. But in case of timeout, there is no integration response. So, you cannot use that feature to configure the method response during timeout.
You can improve the cold start time by allocating more memory to your Lambda function. With the default 512MB, I am seeing cold start times of 8-9 seconds for functions written in Java. This improves to 2-3 seconds with 1536MB of memory.
Amazon says that it is the CPU allocation that is really important, but there is not way to directly increase it. CPU allocation increases proportionately to memory.
And if you want close to zero cold start times, keeping the function warm is the way to go, as described here.