Request is not being served and getting timed out.
Observation is request is reaching to worker thread - 16:19:23.051 [armeria-common-worker-nio-2-2] DEBUG c.l.a.server.logging.LoggingService . But it is not reaching to Fork join thread ForkJoinPool.commonPool-worker-0]. This is happening for random requests.
Armeria version is - 1.14.0.
Related
I have an app on Google AppEngine (Python39 standard env) running on gUnicorn and Flask. I'm making a request to the server from client-side app for a long-running operation and seeing that the request processed twice. The second process (worker) started after a while (a hour and a half) after the first one has been working.
I'm not sure is it related to gUnicorn specifically or to GAE.
The server controller has logging at the beginning :
#app.route("/api/campaign/generate", methods=["GET"])
def campaign_generate():
logging.info('Entering campaign_generate');
# some very long processing here
The controller is called by clicking a button from the UI app. I checked the network in DevTools in the browser that only one request fired. And I can see that there's only one request in server logs at the moment of executing of workers (more on this follow).
The whole app.yaml is like this:
runtime: python39
default_expiration: 0
instance_class: B2
basic_scaling:
max_instances: 1
entrypoint: gunicorn -b :$PORT server.server:app --timeout 0 --workers 2
So I have 2 workers with infinite timeouts, basic scaling with max instances = 1.
I expect while the app is processing one request for a long-running operation, another worker is available for serving.
I don't expect the second worker will used to processing the same request, it's a nonsense (if only the user won't start another operation from another browser).
Thanks to timeout=0 I expect gUnicorn will wait indefinitely till the controller finishes. And only one thing that can hinder is GAE'e timeout. But thanks to basic-scaling it's 24 hours. So I expect the app should process requests for several hours without problem.
But what I'm seeing instead is that after the processing the request for a while another execution is started. Here's simplified logs I see in Cloud Logging:
13:00:58 GET /api/campaign/generate
13:00:59 Entering campaign_generate
..skipped
13:39:13 Starting generating zip-archive (it's something that takes a while)
14:25:49 Entering campaign_generate
So, at 14:25, 1:25 after the current request came another processing of the same request started!
And now there're two request processings running in parallel.
Needless to say that this increase memory pressure and doubles execution time.
When the first "worker" finished (14:29:28 in our example) its processing, its result isn't being returned to the client. It looks like gUnicorn or GAE simply abandoned the first request. And the client has to wait till the second worker finishes processing.
Why is it happening?
And how can I fix it?
Regarding http requests records in the log.
I did see only one request in Cloud Logging (the first one) when the processing was active, and even after the controller was called for the second time ('Entering campaign_generate' in logs appeared) there was not any new GET-request in the logs. But after that everything completed (actually the second processing returned a response) a mysterious second GET-request appeared. So technically after everything is done, from the server logs' view (Cloud Logging) it looks like there were two subsequent requests from the client. But there weren't! There was only one, and I can see it in the browser's DevTools.
Those two requests have different traceId and requestId http headers.
It's very hard to understand what's going on, I tried running the app locally (on the same data) but it works as intended.
I have a request that lasts more than 3 minutes, I want the request to be sent and immediately give the answer 200 and after the end of the work - give the result
The workflow you've described is called asynchronous task execution.
The main idea is to remove time or resource consuming parts of work from the code that handles HTTP requests and deligate it to some kind of worker. The worker might be a diffrent thread or process or even a separate service that runs on a different server.
This makes your application more responsive, as the users gets the HTTP response much quicker. Also, with this approach you can display such UI-friendly things as progress bars and status marks for the task, create retrial policies if task failes etc.
Example workflow:
user makes HTTP request initiating the task
the server creates the task, adds it to the queue and returns the HTTP response with task_id immediately
the front-end code starts ajax polling to get the results of the task passing task_id
the server handles polling HTTP requests and gets status information for this task_id. It returns the info (whether results or "still waiting") with the HTTP response
the front-end displays spinner if server returns "still waiting" or the results if they are ready
The most popular way to do this in Django is using the celery disctributed task queue.
Suppose a request comes, you will have to verify it. Then send response and use a mechanism to complete the request in the background. You will have to be clear that the request can be completed. You can use pipelining, where you put every task into pipeline, Django-Celery is an option but don't use it unless required. Find easy way to resolve the issue
I am using spring boot for Rest API services.
We see lots of idle timeout problems when reading data. It reported "java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms " Below is what I configured for the jetty thread pool. Anyone knows why it was failed with timeout 30s not 60s?
int threadPoolIdleTimeout = 60000;
ThreadPool threadpool = new QueuedThreadPool(maxThreads, maxThreads, threadPoolIdleTimeout,
new ArrayBlockingQueue(threadPoolQueueSize));
Unrelated.
That's the thread idle timeout, for reducing the number of idle threads in the thread pool.
The connection idle timeout is a different configuration.
Check the ServerConnector if a normal server connection.
Check the AsyncContext idle timeout if you are using Servlet Async Processing, or Servlet Async I/O.
Check the WebSocket Session if you are doing WebSocket requests.
Check the database DataSource configuration if you are worried about database connection idle timeouts.
Check the HTTP2 Session configuration for dealing with the virtual connections on an HTTP/2 connector.
And many more, etc ...
There's lots of idle timeouts, specific for the situation you are dealing with, be aware of them.
Using WSO2 BPS 3.6.0 we encountered a serious issue
We have a few processes waiting for external events (with timeout) and several processes polling for updates (using wait node).
The problem arises as soon we restart a server:
* timeouts which are passed during the downtime are not processed
* wait nodes are not processed at all
Reading related articels:
https://issues.jboss.org/browse/RIFTSAW-466
wso2 wait loop doesn't work after restart
I found that the timeout timestamps are stored in the ode_job table. So I tried to update the timeout timestamps (before starting up the BPS server)
update ode_job set ts=(near_future_timestamp) where ts>(before_restart) and ts<(near_future_timestamp)
which resolved the scope timeouts, however the wait nodes are not processed anymore even they were stated in future. That effectively blocks all the polling instances without any means to move them further.
Is there a way to "revive" or timeout the wait nodes after restarting the server?
I have a long running task which is NOT async, it will block the response of django, the server stack is nginx+uwsgi, what happens after nginx decide it's timeout, will my task (the uwsgi worker and the django view thread) be killed?
normally the request will continue in uWSGI until a "bad condition" (like sending a chunk of the response to a disconnected client) will fail. It is a configurable behaviour, but i suggest you to not touch it if you have no VERY specific reasons