On server processing - coldfusion

I have a web application that will be doing some processing with submitted data. Instead of making the user wait for what will take at least a few seconds, maybe up to a few minutes during heavy load, I would like to know if there is some way to, within coldfusion, have processing that just occurs on the server.
Basically, the data would be passed to the server, and then the user would be redirected back to the main page to allow them to do other things, but not necessarily be able to see the results right away. Meanwhile, the processing of the data would take place on the server, and be entered into the database when complete.
Is this even possible within coldfusion, or would I need to look into using code that would receive the data and process it as a separate program?

ColdFusion 8 introduced the cfthread tag which may assist you.
<cfthread
required
name="thread name"
optional
action="run"
priority="NORMAL|HIGH|LOW"
zero or more application-specific attributes>
Thread code
</cfthread>

To do this reliably, you can use a database table as a job queue. So you when the user submits the data you insert record into the database indicating there is some work to be done. Then you create a scheduled task in the CF Administrator that polls a script that gets the next job from the queue and does the processing you describe. When complete it can update the database and you can then alert your user that there job is complete.
Make sense?

Another option that will possibly work for you is to use AJAX to post the data to the server. This is a pretty easy method to use, since you can use pretty much the exact same CF code that you have now and instead only need to modify the form submitting page (and you could even use some unobtrusive javascript techniques to have this degrade gracefully if javascript isn't present).
Here's an example using jQuery and BlockUI that will work for unobtrusively-submitting any form on your page in a background thread:
<script>
$(function () {
$("form").on("submit", function (e) {
var f = $(this);
e.preventDefault();
$.ajax({
method: f.attr("method"),
url: f.attr("action"),
data: f.serialize(),
beforeSend(jqXHR, settings) {
f.blockUI({message: "Loading..."});
},
complete(jqXHR, textStatus) {
f.unblockUI();
},
success: function (data, textStatus, jqXHR) {
// do something useful with the response
},
error: function(jqXHR, textStatus, errorThrown) {
// report the error
}
});
});
});
</script>

You should combine all three of these answers to give yourself a complete solution.
Use CF Thread to "kick off" the work.
Add a record to the DB to tell you the process is underway.
Use Ajax to check the DB record to see if the work is complete. When
your thread completes update the record - Ajax finds the work
complete and you display some message or indicator on the user's
screen so they can go on to step 2 or whatever. So each of these
answers holds a clue to a complete solution.
Not sure if this should be an answer or a comment (since I'm not adding anything new here).

We use an CF event gateway for this. The user submits a file via a web form and the event gateway monitors that upload directory. Based upon the file name the gateway knows how it should process the file into the database. This way the only real delay the user faces is the time for the file to actually transmit from their machine up to the server. We however have no need to inform the user of any statuses related to the process though could easily see how to work that into things if we did.

Related

PWA: how to refresh content every time the app is opened

I created a PWA app which sends API call to my domotic server and prints the response on the home page (e.g. outside temperature and vacuum robot status).
While all the data get refreshed at very first app opening, if I minimize the app whithout completely shutting it off I have no data refreshing at all.
I was wondering how to force a refresh every time the app gets re-opened without having to do it manually (no pull-down to refresh, no refresh-button).
Found myself the solution adding the following code in service worker:
self.addEventListener('visibilitychange', function() {
if (document.visibilityState === 'visible') {
console.log('APP resumed');
window.location.reload();
}
});
Here is the solution that works.
You can place this code wherever you have access to the window object:
window.addEventListener("visibilitychange", function () {
console.log("Visibility changed");
if (document.visibilityState === "visible") {
console.log("APP resumed");
window.location.reload();
}
});
Consider this may affect user experience or data loss with a forced reload every time the user swipes between apps.

How to update progress bar while making a Django Rest api request?

My django rest app accepts request to scrape multiple pages for prices & compare them (which takes time ~5 seconds) then returns a list of the prices from each page as a json object.
I want to update the user with the current operation, for example if I scrape 3 pages I want to update the interface like this :
Searching 1/3
Searching 2/3
Searching 3/3
How can I do this?
I am using Angular 2 for my front end but this shouldn't make a big difference as it's a backend issue.
This isn't the only way, but this is how I do this in Django.
Things you'll need
Asynchronous worker procecess
This allows you to do work outside the context of the request-response cycle. The most common are either django-rq or Celery. I'd recommend django-rq for its simplicity, especially if all you're implementing is a progress indicator.
Caching layer (optional)
While you can use the database for persistence in this case, temporary cache key-value stores make more sense here as the progress information is ephemeral. The Memcached backend is built into Django, however I'd recommend switching to Redis as it's more fully featured, super fast, and since it's behind Django's caching abstraction, does not add complexity. (It's also a requirement for using the django-rq worker processes above)
Implementation
Overview
Basically, we're going to send a request to the server to start the async worker, and poll a different progress-indicator endpoint which gives the current status of that worker's progress until it's finished (or failed).
Server side
Refactor the function you'd like to track the progress of into an async task function (using the #job decorator in the case of django-rq)
The initial POST endpoint should first generate a random unique ID to identify the request (possibly with uuid). Then, pass the POST data along with this unique ID to the async function (in django-rq this would look something like function_name.delay(payload, unique_id)). Since this is an async call, the interpreter does not wait for the task to finish and moves on immediately. Return a HttpResponse with a JSON payload that includes the unique ID.
Back in the async function, we need to set the progress using cache. At the very top of the function, we should add a cache.set(unique_id, 0) to show that there is zero progress so far. Using your own math implementation, as the progress approaches 100% completion, change this value to be closer to 1. If for some reason the operation fails, you can set this to -1.
Create a new endpoint to be polled by the browser to check the progress. This looks for a unique_id query parameter and uses this to look up the progress with cache.get(unique_id). Return a JSON object back with the progress amount.
Client side
After sending the POST request for the action and receiving a response, that response should include the unique_id. Immediately start polling the progress endpoint at a regular interval, setting the unique_id as a query parameter. The interval could be something like 1 second using setInterval(), with logic to prevent sending a new request if there is still a pending request.
When the progress received equals to 1 (or -1 for failures), you know the process is finished and you can stop polling
That's it! It's a bit of work just to get progress indicators, but once you've done it once it's much easier to re-use the pattern in other projects.
Another way to do this which I have not explored is via Webhooks / Channels. In this way, polling is not required, and the server simply sends the messages to the client directly.

Google App Engine - http request/response

I have a Java web app hosted on Google App Engine (GAE). The User clicks on a button and he gets a data table with 100 rows. At the bottom of the page, there is a "Make Web service calls" button. Clicking on that, the application will take one row at a time and make a third party web-service call using the URLConnection class. That part is working fine.
However, since there is a 60 second limit to the HttpRequest/Response cycle, all the 100 transactions don't go through as the timeout happens around row 50 or so.
How do I create a loop and send the Web service calls without the User having to click on the 'Make Webservice calls' more than once?
Is there a way to stop the loop before 60 seconds and then start again without committing the HttpResponse? (I don't want to use asynchronous Google backend).
Also, does GAE support file upload (to get the 100 rows from a file instead of a database)
Thank you.
Adding some code as per the comments:
URL url = new URL(urlString);
HttpURLConnection connection = (HttpURLConnection) url
.openConnection();
connection.setDoOutput(true);
connection.setRequestMethod("POST");
connection.setConnectTimeout(35000);
connection.setRequestProperty("Accept-Language", "en-US,en;q=0.5");
connection.setRequestProperty("Authorization", encodedCredentials);
// Send post request
DataOutputStream wr = new DataOutputStream(
connection.getOutputStream());
wr.writeBytes(submitRequest);
It all depends on what happens with the results of these calls.
If results are not returned to a UI, there is no need to block it. You can use Tasks API to create 100 tasks and return a response to a user. This will take a few seconds at most. The additional benefit is that you can make up to 10 calls in parallel by using tasks.
If results have to be returned to a user, you can still use up to 10 threads to process as many requests in parallel as possible. Hopefully, this will bring your time under 1 minute, but you cannot guarantee it since you depend on responses from third-party resources which maybe unavailable at the moment. You will have to implement your own retry mechanism.
Also note that users are not accustomed to waiting for several minutes for a website to respond. You may consider a different approach when a user is notified after the last request is processed without blocking your client code.
And yes, you can load data from files on App Engine.
Try using asynchronous urlfetch calls:
LinkedList<Future<HttpResponse>> futures;
// Start all the request
for (Url url : urls) {
HttpRequest request = new HttpRequest(url, HTTPMethod.POST);
request.setPayload(...)
futures.add(urlfetchservice.fetchAsync(request);
}
// Collect all the results
for (Future<HttpResponse> future : futures) {
HttpResponse response = future.get()
// Do something with future
}

Redirect to template after celery task complete

I use Celery with Django to put my pdf generation in background, while I display a loading page.
But when the task is complete (i.e. my pdf is generated), I want to redirect to the next view which is responsible to send mail and display a friendly confirmation message to the user.
I know i can get the task_postrun or task_success signal, but I can't redirect from there.
I searched for hours but didn't find any solution, any ideas ?
Thanks !
There are two ways:
Ask the server: save the task_id in the model where you are storing the PDF, and create an ajax view to check every X seconds if task is completed, the result of this view will determine if it should redirect or still wait for the PDF.
result = MyTask.AsyncResult(task_id)
result.get()
Real-time web: another way is using pusher with pusher_client_python, when PDF generation is completed (in your PDF creation rutine), make a api call to pusher who will send a notification to the connected client (that one waiting for the result) and will redirect, this approach is more convenient because you don't have to be asking the server every X seconds. You will need to learn about sockets paradigm, but its very easy to implement.
Hope this helps.

Sustain an http connection while django processes a big request (20mins+)

I've got a django site that is producing a csv download. The content of the csv is dictated by user defined parameters. It's possible that users will set parameters that require significant thinking time on the server. I need a way of sustaining the http connection so the browser doesn't kick up an error message. I heard that it's possible to send intermittent http headers to do this. Can anyone point me in the right direction to set this up on a django site?
(unfortunatly I'm stuck with the possibility of slow reports - improving my sql won't mitigate this)
Don't do it online. Trigger an offline task, use a bit of Javascript to repeatedly call a view that checks if the task has finished, and redirect to the finished file when it's ready.
Instead of blocking the user and it's browser for 20 minutes (which is not a good idea) do the time-consuming task in the background. When the task will finish and generate the result simply notify the user so that he/she will just need to download the ready result.