I am writing an application, similar to Seti#Home, that allows users to run processing on their home machine, and then upload the result to the central server.
However, the final result is maybe a 10K binary file. (Processing to achieve this output is several hours.)
What is the simplest reliable automatic method to upload this file to the central server? What do I need to do server-side to prevent blocking? Perhaps having the client send mail is simple and reliable? NB the client program is currently written in Python, if that matters.
Email is not a good solution; you will run into potential ISP blocking and other anti-spam mechanisms.
The easiest way is over HTTP via a simple webservice. Have a listener at your server that accepts the uploaded files as part of a HTTP POST and then dump them wherever they need to be post-processed.
Related
Is it possible to write C++ code to interface with a server, but to be executed on client side, but on the browser instead of native?
Like, for example, imagine using open source classes so that you produce a file.
But because you don't want all this work to be done on the server, you run it on the browser.
So that the client gives a file or two or more as inputs, then the code runs on his machine, the final result is produced, then this file is uploaded to database on the server.
please see google native client project. http://code.google.com/p/nativeclient/
This is strange question.
You can prepare binaries that do task that you want done on client side and make server send proper binary to client when asked for it. Client then runs this binary and returns results to server.
It is possible if you know configurations of client machines (binaries must work on them). Also it have to be some security layer implemented - you don't want to allow every binary run on client (imagine man-in-the-middle attack when some malicious code is run on client).
I think your request contradicts with the idea behind server-side programming. The main purpose in using server-side programs is to make use of infrastructural components like database, network, etc. in a controlled manner. (The most typical usage of server-side applications are web sites with server side coding like JSP and ASP.)
Since servers are machines that are to be kept secure, a remote application should not be permitted to make changes or access filesystem freely. If you want to do changes on a server like doing database operations or reading/writing files, you should use applications that run on the server or provide interfaces like web services or web sites to remote client applications.
So there are a couple solutions when if you want to do work on the browser, then have the results posted in a server database.
First of all, you must set up your server ready for database work. I have done this using the MEAN stack, set up a MongoDB and interfaced it with the Mongoose API.
Now, for the meat of the question, there are many examples of browsers doing intensive work. The majority of these applications thought is not C++, but it is Javascript.
If you really want to focus on C++ (like i did in the past, in the time i asked this question, wanting to make something big for college), then you could do one of the following:
*Use Google Native Client (NaCl). This is a sandbox for running compiled C and C++ code in the browser efficiently and securely, independent of the user’s operating system.
*Maybe you should want to check out Emscripten, which is a framwork for translating C and C++ code to jaascript. This way, you can have your C or C++ binaries that worked, and have them translated to Javascript, in order to have them work in the browser too.
I want to transfer files (upload+download) between clients and server which would be better for me?any recommendation.
one way be would be using http with client sending different parameters in request and server sending or requesting file depending upon algorithm result.
Other way may be, implement WebService on LAMP server and then properly using RPCs to decide upload or download.
I have basic knowledge of Websevices.
Thanks
We have a Django app that needs to post messages and upload files from the web server to another server via an XML API. We need to do X asynchronous file uploads and then make another XML API request when they have finished uploading. I'd also like the files to stream from disk without having to load them completely into memory first. Finally, I need to send the files as application/octet-stream in a POST body (rather than a more typical form data MIME type) and I wasn't able to find a way to do this with urllib2 or httplib.
I ended up integrating Twisted into the app. It seemed perfect for this task, and sure enough I was able to write a beautifully clean implementation with deferreds for each upload. I use my own IBaseProducer to read the data from the file in chunks and send it to the server in a POST request body. Unfortunately then I found out that the Twister reactor cannot be restarted, so I can't just run it and then stop it whenever I want to upload files. Since Twisted is apparently used more for full-blown servers, I'm now wondering whether this was the right choice.
I'm not sure if I should:
a) Configure the WSGI container (currently I'm testing with manage.py) to start a Twisted thread on startup and use blockingCallFromThread to trigger my file uploads.
b) Use Twisted as the WSGI container for the Django app. I'm assuming we'll want to deploy later on Apache and I'm not sure what the implications are if we take this route.
c) Simply can Twisted and use some other approach for the file uploads. Kind of a shame since the Twisted approach with deferreds is elegant and works.
Which of these should we choose, or is there some other alternative?
Why would you want to deploy later on Apache? Twisted is rad. I would do (b) until someone presented specific, compelling reasons not to. Then I would do (a). Fortunately, your application code looks the same either way. blockingCallFromThread works fine whether Twisted is your WSGI container or not - either way, you're just dealing with running code in a separate thread than the reactor is running in.
I have written a simple ftp uploader in C++ with qt using QNetworkAccessManager's put request
its is working as it should but im not happy with the upload speed i get.
i utilize 80% of my upload line while with filezilla i get 100%
The problem is (at least thats what i think) that Qt using only one connection to upload the
data while Filezilla uses multiple connections.
I am considering switching to cURL for the ftp uploads.
Finally my question is: Is curl fast?? does it support multiple ftp upload connectionsfor o single file upload??
To clarify i want to know if while uploading ONE file cURL uses multiple connections to the ftp server to achieve better speads.
if not any suggestions for a library that does that would be wellcome
thank you for your time.
cURL allows multi sessions whereby it can perform multiple transfers concurrently.
Cannot really comment though on whether it is fast. The network interaction is likely to be your bottleneck whichever you use.
Ok so coming in from a completely different field of software development, I have a problem that's a little out of my experience. I'll state it as plainly as possible without giving out confidential details:
I want to make a server that "does stuff" when requested by a client on the same network. The client will most likely be a back-end to a content management system.
The request consists of some parameters, an input file and several output files.
The files are quite large, from 10MB - 100MB of data that must be processed (possibly more). The client can specify destination for output files.
The client needs to be able to find out the status of the request - eg position in queue, percent complete. And obviously when and where to pick up output.
So, my questions are - What is a good method for the client and server to communicate? Should the client poll the server, or provide a "callback" somehow for status updates?
At this point the implementation platform is completely open - anything from C to scripting languages like Ruby are available (at either end), my main issue is how the communication should occur.
First thought, set up some webservices between the machines. But webservices aren't going to be too friendly or efficient with the large files.
Simple appoach:
ServerA hits a web method on ServerB "BeginProcess". The response give you back a FTP location username/password, and ticket number.
ServerA delivers the files to FTP location.
ServerA regularly polls a webmethod "GetProcessStatus(ticketNumber)", possible return values: Awaiting files, Percent complete, Finished
Slightly more complicated approach, without the polling.
ServerA hits a web method on ServerB "BeginProcess(postUrl)", and you send along a URL you want status updates POSTed to. Response: FTP location username/password, and ticket number.
ServerA delivers the files to FTP location.
ServerB sends thru updates to the POST location on ServerA every XXX% completed.
For extra resilience you would keep the GetProcessStatus in case something gets lost in the ether...
Files that will be up to 100MB aren't a good choice for a webservice, since you run a risk of the HTTP session timing out before you have completed your processing.
Having a webservice for checking the status of these jobs would be more ideal. Handle the file transfers via FTP or whatever file transfer method you choose and poll a webservice for updates on status. When the process is completed, you might have an output file url returned that can be downloaded.