I wrote a simple HTTP responding C++ program using http://www.fastcgi.com library + nginx web-server. It works almost great. But I noticed that fcgi's function
int FCGX_Accept_r(FCGX_Request *request);
doesnt return before all the request transmitted to the server. The problem: i am unable to analyze parameters from URI before file transmission is finished: i want to analyze "?sid=" argument to reject transmission from invalid user. It would be a waste of bandwidth to allow anyone send 1GB files to a server.
As i know, web-server always send FCGI_BEGIN_REQUEST via FastCGI connection when new request started, then FCGI_PARAMS etc. So it would be good to react to content of some parameters before user transmitted all the (possibly unneeded and too large) data.
Also i will read more nginx documentation about its FastCGI abilities.
update:
Trying to send 2 GB file to nginx via HTML-form + sniffing the exchange between nginx and fastcgi: nginx doesn't even try to send a byte to FastCGI application before all the 2GB has come. So, this behavior is kinda normal. And if you want the functionality i am asking for, it turns out that you should develop your own nginx's module or simple web-server. Hm...
I think you can control by this :
location /xxx/upload {
access_by_lua '
//check cookie or something can recognize the req is valid or not
';
fcgi_pass XXXX;
}
check this link :
http://openresty.org/
Related
Problem:
OS: Ubuntu 20.04.1 LTS
When a target URL updates its content, recently libcurl has had unexpected polling delays / timeouts anywhere between 2 and 20+ seconds between sending a GET request to the target URL and receiving any response.
I have no idea what has been causing this behaviour, and have detailed all of the strace reports, tshark results, entire libcurl C++ program, attempts to diagnose, and other terminal outputs at the following SO question, but have had no luck in diagnosing this for about four months:
libcurl: abnormal GET response delays
There seems to be something between the client server and remote server that is stopping packets from being returned, but only when the page changes its content. During this polling delay / timeout, no other requests can be sent - therefore any new data uploaded on the remote server cannot be retrieved quickly.
This issue did not exist before mid-July 2021. Given that after four months this problem still hasn't been solved, I want to attempt a workaround that will still send requests to the target when this polling delay presents itself. I won't understand what caused the polling timeouts, but hopefully I will be able to retrieve the data without delays like the program used to do.
Target URL: https://ir.eia.gov/wpsr/table4.csv
Summary questions:
Q1. Is there a timeout option with libcurl that, when exceeded, the program does not exit but instead sends another GET request to the target URL?
Q2. Since this problem only arises when the target URL makes a scheduled content update, could there be a chance the target URL changes its IP address and thus there is some delay caused by a DNS resolution server in between the client and remote side on the return leg? I am going to attempt to use a tool like pingPlotter to see if there is a delay at some specific IP address between the outbound GET request and the response.
Before any scheduled page content changes, the latency between the outbound GET request and the response is <100 ms.
I have a c++ project (server-side) which sends the data to the client side with the help of event source. Now for debugging purposes I have maintained a trace file (text). On all the critical areas in the c++ code where there is a fear of code-break, I have added a line which writes the "success" text to this trace file. It works fine. I can know where the code is success and where it failed.
But I am on the server side. I want to avail this facility to the client too. However, I am unsure about how to do that? Should I stream the file on web-browser, or is there any other way I can send the data "live" to web-browser?
I checked this link, however, I am not sure if I can use this. http://www.tutorialspoint.com/cplusplus/cpp_web_programming.htm
Thanks
your question is a bit confusing, and without any sample of your code it is a bit unclear of what you want to do. however, the best suggestion i can give is to do this:
Store the text document on a server of your choice.
write a program to contact the server and download the data. (Using Winsock.h)
OR
Directly send the file to the computer. you'll have to write a program to contact the server at which point B is located.
for information on writing an application using Winsock.h, check here:
http://msdn.microsoft.com/en-us/library/windows/desktop/ms737629(v=vs.85).aspx
if you really must store the data live in a web browser, then take a look at Java or VisualBasic, as both are heavily supported in all web browsers, while C++ is not.
Let me see if I understand you correctly. You want to send success / failure of the webserver to your client program?
Well, that's part of the job of a webserver, as dictated by the http protocol. A webserver will respond to a client request with a response header, followed by the requested page (if it exists as a valid resource on the server).
For more information on http response headers have a look at this article, or this wikipedia page, which both detail the request / response conversation between browser and webserver.
Hope this helps.
this might be a dumb question...
I wrote a C++ client program that communicates with a web service over HTTPS with the help of the cURL library.
I am wondering if the person using the client can see clearly the traffic originating from his computer using some sniffing program?
Or would he see encrypted data?
Thanks!
Using a utility like netcat to sniff data on the wire, the user would only see encrypted data. The only way to see the raw data is to log it inside the app, before it's passed to cURL, OR to find it in the machine's active RAM (much more difficult since it's likely to be fragmented).
Not if your app checks for valid certificates.
If your users have the ability to use a proxy server with your app, they could use fiddler's decrypt https sessions function to do this, but it results in an invalid certificate which could be made to stop it from working when detected.
He would see the encrypted data. Sniffers only see the packets, so if HTTPS is working as it should, the packets should be encrypted, and that's all the program could see.
If you would like to try it yourself, learn about ettercap-ng.
I doubt that an average user would be able to do that...
BUT there are ways to do this like:
replacing the cURL library with a proxy (if you link dynamically)
running your program under a debugger and placing breakpoints on the cURL functions
replacing the cURL program with a proxy (if you use it as a commandline utility)
digging deep and diessecting the memory at runtime
From my POV it is improbable (since you need some skill + knowledge + some control over the client environment to pull that off) but possible...
The SSL/TLS protocol is typically implemented at the application layer, so the data is encrypted before it is sent.
If the user has access to the certificate key(s) used to encrypt/decrypt the data, then he/she can plug them into WireShark and it can then decode sniffed HTTPS packets off the wire.
So I've been playing around with some simple HTML forwarding with c++. Haven't accomplished much and I have some questions on the backbone.
First: Do I need to use any special libraries other than socket libraries to simply forward HTML data and connections?
Second: When a client connects to an HTML server, is the TCP connection kept open? Or is it closed once data is sent?
Third: When I forward data, from a client to the server, the packet includes the destination address. I should technically be able to read this address and connect to the server via port 80, keep it open, and send and receive on that newly opened port right? Is there anything I have to do? Any time constraints? If I directly forward every single packet directly between the client and server the website should show up correctly on the client, correct?
I would prefer to keep any external libs to a minimum. But if necessary I can expand the program to include any required libraries.
So far I've gotten data to and from both parties, however the website does not function.
[platform] :: windows.primary && posix_compliant.secondary
First: No you do not need other special libraries but not using any that are available would to some extent be reinventing the wheel.
Second: No, HTTP is a connectionless protocol.
Third: An HTTP session begins with a request header, which in your case sounds like a POST. A POST may take more than one package, during which time the connection remains open. The server may well time you out.
You might look at libCURL even if you do not intend using it. (The source for that is in C, and is rather monolithic but it is commonly used).
After doing quite a bit of research, the greatest help I've had in my endeavors has been this website.
This one also helped quite a bit.
LibCURL is certainly the way to go. It's kind of dated, and everything is in C, but it's much easier than redoing everything..
quote from second site:
Like most network protocols, HTTP uses the client-server model: An HTTP client opens a connection and sends a request message to an HTTP server; the server then returns a response message, usually containing the resource that was requested. After delivering the response, the server closes the connection (making HTTP a stateless protocol, i.e. not maintaining any connection information between transactions).
Ok so coming in from a completely different field of software development, I have a problem that's a little out of my experience. I'll state it as plainly as possible without giving out confidential details:
I want to make a server that "does stuff" when requested by a client on the same network. The client will most likely be a back-end to a content management system.
The request consists of some parameters, an input file and several output files.
The files are quite large, from 10MB - 100MB of data that must be processed (possibly more). The client can specify destination for output files.
The client needs to be able to find out the status of the request - eg position in queue, percent complete. And obviously when and where to pick up output.
So, my questions are - What is a good method for the client and server to communicate? Should the client poll the server, or provide a "callback" somehow for status updates?
At this point the implementation platform is completely open - anything from C to scripting languages like Ruby are available (at either end), my main issue is how the communication should occur.
First thought, set up some webservices between the machines. But webservices aren't going to be too friendly or efficient with the large files.
Simple appoach:
ServerA hits a web method on ServerB "BeginProcess". The response give you back a FTP location username/password, and ticket number.
ServerA delivers the files to FTP location.
ServerA regularly polls a webmethod "GetProcessStatus(ticketNumber)", possible return values: Awaiting files, Percent complete, Finished
Slightly more complicated approach, without the polling.
ServerA hits a web method on ServerB "BeginProcess(postUrl)", and you send along a URL you want status updates POSTed to. Response: FTP location username/password, and ticket number.
ServerA delivers the files to FTP location.
ServerB sends thru updates to the POST location on ServerA every XXX% completed.
For extra resilience you would keep the GetProcessStatus in case something gets lost in the ether...
Files that will be up to 100MB aren't a good choice for a webservice, since you run a risk of the HTTP session timing out before you have completed your processing.
Having a webservice for checking the status of these jobs would be more ideal. Handle the file transfers via FTP or whatever file transfer method you choose and poll a webservice for updates on status. When the process is completed, you might have an output file url returned that can be downloaded.