What I need to do is download a page, find a number in the page and multiply it by 4; then send that new number back to the page using POST but with the same connection.
I was doing this completely fine with curl and C++ but two connections. Is there a way to do this with only one connection in curl?
I tried reading the curl --help but I didn't know what to look for and didn't see how to do it.
Related
I am trying to download data from one url which has multiple pages.
ex) https://stackoverflow.com/questions/tagged/c%2b%2b?sort=newest&page=2&pagesize=15
I would like to download all the data from different pages
ex) pages=1, pages=2, pages=3 etc..
If I create a curl handle with one url with certain page,
will it be okay to replace urls (only page number) without worrying about performance?
I've created curl handle every time I move to next page and download the data.
I got an advice from other people that it would be better to create a ssl session and persist its connection, so that I don't have to go through all the SSL handshake whenever I create SSL connections.
Accordint to Libcurl page, it is recommending to use same handle for performance, but I am not sure I could do it with different URLs.
You can and should re-use the same curl instance whenever possible. Just keep on repeating curl_easy_setopt(CURLOPT_URL, ...); curl_easy_perform(...); calls.
Additional notes:
You can download multiple URLs in parallel if you use the curl_multi_init interface (example).
StackExchange has a very good REST API. There is no need to scrape it.
I've to send too many request to a webservice. I can't send all the data in a big single request because of server limitations. I used to use cURL to do a POST request, but in this case cURL will set up and close a different connection for each sending.
Is there any way that cURL reuse a connection for multiple requests? Is there any alternative, e.g. wget, that could achieve this purpose?
I'd rather be able to avoid use a Perl homemade solution with XML::Compile::WSDL11.
Enable HTTP/1.1 Keep-Alive. See lwptut and LWP::ConnCache.
I'm using SOCKET (WINSOCK2.H) to connect to IP camera.
After connect, I want to send the command "GET /nphMotionJpeg?Resolution=640x480&Quality=Standard HTTP/1.1\r\n\r\n" to get video stream.
But this command is not successful because the camera is protected by authentication basic.
Now, I want to insert into request command username and password but i don't know the syntax to do it.
the syntax is explained in the related wikipedia article: https://en.wikipedia.org/wiki/Basic_access_authentication
I made an application using Qt/C++ that reads some values every 5-7 seconds and sends them to a website.
My approach is very simple. I am just reading the values i want to send and then i make an HTTP POST to the website. I also send the username and password to the website.
The problem is that i cannot find out if the request is successful. I mean that if i send the request and server gets it, i will get an HTTP:200 always. For example if the password is not correct, there is no way to know it. It is the way HTTP works.
Now i think i will need some kind of a protocol to take care the communication between the application and the website.
The question is what protocol to use?
If the action performed completes before the response header is sent you have the option of adding a custom status to it. If your website is built on PHP you can call header() to add the custom status of the operation.
header('XAppRequest-Status: complete');
if you can modify the server side script you could do the following
on one end :
You can make the HTTP post request via ajax
and evaluate the result of the ajax request.
On the serve side
On the HTTP request you do your process and if everything goes accordingly you can send data back to the ajax script that called it.
solves your problem .. ?
I want to make a server written in C++ to power my game. I learned the basics of sockets and wrote a basic chat program that worked well. Now I want to create an HTTP server like Apache, but only for the AJAX request-response part.
I think just for the beginning i copied one Apache response text, and i sent the exact response with the C++ server program.
The problem that is that the browser (Firefox) connnects to the apache and everything works fine, except all of the requests get a correct response.
But if i send this with the C++ client, then FireBug tells me that the response status is OK (200) but there is no actual response text. (How is this possible?)
This response-text is exactly the same what apache sends. I made a bit-bit comparison and they were the same.
The php file wich is the original response
<?php echo "AS";echo rand(0,9); ?>
And the origional source code:
Socket.h http://pastebin.com/bW9qxtrR
Socket.cpp http://pastebin.com/S3c8RFM7
main.cpp http://pastebin.com/ckExuXsR
index.html http://pastebin.com/mcfEEqPP < this is the requester file.
ajax.js http://pastebin.com/uXJe9hVC
benchmark.js http://pastebin.com/djSYtKg9
jQuery is not needed.
The main.cpp there is lot of trash code like main3 and main4 functions, these do not affect the result.
I know that the response stuff in the C++ code is not really good because the connection closing is not the best; I will fix that later now I want to send a success response first.
the problem:
the index.html is served through apache on port 80. the browser loads it and starting sending requests.
The request file (program) was on another port , on the 8888 port, which already is a different server which dont enables ajax (dont know why) to get the post data. the program can still communicate with remote servers but cant see the response.
after one whole day i tested a lot with the fiddler program , captured the responses, and that method helped me.
I used the fiddler program to capture the the good answer and to capture the bad. They were the same. After this i turned off my socket application, and forced fiddler to auto respond, and the answer from the 'bad' answer still bat. So after that i replaced the bad with the good and nothing happedned. The bad answer with the good text still bad on the :8888 port but the other on the original :80 port was good, but they were absolutly the same and the same program sended it (fiddler) i think there is something missing if the response is not on the same server address (even not the same port).
after this i thought maybe there is a missing header file, or something ike this.
So i configurated apache to listen on the 80 port, loaded in the index.html. after this i shut down the apache server and changed the port to 8888 and i run the ajax requests and i recognized that they are wrong, but they were sent by the apache, and all of the previorus requests (on the same port) were good. so the problem is only with the ajax stuff :D
many thanks to Tony Lee for the Fiddler suggestion.
Actually there is no solution but there is an answer why the problem exists.
I don't know how you're verifying bit-by-bit - if you used fiddler to capture the traffic then this is a mystery.
I'm going to guess the unsent buffered data is lost when you close the socket. See the MSDN article Graceful Shutdown, Linger Options, and Socket Closure. Call shutdown() before you call closesocket() to ensure a clean shutdown.
Not really an answer to your question, but you might find it useful.
Instead of Apache code you can try libevent. It has functions just to make http servers and it probably will be much faster than Apache code.
Check this link. There is some info about building http server with libevent.