I'm looking to use a web service that offers a streaming api. This api can typically be used by the java method java.net.URL.openStream();
Problem is I am trying to design my program in C++ and have no idea what libraries (I've heard the cUrl library is very good at this sort of thing) to use, or how to use them to do what I want.
The idea is that after opening the file as a stream I can access continually updating data in realtime.
Any help would be much appreciated.
Boost.Asio socket iostreams seem to be what you're after. Your code will look like this:
ip::tcp::iostream stream("www.someserver.com", "http");
if (!stream)
{
// Can't connect.
}
// Use stream as a regular C++ input stream:
std::string text;
std::getline(stream, text);
If you're new to C++ and have no experience with iostreams then this page is an excellent source of information. In particular, check the docs of the istream class to see what kind of operations your Boost.ASIO stream will support. You'll find that they're not so different from those in the Java IO API.
EDIT: Eric is right, you'll have to send some requests to the server (using the same stream) so it's probably less similar to Java's openStream than I thought. The following example shows how to make those requests:
http://blog.think-async.com/2007_01_01_archive.html
It depends what you're after. Manuel's suggestion of boost::asio::ip::tcp::iostream is good if you want something at a lower level, directly returning the "raw" server response (However, I suspect that something is missing in the example provided in his answer: I think that a "GET" request should be written to the stream before reading from it. See this example from the Asio docs).
I have no experience with java.net.URL.openStream(), but it seems that it is at a little higher level in that only returns the body (and not the headers) of the reply, takes care of HTTP redirects, etc. In that case, yes, libcurl may be more what you want. You could also take a look at the cpp-netlib library, which is built on top of Boost.Asio. It is still in its infancy, but its http::client seems to already provide something pretty similar to what is provided by Java URL.openStream()
Related
I'm looking for a method to send a std::string across a TCP / IP network. It must be platform independent.
The methods I've looked at are:
C sockets - messy.
boost::asio - too complicated
Qt - requires you to inherit QObject and also only single threaded.
What is the simplest, yet strongest method to do this?
Also some example code using the given library for both server side and client side functions, to connect and transfer a std::string would be amazing!
What do you mean by strongest ? Fastest ? Simpliest ?
The server (eg receiver of the string) :
should it be sync or async ?
does it have to answer something or not ?
The anwsers should drive your choice.
If you want something very simple, look for "portable telnet c++ server client", you will get a lot of libs and code samples.
However, to answer your question, here are some portable libs:
Sockets lib, here is a sample
ZeroMQ : may be overkill for a so simple use case, but simple to use anyway
the question is a bit ambiguous :
a) what do you mean by send "std::string" over the network? do you want to completely serialize this object, send it and deserialize it in the other side? or simply send the actual text and build it again as std::string (which is in my opinion simpler and preferred)?
b) it sounds like your question is really "what platforms/libraries there are out there for networking?".
for a : if you decide on sending a text and re-building it as std::string , then you have no problem, just send the text as C-string. you might want to wrap it with some data format as XML or JSON in order to make sure the text was sent correctly. if you DO want to serialize std::string , then there is no clean way of doing so. you might want to reinterpret_cast the object into char array , send it in this way and writing it to some memory. this is not safe or simple.
for b : there are many libraries out there. POCO is well known and quit ok.
JUCE is a simple ,cross platform , modern C++ platform which support object-oriented Socket classes. you might want to use them as well.
So using any free opensource cross platform library like boost how to create a web service capable of reciving a data stream (for example stream of mp3 frames) on one URL like http://adress:port/service1/write/ and capable of sharing latest recived data to all consumers on http://adress:port/service1/read/ so of course mp3 is just an example of packed stream-able data - generally it can be anything packed. How to create such thing?
Generaly I am honesly triing to understend how to do such thing with C++ Network Library but it is just quite unclear to me.
The boost::asio documentation has four examples of complete HTTP server implementations, each with a slightly different threading architecture.
http://www.boost.org/doc/libs/1_43_0/doc/html/boost_asio/examples.html
You do not say what platform to use, but if Windows is an alternative, the Windows HTTP API easy to use and a great performer.
http://msdn.microsoft.com/en-us/library/aa364510(VS.85).aspx
When I say non-standard libraries, I am referring to things like Boost, libCurl and anything else that may be able to do this far easier than standard C++ can. The reason for this is that I am writing an application as a piece of coursework (the class is dedicated to C++) and I am required to use only standard libraries and functions.
I am looking to download a RSS file, using a URL that the user will supply (I'm building a rudimentary RSS client), and the biggest problem I'm facing is that I'm not sure how to get the file down. Once I get past that bit, parsing it for the xml tags and displaying the content will be relatively straightforward. I've been looking around and I've only found solutions that say to use non-standard libraries, usually libCurl. If someome could just give me a quick heads up about what I should be looking at for this, then I'd be grateful.
Also, if you think you're helping me cheat, you're not. The assignment is to build an application of our choice and we're being graded on our use of the various feature of the language (it must contain so many classes, use these variables types, etc).
Check out Beej's Guide to Network Programming for a quick but excellent introduction to sockets. If you cannot use any non-standard libraries, your only option is to manually connect on port 80 and make the request yourself.
Assuming even a beginner-level knowledge of C++, that should be all you need.
First off, it can't be done using only standard C++. There is no network interface in either standard C++ or standard C.
If you're required to take a "do-it-yourself" approach, then probably the intention is that you would use your platform's sockets API. In the case of linux, this is part of the POSIX standard, not C++, and is available from <sys/socket.h>.
The basic procedure is: parse the URL; look up the IP address of the domain; create a socket; connect the socket; write an HTTP request to the socket; read the HTTP response back from the socket; clean up.
Obviously, an HTTP library is far more convenient, especially since an HTTP download can get more complicated than what I describe above (for example, if the server responds with a redirect). Pretty much all linux distributions will provide libcurl, and/or the curl and wget programs.
Writing a program to make a socket connection is relatively trivial.
http://www.linuxhowtos.org/C_C++/socket.htm
Now that you have a socket open to an HTTP server you need to understand how to ask for a document and how to decode the reply:
http://www.w3.org/Protocols/rfc2616/rfc2616-sec5.html
Basically you need to send:
GET<SP><URL><SP>HTTP/1.1<CRLF>
Where:
SP: Single Space
CRLF: \r\n
URL: The Full URL of the page including the server name.
What you get back will be
http://www.w3.org/Protocols/rfc2616/rfc2616-sec6.html#sec6
HTTP/1.1<SP>200<SP>OK<CRLF>
(<Header><CRLF>)*
<CRLF>
<Document>
The above means:
The first line is the response line that should contain 200 OK.
If it does not then there is some kind of error and you should just give up.
This is followed by 0 or more header lines
Just ignore these lines
There will be 1 empty line to mark the end of the headers.
Then the document will be on the stream.
If you really want to do it without using libcurl you can always open a tcp socket and then send:
GET /myurl
(http 1.0 or preferably use http 1.1)
Basically you're writing a very simple http protocol client implementation.
You can download source code for wget standard utility
Since you are not allowed to use non-standard libraries, you could write your own primitive wrapper class for the linux "curl" command (I'm assuming you are using linux). Curl is a very powerful command, and it can probably do what you need it to.
There are a lot of examples how to make HTTP request to a server and get reply via boost.asio library. However, I couldn't find a good example of simple interface and wondering, if I need to implement it myself.
For instance, if I need to get content of http://www.foo.bar/path/to/default.html, is there any way to get a content without validating URL, making HTTP request and parsing server answer?
Basically, I am looking for something like this:
std::string str = boost::asio::get_content("http://www.foo.bar/path/to/default.html");
std::cout << str;
#
<HTML>
<BODY>
Simple HTML page!
</BODY>
</HTML>
There are couple of things that I would like to avoid using boost.asio.
Avoid parsing and validating URL.
Manually creating HTTP request.
Cutting HTTP response from HTML page content.
Since then, there is a newcomer; the C++ Network Library: cpp-netlib as pointed out here.
You wanted to use asio. I suppose you fancied the portability and the ease of use of this lib, so cpp-netlib will be a great choice in that case. It is based on same principles as boost and its authors aim at integrating it into boost.
It is pretty simple to use:
http::client client;
/*<< Creates a request using a URI supplied on the command line. >>*/
http::client::request request("http://www.foo.bar/path/to/default.html");
/*<< Gets a response from the HTTP server. >>*/
http::client::response response = client.get(request);
/*<< Prints the response body to the console. >>*/
std::cout << body(response) << std::endl;
I haven't tried this one but it seems to be possible to do exactly what you need:
cout << body(client().get(client::request("http://www.foo.bar/path/to/default.html")));
This question was asked a long time ago, sorry for digging it out of its grave.
You'll need to implement these functions yourself. Boost.Asio is a socket library primarily, that can be used to implement various protocols. But there's no built-in convenience functions just for some specific protocol like HTTP or SMTP. (Well, actually there's built in DNS resolution, but that's about it.)
However, the Boost.Asio source code comes with pre-made examples of an HTTP client/server, so you can easily start with that.
boost.asio is powerful and sophisticated, but probably overkill for this.
Have you looked at libcurl?
From the person who wrote boost.asio
http://think-async.com/Urdl/doc/html/urdl/getting_started/integrating_with_boost_asio.html
boost::urdl is a library for reading urls into strings easily.
boost.asio doesn't provide such functionality. But I believe there are a number of libraries that do. See POCO libraries for example.
Meanwhile, Boost.Beast made an appearance, which wraps Boost.Asio to provide a simpler interface (not only) for HTTP GET.
I want to be able to download a URL in C++. Something as simple as:
std::string s;
s=download("http://www.example.com/myfile.html");
Ideally, this would include URLs like:
ftp://example.com/myfile.dat
file:///usr/home/myfile.dat
https://example.com/myfile.html
I was using asio in Boost, but it didn't really seem to have the code for handling protocols like ftp and https. Now I discovered QT has more what I need (http://doc.trolltech.com/2.3/network.html).
It's tempting to make the switch to Qt, but it seems a bit heavy and intersects a lot of Boost functionality. Is it worth learning yet another API (Qt) or can Boost do more than I think?
Not a direct answer, but you might like to consider libCURL, which is almost exactly what you describe.
There are sample applications here, and in particular this demonstrates how simple usage can be.
I wouldn't go to Qt just for the networking stuff, since it's really not all that spectacular; there are a lot of missing pieces. I'd only switch if you need the GUI stuff, for which it is top notch.
libCURL is pretty simple to use and more robust than the Qt stuff.
You can use URLDownloadToFile.
#include <Urlmon.h>
HANDLE hr;
hr=URLDownloadToFile(NULL, L"http://www.example.com/myfile.html",L"mylocalfile.html",BINDF_GETNEWESTVERSION,NULL);
According to MSDN, BINDF_GETNEWESTVERSION - is a "Value that indicates that the bind operation retrieves the newest version of the data or object available. In URL monikers, this flag maps to the WinInet flag, INTERNET_FLAG_RELOAD, which forces a download of the requested resource".
The Poco Project has classes for cross-platform HTTP and FTP (and a lot of other stuff). There is overlap with boost. I recently found this, but have not used it.
You can use the URLDownloadToFile or URLOpenBlockingStream, although cURL, libcurl are the proper tools for that kind of jobs.
I got it working without either libcurl nor WinSock: https://stackoverflow.com/a/51959694/1599699
Special thanks to Nick Dandoulakis for suggesting URLOpenBlockingStream! I like it.