I need to execute parallel HTTP requests using Libcurl.
From what I understand I need to create a new handle for each thread
and use CURLOPT_WRITEDATA with some kind of Thread Local Storage.
Does the multi interface makes this task a little easier?
I'm also using cookies, does using CURLOPT_COOKIEFILE and CURLOPT_COOKIEJAR will make
Libcurl load the cookie file for each thread?
As you probably know, libcurl is not thread safe, so you should ensure that the libcurl handle is never shared between multiple threads. Each thread will need to store local data (among other things, the connection handle).
From this, it ensues that for each handle, i.e., for each thread, libcurl will need to read the cookie file from scratch, since this information cannot be shared. This is not a problem, in my opinion, although there could be issues when updating it (you will have multiple thread attempting it).
About the multi interface, it allows you to multiplex multiple transfers, so it is another approach to what you are trying to do but in a single thread.
UPDATE March 2013
libcurl is now thread-safe.
libcurl is free, thread-safe, IPv6 compatible, feature rich, well supported, fast, thoroughly documented and is already used by many known, big and successful companies and numerous applications."
This is not a direct answer, but why do you need multithreading for parallel HTTP requests?
The multi interface is designed for this purpose: you add multiple handles and then process all of them with one call, all in the same thread. From the documentation:
Enable multiple simultaneous transfers in the same thread without
making it complicated for the application.
If you want multiple threads, I suggest you use the easy interface in each thread, and forget about the multi interface.
Sharing simply shares data between easy handles, you can use the interface with/without the multi interface. If you do use multiple threads, you have to provide your own locking.
Also check out libcurl share interface. It was designed for this purpose, i.e. to share data between requests:
You can have multiple easy handles share data between them. Have them
update and use the same cookie database, DNS cache, TLS session cache!
This way, each single transfer will take advantage from data updates
made by the other transfer(s). The sharing interface, however, does
not share active or persistent connections between different easy
handles.
Related
We are having a C++ application to send and receive WebSocket messages
One thread to send the message (using WinHttpWebSocketSend)
the second thread to receive (using WinHttpWebSocketReceive)
But the same WebSocket handle is used across these 2 threads. Will it cause any problems? I don't know if we have to handle it another way. It works in our application - we are able to send and receive messages - but I don't know if it will have any problem in the production environment. Any one has better ideas?
Like most platforms, nearly all Windows API system calls do not provide thread barriers beyond preventing simultaneous access to the key parts of the kernel. While I could not say for sure (the documentation doesn't seem to answer your explicit question) I would be surprised if the WinHTTP API provides barriers that prevent multiple threads from stepping on each other (so to speak)--particularly because it's really just a "helper" API that uses the somewhat lower level Winsock stuff directly--and I would take it upon myself to implement the necessary barriers.
I'm also wondering why you're using threads in this manner to begin with. I know essentially nothing about the WinHTTP API, but I did notice WINHTTP_OPTION_ASSURED_NON_BLOCKING_CALLBACKS which leads me to believe that you can implement an asynchronous approach which would prevent any thread-safety issues to begin with (and probably be much faster and memory efficient).
It appears that the callback mechanism for WinHTTP is rather expressive. See WINHTTP_STATUS_CALLBACK. Presumably, you can simply use non-blocking operation, create an event listener, and associate the connection handle with dwContext. No threads involved.
I have a message object serialized as binary data stream (it can be any std::streambuf), and i want to transfer it to another process. The key is, server application must handle many clients, connection have to be asynchronous (because of multiple clients), and bidirectional (under the hood it may be implemented by two separated connections). Messages have variable length and should be queued. What method of IPC should i pick for this? Is there any simple way to transfer stream buffer through applications? Speed is not critical, but it will be good not to block application for too much time. Anything will be done locally under Windows (i aim to XP and newer), no network support required.
I also need a possibility of listening to incoming connections. Server should automatically detect new connections, do some handshake and accept, if it is compatible. I am aware of that i need to write many of things i mentioned on my own. Anyway, it must be possible to achieve but of course simpler is better.
You can use named pipes in windows.
See MSDN ref: http://msdn.microsoft.com/en-us/library/aa365150%28v=vs.85%29.aspx
You can also set it to be full duplex (bi-directional) and asynchronous. If you are familiar with File I/O APIs on windows then it should be straightforward to use.
I have an application (A) that needs to launch another application (B). I need to pass data between the applications. I can think of two approaches. The first is to open a socket. The second is to share data via a dll.
The opening socket approach is straight forward.
The dll approach I have some questions? I can load plug-in dlls into B. I want to create a dll that A can use to pass data to B. When loading dlls, is only one instance of the dll loaded? If so, does this mean that data can be shared between applications that load the dll?
What is the better choice?
Are there other ways of doing this?
You can't effectively share data via a DLL. Other ways:
disk files
pipes
shared memory
messages
RPC
CORBA
COM
etc.
The simplest method (assuming Windows since you mention a DLL) is probably to use CreateProcess and open a pipe to the child process, as described in simplified form here: http://msdn.microsoft.com/en-us/library/ms682499.aspx
Named Pipes can be an alternative, especially if you aren't in control of the lifetime of all of the processes. http://msdn.microsoft.com/en-us/library/aa365590.aspx
For simple cases, mailslots may be a sufficient alternative.
http://msdn.microsoft.com/en-us/library/aa365574.aspx#base.using_a_mailslot_for_ipc
Here's a longer list of various Interprocess Communication techniques for Windows.
http://msdn.microsoft.com/en-us/library/aa365574.aspx
For something happening locally, using sockets seems sort of overkill. Plus you have to implement your own security mechanism to prevent spoofing attacks, rather than depending on the integrated security mechanism of most of the other IPC methods.
Its always good to explore alternative possible solutions, but I personally believe that using sockets as a transport layer for data between applications is not only future proof, but scalable as well. Using sockets will eliminate the need for you to write copious amounts of OS specific code, which could proclude you from porting your application in the future to non-Windows operating systems.
I would suggest sockets.
You can have a shared cache (example a windows service or hidden process) that can be listening - returning data to all subscribers. This using a Observer pattern approach.
I would agree somewhat with Juan Zamora M except that the service providing the data should have an API that can be requested when needed not pushed when changed via listeners.
This might help. Sharing Files and Memory
Simple question - is the C++ API provided by Perforce thread-safe? There is no mention of it in the documentation.
By "thread-safe" I mean for server requests from the client. Obviously there will be issues if I have multiple threads trying to set client names and such on the same connection.
But given a single connection object, can I have multiple threads fetching changelists, getting status, translating files through a p4 map, etc.?
Late answer, but... From the release notes themselves:
Known Limitations
The Perforce client-server protocol is not designed to support
multiple concurrent queries over the same connection. For this
reason, multi-threaded applications using the C++ API or the
derived APIs (P4API.NET, P4Perl, etc.) should ensure that a
separate connection is used for each thread or that only one
thread may use a shared connection at a time.
It does not look like the client object has thread affinity, so in order to share a connection between threads, one just has to use a mutex to serialize the calls.
If the documentation doesn't mention it, then it is not safe.
Making something thread-safe in any sense is often difficult and may result in a performance penalty because of the addition of locks. It wouldn't make sense to go through the trouble and then not mention it in the documentation.
We're doing a small benchmark of MySQL where we want to see how it performs for our data.
Part of that test is to see how it works when multiple concurrent threads hammers the server with various queries.
The MySQL documentation (5.0) isn't really clear about multi threaded clients. I should point out that I do link against the thread safe library (libmysqlclient_r.so)
I'm using prepared statements and do both read (SELECT) and write (UPDATE, INSERT, DELETE).
Should I open one connection per thread? And if so: how do I even do this.. it seems mysql_real_connect() returns the original DB handle which I got when I called mysql_init())
If not: how do I make sure results and methods such as mysql_affected_rows returns the correct value instead of colliding with other thread's calls (mutex/locks could work, but it feels wrong)
As maintainer of a fairly large C application that makes MySQL calls from multiple threads, I can say I've had no problems with simply making a new connection in each thread. Some caveats that I've come across:
Edit: it seems this bullet only applies to versions < 5.5; see this page for your appropriate version: Like you say you're already doing, link against libmysqlclient_r.
Call mysql_library_init() (once, from main()). Read the docs about use in multithreaded environments to see why it's necessary.
Make a new MYSQL structure using mysql_init() in each thread. This has the side effect of calling mysql_thread_init() for you. mysql_real_connect() as usual inside each thread, with its thread-specific MYSQL struct.
If you're creating/destroying lots of threads, you'll want to use mysql_thread_end() at the end of each thread (and mysql_library_end() at the end of main()). It's good practice anyway.
Basically, don't share MYSQL structs or anything created specific to that struct (i.e. MYSQL_STMTs) and it'll work as you expect.
This seems like less work than making a connection pool to me.
You could create a connection pool. Each thread that needs a connection could request a free one from the pool. If there's no connection available then you either block, or grow the pool by adding a new connection to it.
There's an article here describing the pro's and cons of a connection pool (though it is java based)
Edit: Here's a SO question / answer about connection pools in C
Edit2: Here's a link to a sample Connection Pool for MySQL written in C++. (you should probably ignore the goto statements when you implement your own.)
Seems clear to me from the mySQL Docs that any specific MYSQL structure can be used in a thread without difficulty - using the same MYSQL structure in different threads simultaneously is clearly going to give you extremely unpredictable results as state is stored within the MYSQL connection.
Thus either create a connection per thread or used a pool of connections as suggested above and protect access to that pool (i.e. reserving or releasing a connection) using some kind of Mutex.
MySQL Threaded Clients in C
It states that mysql_real_connect() is not thread safe by default. The client library needs to be compiled for threaded access.