Do any asynchronous connectors exist for Mysql that can be used within a C or C++ application? I'm looking for something that can be plugged into a reactor pattern written in Boost.Asio.
[Edit:] Running a synchronous connector in threads is not an option.
http://forums.mysql.com/read.php?45,183339,183339
enjoy
Updated link to the original article showing how to do async mysql queries:
http://jan.kneschke.de/projects/mysql/async-mysql-queries-with-c-api/
I had a similar problem with a very different technologies: Twisted python (reactor-based IO) and sqlAlchemy (??). While searching for a solution, I found about an sAsync project that simply created a separate thread for sqlAlchemy and then responded to requests.
Given that ASIO is based on low level OS features (such as aio_read() or ReadFileEx() etc) and an OS-level reactor (or proactor, in Windows' case) I don't think you have another chance than emulating the 'asynchronousness' by similar means.
Running a synchronous connector in threads is not an option
Think about it: the libmysqlclient / mysqlclient.dll you're using makes synchronous socket calls. The OS scheduler will correctly switch to another thread until the I/O is finished, so what's the difference? (apart from the fact that you shouldn't make 2k threads for this..)
Edit: mysql_real_connect() supports an UNIX socket parameter. You can supposedly read yourself from the mysql server port and write to that UNIX socket only using ASIO. Like a proxyfication.
I know this is an old question, but consider looking at the new Boost.Mysql library: https://anarthal.github.io/mysql/index.html
[ Running a synchronous connector in threads is not an option
Think about it: the libmysqlclient / mysqlclient.dll you're using makes synchronous socket calls. The OS scheduler will correctly switch to another thread until the I/O is finished]
This is bugging me! - the 'another thread' could as easily be a second sync. connection to mysql, and should be handled by mysql just as it would another client altogether? My gutfeel is that it should work using multiple threads.
MySQL Connector/C++ is a C++ implementation of JDBC 4.0
The reference customers who use MySQL Connector/C++ are:
- OpenOffice - MySQL Workbench
Learn more: http://forums.mysql.com/read.php?167,221298
There is a project called DBSlayer that puts another layer in front of MySQL that you talk to through JSON. http://code.nytimes.com/projects/dbslayer
have you considered using libdrizzle? i have used only an old version, from when it was a separate project from drizzle, and i tested the asynchronous query features, but i never did any actual benchmarks worth mentioning.
I think the only solution will be to create an asynchronous service that wraps a standard connector. You'll need to understand the ODBC APIs though.
Related
I find creating a thread for each client connection costly, also there is a limit to the number of threads I can create, and so a limit to the number of connections. What other options do I have? I have read that select() can be used but I need to do some tweaking to allow for a large number of connections.
Note that I do not want to use an external library.
have a look at the asio library - available standalone or as part of boost
Here is a link to the example page - you will see a single-threaded http server. extremely simple, very lightweight, no linking to other libraries necessary.
http://think-async.com/Asio/asio-1.11.0/doc/asio/examples/cpp11_examples.html#asio.examples.cpp11_examples.http_server
Use I\O Completion Ports these allow you to scale to 100's of thousands of connections using a small number of threads.
If cross platform is NOT your goal then it's best to write directly to the Windows IOCP API rather than using any of the cross platform libraries.
Note that I have some articles and some code you can download here explains how to use IOCP.
How does ZeroC ICE compare to 0MQ? I know that 0MQ/Crossroads and DDS are very similar, but cant seem to figure out where ICE comes in.
I need to quickly implement a system that offloads real-time market-data from C++ to C#, as a first phase of my project. The next phase will be to implement an Event Based architecture with an underlying Pub/Sub design.
I am willing to use TCP.. but the the system is currently running on a single 24 core server.. so an IPC option would be nice. From what I understand ICE is only TCP, while DDS and 0mq have an IPC option.
Currently ,I am leaning towards using Protobuf with either ICE or Crossroads IO. Got turned off from the OpenSplice DDS website. Ive done lots research on the various options, was originally considering OpenMPI + boost:mpi, but there does not seem to be MPI for .NET.
My question is:
How does ICE compare to 0MQ? I cant wrap my head around this. Was unable to find anything online that compares the two.
thanks in advance.
........
More about my project:
Currently using CMAKE C++ on Windows, but the plan is to move to CentOS at some point. An additional desired feature is to store the tic data and all the messages in a "NoSql" database such as Hbase/Hadoop or HDF5. Do any of these middleware/messaging/pub-sub libraries have any database integration?
Some thoughts about ZeroC:
Very fast; Able to have multiple endpoints; Able to load balance on the endpoints; Able to reconnect to a different endpoint in case one of the node goes down. This is transparent to the end user; Has good tool chain (IceGrid, IceStorm, IceBox, etc); Distributed, high availability, multiple failover, etc
Apart from that, I have used it for hot swapping code modules (something similar to Erlang) by having the client create the proxy with multiple endpoints, and later on bring down each endpoint for a quick upgrade one by one. With the transparent retry to a different endpoint, I could have the system up and running the whole time i did an upgrade. Not sure if this is an advertised feature or an unadvertised side-effect :)
Overall, it is very easy to scale out your servers if need be using ZeroC Ice.
I know ZeroMQ provides a fantastic set of tools and messaging patterns and I would keep using it for my pet projects. However, The problem that i see is that it is very easy to go overboard and lose track of all your distributed components. This is a must have in a distributed environment. How will you know where your clients/server are when you need to upgrade? If one of components down the chain does not receive a message, how to identify where the issue is? the publisher? the client? or any one of the bridges (REP/REQ, XREP/XREQ, etc) in between?
Overall, ZeroC provides a much better toolset and ecosystem for enterprise solutions.
And it is open source :)
Jaybny,
ZMQ:
If you want real good performance and the only job for Phase 1 of your job is to move data from C++ to C#, then Zmq is the best option.
Having a pub/sub model for event driven architecture is also something that Zmq can help you with, with its in-built messaging pattern.
Zmq also supports your IPC requirements in this case. Eg: you can have one instance of your application that consumes 24 cores by multithreading and communicating via IPC.
ZeroC Ice:
Ice is a RPC framework very much like CORBA.
Eg.
Socket/ZMQ - You send message over the wire. Read it at the other end, parse the message, do some action, etc.
ZeroC Ice - Create a contract between client and server. Contract is nothing but a template of a class. Now the client calls a proxy method of that class, and the server implements/actions it and returns the value. Thus, int result = mathClass.Add(10,20) is what the client calls. The method, parameters, etc is marshalled and sent to the server, server implements the Add method, returns the result, and the client gets 30 as the result. Thus on the client side, the api is nothing but a proxy for a servant running on a remote host.
Conclusion:
ZeroC ICE has some nice enterprisy features which are really good. However, for your project requirements, ZMQ is the right tool.
Hope this helps.
For me.. the correct answer was Crossroads I/O . It does everything I need.. but still unable to pub/sub when using protobufs... im sure ZeroC ICE is great for distributed IPC, but 0MQ/Crossroads, gives you the added flexibility to use Inter-Thread-Communication.
Note: on windows, 0mq does not have IPC.
So, all in all, the crossroads fork of 0mq is the best. but you will have to roll your own windows/ipc (or use tcp::127..) , and publisher side topic filtering features for pub/sub.
nanomsg, from the guy who wrote crossroads and 0mq (i think).
http://nanomsg.org/
I have seen lately a couple of programs that can be launched as daemons (e.g. linphonecsh) but also offer a second invocation method that will exchange information with the running daemon. In the linpohone case, linephonecsh with one set of parms launches the daemon but if invoked with a different set of parms it can query the status of the daemon (call in progress, call duration, hangup, exit, etc.).
So, since I need to write an app that could go either way, app or daemon, I was wonering about how one does this neat trick. I suppose UNIX domain sockets would work as might named interprocess pipes. D-bus perhaps?
And where might one see a good C/C++ example of this?
Any suggestion and alternate approaches are solicited.
You have a few options:
Shared memory
Pipes
UNIX domain sockets.
You should decide which one suits you best, based on the details of your task. I assume you're on Linux, so a chapter from the book "Advanced Linux Programming" on inter-process communication will help. It provides code examples, too.
This seems to be typical application:
1. One part of the program should scan for audio files in background and write tags to the database.
2. The other part makes search queries and shows results.
The application should be crossplatform.
So, the main search loop, including adding data to database is not a problem. The questions are:
1. What is the best way to implement this background working service? Boost(asio) or Qt(services framework?)?
2. What is the best approach, to make a native service wrapper using mentioned libraries or emulate it using non gui application?
3. Should I connect gui to the service(how they will communicate using boost or qt?) or directly to the database (could locks be there?)?
4. Will decsision in point 1 consume all CPU usage? And how to avoid that? How to implement scanning for files less cpu consumable?S
I like to use Poco which has a convenient ServerApplication class, which can be used in an application that can be run as either a normal command-line application, or as a Windows service, or as a *nix daemon without having to touch the code.
If you use a "real" database (MySQL, PostgreSQL, SQL Server), then querying the database from the GUI application is probably fine and easier to do. If you use another type of database that isn't necessarily multi-user friendly, then you should communicate with the service using loopback sockets or pipes.
As far as CPU usage, you could just use a bunch of "sleep" calls within your code that searches files to make sure it doesn't hog the CPU and IO ports. Or use some kind of interval notification to allow it to search in chunks periodically.
googling about asynchronous /non-blocking connectors for mysql i went basically to this post
However, it's been 2 years and following whats happening on drizzle is a bit confusing at the moment. libdrizzle was a separate dependency at some point but they decided to merge it with the rest of the project. Are there other options for asynchronous database access from c++?
I've been looking at OTL, ODB and OpenDBX, but they all seem to be synchronous (require a separate thread for non-blocking operation)
I had the same desire and came to the conclusion that it's not supported. Even with the MySQL C API you can use the low-level functions to issue queries and wait for a response asynchronously, but you cannot ever get full asynchronous result collection--you always end up blocking from the time the first piece of the result is returned until the last.
I don't have direct experience with it, but I've read that Postgres does support full asynchrony (at least in the C API).
I used to used MySAC in my own project. It works well though is a little outdated. I just quote the description from their website:
MySAC is a library that provides mechanisms for making asynchronous request to MySQL database.
And maybe you will interested in https://github.com/huxingyi/myc if you use libuv. It's a pure c mysql connector wrote by me, you can implement your own network layer or just use the implemented libuv based uvmyc inside the example folder.