googling about asynchronous /non-blocking connectors for mysql i went basically to this post
However, it's been 2 years and following whats happening on drizzle is a bit confusing at the moment. libdrizzle was a separate dependency at some point but they decided to merge it with the rest of the project. Are there other options for asynchronous database access from c++?
I've been looking at OTL, ODB and OpenDBX, but they all seem to be synchronous (require a separate thread for non-blocking operation)
I had the same desire and came to the conclusion that it's not supported. Even with the MySQL C API you can use the low-level functions to issue queries and wait for a response asynchronously, but you cannot ever get full asynchronous result collection--you always end up blocking from the time the first piece of the result is returned until the last.
I don't have direct experience with it, but I've read that Postgres does support full asynchrony (at least in the C API).
I used to used MySAC in my own project. It works well though is a little outdated. I just quote the description from their website:
MySAC is a library that provides mechanisms for making asynchronous request to MySQL database.
And maybe you will interested in https://github.com/huxingyi/myc if you use libuv. It's a pure c mysql connector wrote by me, you can implement your own network layer or just use the implemented libuv based uvmyc inside the example folder.
Related
How does ZeroC ICE compare to 0MQ? I know that 0MQ/Crossroads and DDS are very similar, but cant seem to figure out where ICE comes in.
I need to quickly implement a system that offloads real-time market-data from C++ to C#, as a first phase of my project. The next phase will be to implement an Event Based architecture with an underlying Pub/Sub design.
I am willing to use TCP.. but the the system is currently running on a single 24 core server.. so an IPC option would be nice. From what I understand ICE is only TCP, while DDS and 0mq have an IPC option.
Currently ,I am leaning towards using Protobuf with either ICE or Crossroads IO. Got turned off from the OpenSplice DDS website. Ive done lots research on the various options, was originally considering OpenMPI + boost:mpi, but there does not seem to be MPI for .NET.
My question is:
How does ICE compare to 0MQ? I cant wrap my head around this. Was unable to find anything online that compares the two.
thanks in advance.
........
More about my project:
Currently using CMAKE C++ on Windows, but the plan is to move to CentOS at some point. An additional desired feature is to store the tic data and all the messages in a "NoSql" database such as Hbase/Hadoop or HDF5. Do any of these middleware/messaging/pub-sub libraries have any database integration?
Some thoughts about ZeroC:
Very fast; Able to have multiple endpoints; Able to load balance on the endpoints; Able to reconnect to a different endpoint in case one of the node goes down. This is transparent to the end user; Has good tool chain (IceGrid, IceStorm, IceBox, etc); Distributed, high availability, multiple failover, etc
Apart from that, I have used it for hot swapping code modules (something similar to Erlang) by having the client create the proxy with multiple endpoints, and later on bring down each endpoint for a quick upgrade one by one. With the transparent retry to a different endpoint, I could have the system up and running the whole time i did an upgrade. Not sure if this is an advertised feature or an unadvertised side-effect :)
Overall, it is very easy to scale out your servers if need be using ZeroC Ice.
I know ZeroMQ provides a fantastic set of tools and messaging patterns and I would keep using it for my pet projects. However, The problem that i see is that it is very easy to go overboard and lose track of all your distributed components. This is a must have in a distributed environment. How will you know where your clients/server are when you need to upgrade? If one of components down the chain does not receive a message, how to identify where the issue is? the publisher? the client? or any one of the bridges (REP/REQ, XREP/XREQ, etc) in between?
Overall, ZeroC provides a much better toolset and ecosystem for enterprise solutions.
And it is open source :)
Jaybny,
ZMQ:
If you want real good performance and the only job for Phase 1 of your job is to move data from C++ to C#, then Zmq is the best option.
Having a pub/sub model for event driven architecture is also something that Zmq can help you with, with its in-built messaging pattern.
Zmq also supports your IPC requirements in this case. Eg: you can have one instance of your application that consumes 24 cores by multithreading and communicating via IPC.
ZeroC Ice:
Ice is a RPC framework very much like CORBA.
Eg.
Socket/ZMQ - You send message over the wire. Read it at the other end, parse the message, do some action, etc.
ZeroC Ice - Create a contract between client and server. Contract is nothing but a template of a class. Now the client calls a proxy method of that class, and the server implements/actions it and returns the value. Thus, int result = mathClass.Add(10,20) is what the client calls. The method, parameters, etc is marshalled and sent to the server, server implements the Add method, returns the result, and the client gets 30 as the result. Thus on the client side, the api is nothing but a proxy for a servant running on a remote host.
Conclusion:
ZeroC ICE has some nice enterprisy features which are really good. However, for your project requirements, ZMQ is the right tool.
Hope this helps.
For me.. the correct answer was Crossroads I/O . It does everything I need.. but still unable to pub/sub when using protobufs... im sure ZeroC ICE is great for distributed IPC, but 0MQ/Crossroads, gives you the added flexibility to use Inter-Thread-Communication.
Note: on windows, 0mq does not have IPC.
So, all in all, the crossroads fork of 0mq is the best. but you will have to roll your own windows/ipc (or use tcp::127..) , and publisher side topic filtering features for pub/sub.
nanomsg, from the guy who wrote crossroads and 0mq (i think).
http://nanomsg.org/
I am penning down the features that a remote logging
library might need when built from scratch.
I looked up this: http://www.aggsoft.com/serial-data-logger.htm
I wish to know that what differences can be between a
remote logging library and a remote logger software.
Few things that I thought of:
1. The library can be used in C++ programs to log error messages on the fly.
2. The library will require programming knowledge on the end user's part.
3. The software cannot be used "inside" a C++ program, so we won't be able to log the error messages on the fly? Not sure about this one.
I would like to know that besides logging error messages, what are the things for which it makes sense to use the remote logging library? Sharing big files? Anything else than these two things?
Secondly which is better in what way out of a library and a software - in the current case?
As I mentioned in the my comments to your question, I would think that a logging library would provide some sort of an API/SDK, whereas remote software would not. The same would hold true if its sending messages via TCP/UDP or a serial port. The difference between the 2 options would be how much coding you would have to do. That is, how much would you have to reinvent the wheel?
IMHO, nearly all debug environment/tools support redirect the console output the serial port (using print, or other API). It usually not a a task of Application programmer.
There are other methods for "remote logging":
1) syslog, syslog-ng 's remote service
2) save log local, fetch using ftp
I'm looking for a jabber server library in C++.
I tried glooxd but it's tough to compile, buggy and no activity since more than a year now.
What I'm trying to do, is to be able to build a process that accept xmpp stream, implement it own way to authentify and build custom rosters.
Check out Swiften, a relatively new addition to the XMPP scene. It's primarily used in the client Swift, but also by Spectrum 2, which can act as a server to clients.
In the Swift git repo, there's also a tool called Slimber, that acts as a client in serverless messaging mode, and then presents that as a normal client interface. The server parts of Spectrum 2 and Slimber may be useful for you to study.
Check out the libxmpp project on Sourceforge. I don't know much about it. However, a number of years ago, I wrote a C++ layer on top of the loudmouth library. It's not hard to wrap the C library constructs in thin C++ classes.
Libxmpp: http://sourceforge.net/projects/xmpp/
Loudmouth: https://launchpad.net/loudmouth
I'm looking for a library to deal with multiple simultaneous HTTP connections (pref. on a single thread) to use in C++ in Windows so it can be Win32 API based. This is for a client application that must process a list of requests but keep 4 running at all times until the list is complete.
So far, I have tried cURL (multi interface) which seems to be the most appropriate that I have found but my problem is that I may have a queue of 200 requests but I need to only run 4 of them at a time. This becomes problematic when one request may take 2 seconds and another may take 2 mins as you have to wait on all handles and receive the result of all requests in one block. If anyone knows a way round this it would be very useful.
I have also tried rolling my own using WinHTTP but I need to throttle the requests so they would ideally need to be on a single thread and use callbacks for data which WinHTTP does not do.
The best thing I've found which would solve all my problems is ASIHTTPRequest but unfortunately it's Mac OSX only.
Thanks,
J
Have you looked at boost.asio? http://www.boost.org/doc/libs/1_46_1/doc/html/boost_asio.html
Its meant to scale well and has http server examples:
http://www.boost.org/doc/libs/1_46_1/doc/html/boost_asio/examples.html
Did you tried Boost Asio ?
Is multiplataform and stellar performance, and with nice examples of HTTP.
http://www.boost.org/doc/libs/1_46_1/doc/html/boost_asio.html
Asio is a great library but it's generic, the HTTP examples are just that: examples, there's no support for redirection, authentication and so on.
I know of two libraries built on top of Boost & Asio that support the HTTP protocol: cpp-netlib and Pion Network Library but AFAIK neither directly supports what you want.
All that being said if you're comfortable with using libcurl it should be pretty easy to use the "easy" interface with callbacks and implement the requests queue yourself.
libcurl's multi interface supports exactly what you're asking for.
Do any asynchronous connectors exist for Mysql that can be used within a C or C++ application? I'm looking for something that can be plugged into a reactor pattern written in Boost.Asio.
[Edit:] Running a synchronous connector in threads is not an option.
http://forums.mysql.com/read.php?45,183339,183339
enjoy
Updated link to the original article showing how to do async mysql queries:
http://jan.kneschke.de/projects/mysql/async-mysql-queries-with-c-api/
I had a similar problem with a very different technologies: Twisted python (reactor-based IO) and sqlAlchemy (??). While searching for a solution, I found about an sAsync project that simply created a separate thread for sqlAlchemy and then responded to requests.
Given that ASIO is based on low level OS features (such as aio_read() or ReadFileEx() etc) and an OS-level reactor (or proactor, in Windows' case) I don't think you have another chance than emulating the 'asynchronousness' by similar means.
Running a synchronous connector in threads is not an option
Think about it: the libmysqlclient / mysqlclient.dll you're using makes synchronous socket calls. The OS scheduler will correctly switch to another thread until the I/O is finished, so what's the difference? (apart from the fact that you shouldn't make 2k threads for this..)
Edit: mysql_real_connect() supports an UNIX socket parameter. You can supposedly read yourself from the mysql server port and write to that UNIX socket only using ASIO. Like a proxyfication.
I know this is an old question, but consider looking at the new Boost.Mysql library: https://anarthal.github.io/mysql/index.html
[ Running a synchronous connector in threads is not an option
Think about it: the libmysqlclient / mysqlclient.dll you're using makes synchronous socket calls. The OS scheduler will correctly switch to another thread until the I/O is finished]
This is bugging me! - the 'another thread' could as easily be a second sync. connection to mysql, and should be handled by mysql just as it would another client altogether? My gutfeel is that it should work using multiple threads.
MySQL Connector/C++ is a C++ implementation of JDBC 4.0
The reference customers who use MySQL Connector/C++ are:
- OpenOffice - MySQL Workbench
Learn more: http://forums.mysql.com/read.php?167,221298
There is a project called DBSlayer that puts another layer in front of MySQL that you talk to through JSON. http://code.nytimes.com/projects/dbslayer
have you considered using libdrizzle? i have used only an old version, from when it was a separate project from drizzle, and i tested the asynchronous query features, but i never did any actual benchmarks worth mentioning.
I think the only solution will be to create an asynchronous service that wraps a standard connector. You'll need to understand the ODBC APIs though.