I have a C++ application that needs to send structured data to an Akka actor. The best option I found (Google, stackoverflow...) is to use protocol buffer and ZeroMQ, since it looks like everyone recommends it.
However I struggled the whole day trying to make it work, having various crashes into my Scala actor code (with strange Windows socket errors). And when I take a deeper look at it, I notice that it seems zeromq disappeared from the Akka official documentation a while ago, and the most recent documentation I read about it said that ZeroMQ 3 was still not supported by zeromq-scala-bindings underneath (while the version 4 is already out).
Would it be a better option to use the Camel-netty extension and pass the information through JSON ?
Thanks !
A fairly simple way would be to write a HTTP endpoint using Spray.io. Spray has JSON support and since it is built on Akka it communicates seamlessly with other Actors. This has the advantage that the data you send to the endpoint does not have to match the message format the Actor is expecting. You can change the message the actor is expecting without changing what your C++ code sends. For bi-directional communication there is also web socket support.
Related
I'm brand new to c++ and know next to nothing about web protocols or websockets, so this may seem ridiculous.
I make websites that are 100% ajax and want to incorporate websockets. Fastcgi++ is everything I could hope for for the ajax demands, but it doesn't have websockets, and I chose websocket++ over libwebsockets since websocket++ is more or less a simple #include, so I assumed that I could incorporate it into fastcgi++.
I think I've figured out fastcgi++, and it looks like most of the action happens in Fastcgipp::Request then Fastcgipp::Http::Sessions for session data http://www.nongnu.org/fastcgipp/doc/2.1/a00005.html; however, I think I have to do the same thing with websocket++'s server::handler for handling the websocket https://github.com/zaphoyd/websocketpp/wiki/Creating-Applications-using-WebSocket--, and now I'm lost at sea.
Enter my complete inexperience with c++: I think I have to use virtual inheritance, but I have no idea. Also, if I could even properly "subclass" both, how do I make sure that they don't run over each other?
Please show me a basic example of how websocket++ can use fastcgi++'s session management.
A WebSocket connection cannot be processed by an HTTP request/response workflow. In order to use something like fastcgi++ with both regular HTTP requests and with WebSocket requests it would need to have some way of recognizing a WebSocket handshake and piping that off to another handler instead of processing it as HTTP. I don't see an obvious pass through mode of that sort in its documentation, but I could be missing something.
If such a feature exists, WebSocket++ can be used in stream mode where it disables all of its network elements and just processes streams of bytes piped in from another networking library.
Some alternatives:
WebSocket++ supports HTTP pass through. This is essentially the opposite of what is described above. WebSocket++ would be used as the networking layer. It would process incoming WebSocket connections and would pass off HTTP requests to some other subsystem.
WebSocket++ and fastcgi++ could be run on different ports or different hostnames. This could be done in the same program or separate programs. With client side requests directed to the appropriate host/port.
Disclaimer: I am the author of WebSocket++
I am a novice in C++, but have a lot of experience in Java EE.
I need to write a small app which is heavily asynchronous. It receives data from HTTP and stores it in a Queue (it needs to have guaranteed delivery and very high throughput), something like ActiveMQ or OpenMQ, maybe via JMS.
Then another C++ app/listener takes out data from the Queue (through some Listener which is activated by the Queue directly, not by my pooling), connects to a MySQL database and does some business logic calculations and sends the message to another Queue.
In Java EE this would be a web app that would send messages to a JMS queue. Message-Driven Beans would be consumers of these messages in an EJB module, and a Session EJB would send messages to the outgoing JMS queue.
Can someone with C++ experience please explain some basics to me:
Is JMS the only option for C++ for guaranteed delivery Queues? Do you suggest ActiveMQ or something else, having in mind that the message Consumer would be in C++.
Do I need to create some kind of a multi-threaded daemon in C++ that listens for Queue messages, or is this thread creation (message consumption) part of ActiveMQ's implementation of C+ consumers?
Any other suggestions on how to implement the scenario above would be very much appreciated.
EDITED: I would prefer a message broker and client to be in C++. ActiveMQ is a Java product, which is not really what we need.
1 - JMS - Java Message Service - is just an API reference for Java and Java only. There is no standard in messaging that applies to C++ except for AMQP (which is, to my opinion, not really working cross implementation as good as it should). To C++ you kinda have to rely on specific vendor libraries for each message broker implementation.
Suggestions of implementations:
ActiveMQ - It has a nice C++ API (Called CMS) which is modeled and named after JMS - so you will feel familiar with the API. The main broker will run on Java non the less - might be the simplest choice.
IBM WebSphere MQ - Not open source, enterprise class broker that runs native (written in C), and has C++ libraries. Rather nice when you get over the learning curve and the price does not matter.
RabbitMQ - Very popular reliable messaging with high performance and open source. Has C++ client libs but is written in Erlang and runs within the erlang/otp runtime.
Apache QPID - Less known AMQP/JMS broker. Comes in two flavours server side, Java and C++ where the C++ broker has better performance. Comes with C++ client libs.
2 - For multi threading, the JMS specs does not really come with a solution either. It's more like the Java EE container (or Spring Framework) that simply wrapps the management of threads and relieves the developer from it. ActiveMQ does not come with much more than a few support classes in this case and as far as I know, none of the other vendor libraries does either. So, look for some library that wrapps threading (I have no clue) or deal with the consumer threads yourself. It shouldn't all that messy, done right.
I have some framework doing specific task in C++ and a django-based web app. The idea is to launch this framework, receive some data from it, send some data or request and check it's status in some period.
I'm looking for the best way of communication. Both apps run on the same server. I was wondering if a json server in C++ is a good idea. Django would send a request to this server, and server would parse it, and delegate a worker thread to complete the task. Almost all data that need to be send is string-like. Other data will be stored in database so there is no problem with that.
Is JSON a good idea? Maybe you know some better mechanism for local communication between C++ and django?
If your requirement is guaranteed to always have the C++ application on the same machine as the Django web application, include the C++ code by converting it into a shared library and wrapping python around it. Just like this Calling C/C++ from python?
JSON and other serializations make sense if you are going to do remote calls and the code needs to communicate across machines.
JSON seems like a fair enough choice for data serialization - it's good at handling strings and there's existing libraries for encoding/decoding JSON in both python & C++.
However, I think your bigger problem is likely to be the transport protocol that you use for transferring JSON between your client and server. Here's some options:
You could build an HTTP server into your C++ application (which I think might be what you mean by "JSON server" in your question), which would work fine, though might be a bit of a pain to implement unless you get a hold of a library to handle the hard work for you.
Another option might be to use the 0MQ library to send JSON (or otherwise) messages between your client and server. I think this would probably be a lot easier than implementing a full HTTP server, and 0MQ has some interprocess communication code that would likely be a lot faster than sending things over the network.
A third option would just be to run your C++ as a standalone application and pass the data in to it via stdin or command line parameters. This is probably the simplest way to do things, though may not be the most flexble. If you were to go this way, you might be better off just building a Python/C++ binding as suggested by ablm.
Alternatively you could attempt to build some sort of job queue based on redis or something other database system. The idea being that your django application puts some JSON describing the job into the job queue, and then the C++ application could periodically poll the queue, and use a seperate redis entry to pass the results back to the client. This could have the advantage that you could reasonably easily have several "workers" reading from the job queue with minimal effort.
There's almost definitely some other ways to go about it, but those are the ones that immediately spring to mind.
I'm trying to implement a solution using HornetQ. Since I need to access it through a C++ application, that raises me a problem. I'm compiling the activemq-cpp builtin example, and changing it to work with stomp instead of openwire (HornetQ doesn't understand openwire). The application refuses to produce messages on the intended queue. Seems that a lot of people are having the same issue, but no one has the answer. (someone said it's a bug on the cms API)
Anyone has a pratical example of HornetQ working with a C++ app?
PS: Obviously the activemq-cpp example works with an activemq server using openwire.
HornetQ is probably mapping destination names differently then the ActiveMQ C++ Stomp client, for instance in ActiveMQ a topic destination is prefixed with /topic/ and a queue is /queue/. I beleive this is different in HornetQ but not really sure. You may want to look in their docs for what they use, if its configurable then you could alter it to match what the CMS client is sending. You could also modify your local copy of CMS to send the destination name using the HornetQ prefix.
Regards
Tim.
www.fusesource.com
The only solution I have seen is a HornetQ to ActiveMQ bridge written in java then have the C++ app work with ActiveMQ. You might be able to do something with JNI to handle marshaling messages into your app.
(Edited to try to explain better)
We have an agent, written in C++ for Win32. It needs to periodically post information to a server. It must support disconnected operation. That is: the client doesn't always have a connection to the server.
Note: This is for communication between an agent running on desktop PCs, to communicate with a server running somewhere in the enterprise.
This means that the messages to be sent to the server must be queued (so that they can be sent once the connection is available).
We currently use an in-house system that queues messages as individual files on disk, and uses HTTP POST to send them to the server when it's available.
It's starting to show its age, and I'd like to investigate alternatives before I consider updating it.
It must be available by default on Windows XP SP2, Windows Vista and Windows 7, or must be simple to include in our installer.
This product will be installed (by administrators) on a couple of hundred thousand PCs. They'll probably use something like Microsoft SMS or ConfigMgr. In this scenario, "frivolous" prerequisites are frowned upon. This means that, unless the client-side code (or a redistributable) can be included in our installer, the administrator won't be happy. This makes MSMQ a particularly hard sell, because it's not installed by default with XP.
It must be relatively simple to use from C++ on Win32.
Our client is an unmanaged C++ Win32 application. No .NET or Java on the client.
The transport should be HTTP or HTTPS. That is: it must go through firewalls easily; no RPC or DCOM.
It should be relatively reliable, with retries, etc. Protection against replays is a must-have.
It must be scalable -- there's a lot of traffic. Per-message impact on the server should be minimal.
The server end is C#, currently using ASP.NET to implement a simple HTTP POST mechanism.
(The slightly odd one). It must support client-side in-memory queues, so that we can avoid spinning up the hard disk. It must allow flushing to disk periodically.
It must be suitable for use in a proprietary product (i.e. no GPL, etc.).
How is your current solution showing its age?
I would push the logic on to the back end, and make the clients extremely simple.
Messages are simply stored in the file system. Have the client write to c:/queue/{uuid}.tmp. When the file is written, rename it to c:/queue/{uuid}.msg. This makes writing messages to the queue on the client "atomic".
A C++ thread wakes up, scans c:\queue for "*.msg" files, and if it finds one it then checks for the server, and HTTP POSTs the message to it. When it receives the 200 status back from the server (i.e. it has got the message), then it can delete the file. It only scans for *.msg files. The *.tmp files are still being written too, and you'd have a race condition trying to send a msg file that was still being written. That's what the rename from .tmp is for. I'd also suggest scanning by creation date so early messages go first.
Your server receives the message, and here it can to any necessary dupe checking. Push this burden on the server to centralize it. You could simply record every uuid for every message to do duplication elimination. If that list gets too long (I don't know your traffic volume), perhaps you can cull it of items greater than 30 days (I also don't know how long your clients can remain off line).
This system is simple, but pretty robust. If the file sending thread gets an error, it will simply try to send the file next time. The only time you should be getting a duplicate message is in the window between when the client gets the 200 ack from the server and when it deletes the file. If the client shuts down or crashes at that point, you will have a file that has been sent but not removed from the queue.
If your clients are stable, this is a pretty low risk. With the dupe checking based on the message ID, you can mitigate that at the cost of some bookkeeping, but maintaining a list of uuids isn't spectacularly daunting, but again it does depend on your message volume and other performance requirements.
The fact that you are allowed to work "offline" suggests you have some "slack" in your absolute messaging performance.
To be honest, the requirements listed don't make a lot of sense and show you have a long way to go in your MQ learning. Given that, if you don't want to use MSMQ (probably the easiest overall on Windows -- but with [IMO severe] limitations), then you should look into:
qpid - Decent use of AMQP standard
zeromq - (the best, IMO, technically but also requires the most familiarity with MQ technologies)
I'd recommend rabbitmq too, but that's an Erlang server and last I looked it didn't have usuable C or C++ libraries. Still, if you are shopping MQ, take a look at it...
[EDIT]
I've gone back and reread your reqs as well as some of your comments and think, for you, that perhaps client MQ -> server is not your best option. I would maybe consider letting your client -> server operations be HTTP POST or SOAP and allow the HTTP endpoint in turn queue messages on your MQ backend. IOW, abstract away the MQ client into an architecture you have more control over. Then your C++ client would simply be HTTP (easy), and your HTTP service (likely C# / .Net from reading your comments) can interact with any MQ backend of your choice. If all your HTTP endpoint does is spawn MQ messages, it'll be pretty darned lightweight and can scale through all the traditional load balancing techniques.
Last time I wanted to do any messaging I used C# and MSMQ. There are MSMQ libraries available that make using MSMQ very easy. It's free to install on both your servers and never lost a message to this day. It handles reboots etc all by itself. It's a thing of beauty and 100,000's of message are processed daily.
I'm not sure why you ruled out MSMQ and I didn't get point 2.
Quite often for queues we just dump record data into a database table and another process lifts rows out of the table periodically.
How about using Asynchronous Agents library from .NET Framework 4.0. It is still beta though.
http://msdn.microsoft.com/en-us/library/dd492627(VS.100).aspx