Setup: 4 processes need to monitor a centralized source (I am thinking Message Queue) and be alerted when a message is put into the queue. This is for reporting errors that each process needs to know about to decide what they individually will do about it.
I have a few questions however, since I want to make sure Message Queues are the best way to go about it before I venture to far down that road.
Does the Message Queue API on Windows CE have events/notifications of some sort? If not, does the Windows CE OS itself allow you to tie in to the queue somehow for alerts of new messages arriving? Is there a better Inter Process communication tool for this kind of setup and requirements?
EDIT: Also, this will be a running log of errors that cant be lost, so I will put it on my flash memory.
What always works is a simple TCP connection via loopback. This also makes it easy to debug on a desktop system, in case you want. Concerning CE's message queues, those are always 1:1, so you'd need as many as you have connections between processes. Also, if you want to use them in a TCP stream-like manner, you'll need one for each direction.
There is another way you could use, and that is to monitor the file that you're logging the info to. I'm not sure to what extent CE supports a notification API for file changes, but if you don't need low-latency reactions, you can always poll every now-and-then. Alternatively, use a named event after writing to the logfile.
Related
I need to write a C++ application that should read firewall status of Windows, and then need to keep an eye continuously if admin/someuser
changes the firewall status (lets say when my program was started firewall was disabled and after sometime Admin enabled it).
To implement this, I have created a thread that periodically(10 seconds) poll the code that checks Windows firewall status, but this doesn't look an efficient solution to me as continuous polling is required.
Is there a way to get event automatically in my program if firewall status changes (for example, FindFirstChangeNotification, using this I can get notification if any change in directory)? This will avoid continuous polling and will make program more efficient I think.
Any help is appreciated.
I know there is Windows ETW which anti-viruses use and which has all the info you need. It is a big system log where you subscribe to log/event providers. Pretty much everything that happens in system gets reported there via event which you can listen/wait for. I don't know the links to more useful pages with a list of loggers connected to ETW so here is the more general page: https://learn.microsoft.com/en-us/windows-hardware/drivers/devtest/event-tracing-for-windows--etw-. You need to find out how to use C++ ETW API and the name/ID of the firewall events provider with a list of event types, then using API subscribe to this provider and setup a callback for when an event that interests you (here change of firewall status) occurs and that is it.
We have 2 systems between which we want messages to be exchanged. I am currently designing the application and have been given 2 choices.
System 1 to push messages to an intermediate location (FTP or SQS) and system 2 (running BizTalk) reading the messages from that location and processing it.
Exposing Schema/Orchestration as a web service in system 2 which would be consumed by system 1.
Any suggestions which method would be better in terms of error handling and scalability.
If you can, always go for an asynchronous approach, through a queuing system. This way, your application can be running independent of your back end. And then I would advise for Service Bus for Windows Server (heavier installation), Windows Azure Service Bus (as a service, in the cloud, internet connection needed) or with MSMQ (store and forward included!). These provide transactional behavior and can be considered as very reliable. Other lightweight options are indeed through file exchange or FTP.
Web service or REST connectivity is also very easy to set up, but then you have synchronous behavior, which has its benefits:
you can get a 'real-time' ack back when your message is delivered by BizTalk
it's easy to set up and to monitor
So, as mostly, the answer is 'it depends'.
There's only a 'best way' for you particular app and there are a number of conditions to consider.
The easiest way is a shared location on the File System (OS File System vs FTP doesn't matter so much), especially if order is not important.
If order has to be maintained to there's a guaranteed delivery requirement, then a Message Queue is a good choice, MSMQ/WMQ.
Of course, HTTP/SOAP is always an option.
Realistically, any of these methods will get the message there so you have to consider the benefits of each protocol.
I want to have my applications communicate to each other. I think something like a server-client model would suit me well, but I was also wondering if there was a different way. I want this way to not involve those windows event hooks.
Note:
I am assuming you want to communicate with different processes on the same machine, although many of these concepts can apply across computers as well.
What you are looking for is IPC (Inter Process Communication).
You can do IPC via:
File
Signal
Socket
Message queue
Pipe
Named pipe
Semaphore
Shared memory
Message passing
memory-mapped file
My personal recommendation is a loopback connection via socket commands. It is difficult to give you much help beyond that without knowing more details about what you want to do.
You could loopback over Ethernet or use named pipes.
Look up Inter-process Communication (IPC) for a list of all related topics.
Edit: Given your comments about both being in different processes, then you are best off sending information across a network (ie sockets programming). This would give you the added advantage of being able to run the main process and the debug process on different machines. It's a bit hard to give you much information on it. You'll need to figure out what sort of requests you will send across the network and what sort of data you will send back in response. Effectively you need to design your own simple protocol.
I'm working on the design of a remote control application. From my iPhone or a web browser, I'll send a few commands. Soon my home computer will perform the commands and send back results. I know there are remote desktop apps, but I want something programmable, something simpler, and something that I wrote.
My current direction is to use Amazon Simple Queue Service (SQS) as the message bus. The iPhone places some messages in a queue. My local Java/JRuby program notices the messages on the queue, performs the work and sends back status via a different queue.
This will be a very low-volume application. At $1.00 for a million requests (plus a handful of data transfer charges), Amazon SQS looks a lot more affordable than having my own server of any type. And super reliable, that's important for me too.
Are there better/standard toolkits or architectures for this kind of remote control? Cost is not a big issue, but I prefer the tons I learn by doing it myself.
I'm moderately concerned about security, but doubt it will be a problem. The list of commands recognized will be very short, and only recognized in specific contexts. No "erase hard drive" stuff.
update: I'll probably distribute these programs to some other people who want the same function, but who don't have Amazon SQS accounts. For now, they'll use anonymous access to my queues, with random 80-character queue names.
Well, I think it's a clever approach -- and as you said, the costs for your little traffic aren't even worth mentioning.
As I mentioned in the comment, it's a good way to leave your home machine behind your firewall and not have an open port on the internet.
I would suggest using OnlineMQ.com as a start; they have a free package.
(Edited to try to explain better)
We have an agent, written in C++ for Win32. It needs to periodically post information to a server. It must support disconnected operation. That is: the client doesn't always have a connection to the server.
Note: This is for communication between an agent running on desktop PCs, to communicate with a server running somewhere in the enterprise.
This means that the messages to be sent to the server must be queued (so that they can be sent once the connection is available).
We currently use an in-house system that queues messages as individual files on disk, and uses HTTP POST to send them to the server when it's available.
It's starting to show its age, and I'd like to investigate alternatives before I consider updating it.
It must be available by default on Windows XP SP2, Windows Vista and Windows 7, or must be simple to include in our installer.
This product will be installed (by administrators) on a couple of hundred thousand PCs. They'll probably use something like Microsoft SMS or ConfigMgr. In this scenario, "frivolous" prerequisites are frowned upon. This means that, unless the client-side code (or a redistributable) can be included in our installer, the administrator won't be happy. This makes MSMQ a particularly hard sell, because it's not installed by default with XP.
It must be relatively simple to use from C++ on Win32.
Our client is an unmanaged C++ Win32 application. No .NET or Java on the client.
The transport should be HTTP or HTTPS. That is: it must go through firewalls easily; no RPC or DCOM.
It should be relatively reliable, with retries, etc. Protection against replays is a must-have.
It must be scalable -- there's a lot of traffic. Per-message impact on the server should be minimal.
The server end is C#, currently using ASP.NET to implement a simple HTTP POST mechanism.
(The slightly odd one). It must support client-side in-memory queues, so that we can avoid spinning up the hard disk. It must allow flushing to disk periodically.
It must be suitable for use in a proprietary product (i.e. no GPL, etc.).
How is your current solution showing its age?
I would push the logic on to the back end, and make the clients extremely simple.
Messages are simply stored in the file system. Have the client write to c:/queue/{uuid}.tmp. When the file is written, rename it to c:/queue/{uuid}.msg. This makes writing messages to the queue on the client "atomic".
A C++ thread wakes up, scans c:\queue for "*.msg" files, and if it finds one it then checks for the server, and HTTP POSTs the message to it. When it receives the 200 status back from the server (i.e. it has got the message), then it can delete the file. It only scans for *.msg files. The *.tmp files are still being written too, and you'd have a race condition trying to send a msg file that was still being written. That's what the rename from .tmp is for. I'd also suggest scanning by creation date so early messages go first.
Your server receives the message, and here it can to any necessary dupe checking. Push this burden on the server to centralize it. You could simply record every uuid for every message to do duplication elimination. If that list gets too long (I don't know your traffic volume), perhaps you can cull it of items greater than 30 days (I also don't know how long your clients can remain off line).
This system is simple, but pretty robust. If the file sending thread gets an error, it will simply try to send the file next time. The only time you should be getting a duplicate message is in the window between when the client gets the 200 ack from the server and when it deletes the file. If the client shuts down or crashes at that point, you will have a file that has been sent but not removed from the queue.
If your clients are stable, this is a pretty low risk. With the dupe checking based on the message ID, you can mitigate that at the cost of some bookkeeping, but maintaining a list of uuids isn't spectacularly daunting, but again it does depend on your message volume and other performance requirements.
The fact that you are allowed to work "offline" suggests you have some "slack" in your absolute messaging performance.
To be honest, the requirements listed don't make a lot of sense and show you have a long way to go in your MQ learning. Given that, if you don't want to use MSMQ (probably the easiest overall on Windows -- but with [IMO severe] limitations), then you should look into:
qpid - Decent use of AMQP standard
zeromq - (the best, IMO, technically but also requires the most familiarity with MQ technologies)
I'd recommend rabbitmq too, but that's an Erlang server and last I looked it didn't have usuable C or C++ libraries. Still, if you are shopping MQ, take a look at it...
[EDIT]
I've gone back and reread your reqs as well as some of your comments and think, for you, that perhaps client MQ -> server is not your best option. I would maybe consider letting your client -> server operations be HTTP POST or SOAP and allow the HTTP endpoint in turn queue messages on your MQ backend. IOW, abstract away the MQ client into an architecture you have more control over. Then your C++ client would simply be HTTP (easy), and your HTTP service (likely C# / .Net from reading your comments) can interact with any MQ backend of your choice. If all your HTTP endpoint does is spawn MQ messages, it'll be pretty darned lightweight and can scale through all the traditional load balancing techniques.
Last time I wanted to do any messaging I used C# and MSMQ. There are MSMQ libraries available that make using MSMQ very easy. It's free to install on both your servers and never lost a message to this day. It handles reboots etc all by itself. It's a thing of beauty and 100,000's of message are processed daily.
I'm not sure why you ruled out MSMQ and I didn't get point 2.
Quite often for queues we just dump record data into a database table and another process lifts rows out of the table periodically.
How about using Asynchronous Agents library from .NET Framework 4.0. It is still beta though.
http://msdn.microsoft.com/en-us/library/dd492627(VS.100).aspx