We're developing both std and realtime applications that run on a RT-Linux.
question is, what would be an efficient way of logging application traces from both realtime and non-realtime processes?
By effecient, I mean that the process of logging application traces shouldn't cause RT-perf hit by increasing latency, etc.
Traces should ideally be stored into a single file with timestamp, to make it easier to track interaction between processes.
For real time Logging I'll advise use different aproaches than bare logging to files. Writing to files a lot of information will hurt your performance.
I can advice other more lighter mechanismS:
Use statistics/counters to get filling what your application is doing
Write/encode logs in some binary format to be processed offline. This binary format may be more compact and thus lighter.
Since you are on linux, you can use syslog() :
openlog() opens a connection to the system logger for a program.
this means your program forwards messages to another program, which can be of low priority.
If you want something more fancy, then boost logging.
Related
I'm trying to make a software that backups my entire hard drive.
I've managed to write a code for reading the raw data from hard disk sectors. However, i want to have incremental backups. For that i need to know the changed made to OS settings, file changes, everything.
My question is -
Using FileSystemWatcher and Inotify, will i be able to know every change made to every sector in the hard drive ? (OS settings etc)
I'm coding it in C++ for linux and windows.
(Saw this question on Stackoverflow which gave me some idea)
Inotify is to detect changes while your program is running, I'm guessing that FilySystemWatches is similar.
One way to solve this is to have a checksum on each sector or multiple of sectors, and when making a backup you compare the checksums to the list you have and only backup blocks that have been changed.
The MS Windows FileSystemWatcher mechanism is more limited than Linux's Inotify, but both probably will do what you need. The Linux mechanism provides (optional) notification for file reads, which causes the "access timestamp" to be updated.
However, the weakness from your application's perspective is that all file modifications made from system boot up to your program getting loaded (and unload to shutdown) will not be monitored. Your application might need to look through file modification timestamps of many files to identify changed files, depending on the level of monitoring you are targeting.
Both architectures maintain a timestamp for each file tracking when the file was last accessed. If that being updated is a trigger for a backup notification, the Windows mechanism lacking such notification will cause mismatched behavior on the platforms. Windows' mechanism can also drop notifications due to buffer size limitations. Here is a real gem from the documentation:
Note that a FileSystemWatcher does not raise an Error event when an event is missed or when the buffer size is exceeded, due to dependencies with the Windows operating system. To keep from missing events, follow these guidelines:
Increasing the buffer size with the InternalBufferSize property can prevent missing file system change events.
Avoid watching files with long file names. Consider renaming using shorter names.
Keep your event handling code as short as possible.
At least you can control two out of three of these....
I am penning down the features that a remote logging
library might need when built from scratch.
I looked up this: http://www.aggsoft.com/serial-data-logger.htm
I wish to know that what differences can be between a
remote logging library and a remote logger software.
Few things that I thought of:
1. The library can be used in C++ programs to log error messages on the fly.
2. The library will require programming knowledge on the end user's part.
3. The software cannot be used "inside" a C++ program, so we won't be able to log the error messages on the fly? Not sure about this one.
I would like to know that besides logging error messages, what are the things for which it makes sense to use the remote logging library? Sharing big files? Anything else than these two things?
Secondly which is better in what way out of a library and a software - in the current case?
As I mentioned in the my comments to your question, I would think that a logging library would provide some sort of an API/SDK, whereas remote software would not. The same would hold true if its sending messages via TCP/UDP or a serial port. The difference between the 2 options would be how much coding you would have to do. That is, how much would you have to reinvent the wheel?
IMHO, nearly all debug environment/tools support redirect the console output the serial port (using print, or other API). It usually not a a task of Application programmer.
There are other methods for "remote logging":
1) syslog, syslog-ng 's remote service
2) save log local, fetch using ftp
This seems to be typical application:
1. One part of the program should scan for audio files in background and write tags to the database.
2. The other part makes search queries and shows results.
The application should be crossplatform.
So, the main search loop, including adding data to database is not a problem. The questions are:
1. What is the best way to implement this background working service? Boost(asio) or Qt(services framework?)?
2. What is the best approach, to make a native service wrapper using mentioned libraries or emulate it using non gui application?
3. Should I connect gui to the service(how they will communicate using boost or qt?) or directly to the database (could locks be there?)?
4. Will decsision in point 1 consume all CPU usage? And how to avoid that? How to implement scanning for files less cpu consumable?S
I like to use Poco which has a convenient ServerApplication class, which can be used in an application that can be run as either a normal command-line application, or as a Windows service, or as a *nix daemon without having to touch the code.
If you use a "real" database (MySQL, PostgreSQL, SQL Server), then querying the database from the GUI application is probably fine and easier to do. If you use another type of database that isn't necessarily multi-user friendly, then you should communicate with the service using loopback sockets or pipes.
As far as CPU usage, you could just use a bunch of "sleep" calls within your code that searches files to make sure it doesn't hog the CPU and IO ports. Or use some kind of interval notification to allow it to search in chunks periodically.
I'm work on a build tool that launches thousands of processes (compiles, links etc). It also distributes executables to remote machines so that the build can be run accross 100s of slave machines. I'm implementing DLL injection to monitor the child processes of my build process so that I can see that they opened/closed the resources I expected them to. That way I can tell if my users aren't specifying dependency information correctly.
My question is:
I've got the DLL injection working but I'm not all that familiar with windows programming. What would be the best/fastest way to callback to the parent build process with all the millions of file io reports that the children will be generating? I've thought about having them write to a non-blocking socket, but have been wondering if maybe pipes/shared memory or maybe COM would be better?
First, since you're apparently dealing with communication between machines, not just within one machine, I'd rule out shared memory immediately.
I'd think hard about trying to minimize the amount of data instead of worrying a lot about how fast you can send it. Instead of sending millions of file I/O reports, I'd batch together a few kilobytes of that data (or something on that order) and send a hash of that packet. With a careful choice of packet size, you should be able to reduce your data transmission to the point that you can simply use whatever method you find most convenient, rather than trying to pick the one that's the fastest.
If you stay in the windows world (None of your machines is linux or whatever) named pipes is a good choice, because it is fast and can be accessed across the machine boundary. I think shared memory is out of the race, because it can't cross the machine boundary. Distributed com allows to formulate the contract in IDL, but i think XML Messages via pipes are also ok. The xml messages have the benefit to work completely independent from the channel. If yo need linux later you can switch to tcp/ip transport and send your xml messages.
Some additional techniques with limitations:
Another forgotten but hot candidate is RPC (remote procedure calls). Lot of windows services rely on this. But i think it is hard to program RPC
If you are on the same machine and you only need to send some status information, you can regisier a windows message via RegisterWindowMessage() and send messages vie SendMessage()
apart from all the suggestions from thomas, you might also just use a common database to store the results. And if that is too slow use one of the more modern(and fast) key/value databases (like tokyo cabinet/memcachedb/etc).
This sounds like a lot of overkill for the task of verifying the files used in a build. How about, just scanning the build files? or capturing the output from the build tools?
I've recently bought myself a new cellphone, running Windows Mobile 6.1 Professional. And of course I am currently looking into doing some coding for it, on a hobby basis. My plan is to have a service running as a DLL, loaded by Services.exe. This needs to gather som data, and do som processing at regular intervals (every 5-10 minutes).
Since I need to run this at regular intervals, it is a bit of a problem for me, that the system typically goes to sleep (suspend) after a short period of inactivity by the user.
I have been reading all the documentation I could find on MSDN, and MSDN blogs about this subject, and it seems to me, that there are three possible solutions to this problem:
Keep the system in an "Always On"-state, by calling SystemIdleTimerReset periodically. This seems a bit excessive, and is therefore out of the question.
Have the system periodically waken up with CeRunAppAtTime, and enter the unattended state, to do my processing.
Use the unattended state instead of going into a full suspend. This would be transparent to the user, but the system would never go into sleep.
The second approach seems to be preferred, however, this would require an executable to be called by the system on wake up, with the only task of notifying my service that it should commence processing. This seems a bit unnecessary and I would like to avoid this extra executable. I could of course move all my processing into this extra executable, but I would like to use some of the facilities provided when running as a service, and also not have a program pop up (even if its in the background) whenever processing starts.
At first glance, the third approach seems to have the same basic problem as the first. However, I have read on some of the MSDN blogs, that it might be possible to actually conserve battery consumption with this approach, instead of going in and out of suspend mode often (The arguments for this was that the nature of the WM platform is to have a very little battery consumption, when the system is idle. And that going in and out of suspend require quite a bit of processing).
So I guess my questions are as following:
Which approach would you recommend in my situation? With respect to keeping a minimum battery consumption, and a nice clean implementation.
In the case of approach number two, is it possible to eliminate the need for a notifying executable? Either through alternative API functions, or existing generic applications on the platform?
In the case of approach number three, do you know of any information/statistics relevant to the claim, that it is possible to extend the battery lifetime when using unattended mode over going into suspend. E.g. how often do you need to pull the system out of suspend, before unattended mode is to be preferred.
Implementation specific (bonus) question: Is it necessary to regularly call SystemIdleTimerReset to stay in unattended mode?
And finally, if you think I have prematurely eliminated approach number one, please tell me why.
Please include in your response whether you base your response on knowledge, or are merely guessing (the latter is also very welcome!).
Please leave a comment, if you think I need to clarify any parts of this question.
CERunAppAtTime is a much-misunderstood API (largely because of the terrible name). It doesn't have to run an app. It can simply set a named system event (see the description of the pwszAppName parameter in the MSDN docs). If you care to know when it has fired (to lat your app put the device to sleep again when it's done processing) simply have a worker thread that is doing a WaitForSingleObject on that same named event.
Unattended state is often used for devices that need to keep an app running continuously (like an MP3 player) but conserve power by shutting down the backlight (probably the single most power consuming subsystem).
Obviously unattended mode uses significantly more powr than suspend, becasue in suspend the only power draw is for RAM self-refresh. In unattended mode the processor is stuill powered and running (and several peripherals may be too - depends on how the OEM defined their unattended mode).
SystemIdleTimerReset simply prevents the power manager from putting the device into low-power mode due to inactivity. This mode, whether suspended, unattended, flight or other, is defined by the OEM. Use it sparingly because when your do it impacts the power consumption of the device. Doing it in unattended mode is especially problematic from a user perspective because they might think the device is off (it looks that way) but now their battery life has gone south.
I had a whole long post detailing how you shouldn't expect to be able to get acceptable battery life because WM is not designed to support what you're trying to do, but -- you could signal your service on wakeup, do your processing, then use the methods in this post to put the device back to sleep immediately. You should be able to keep the ratio of on-time-to-sleep-time very low this way -- but as you say, I'm only guessing.
See also:
Power-Efficient Apps (MSDN)
Power To The People (Developers 1, Developers 2, Devices)
Power-Efficient WM Apps (blog post)