I went through all the answers that were available regarding real-time reading a text file but none seems to work.
In my program 1 have a continuously growing text file being written by a hardware which is giving two coordinates (two columns).
In program 2, I want to read those coordinates in real time and move another hardware to the coordinates that are being written.
The biggest problem is I want to work with shortest possible delay (under 50ms).
I tried notepad++, but its refresh rate is 3 seconds which is too much.
Can anyone tell how can this be done?
Your fastest response is to either poll (read the hardware) directly or to have the hardware create an event (interrupt) that calls your program.
Writing to a file takes time. The OS has to find space on the hard drive, write to the hard drive; and not to mention the time required to ramp up the motors to spin the hard drive.
Writing to memory is a lot quicker. A more efficient method is for the H/W to write to memory rather than a file. Alternately, a memory mapped file or RAM drive will be the next best option.
Also remember that Windows is not a real-time operating system. You have other tasks in your system being swapped out and executed. This takes time away from your "real time" requirements. You may want to research Windows to see if there is an API that allows your program exclusive access to the processor (or makes your program a very high priority).
Research "Windows Drivers" to write code that can service your H/W and perform activities in real time.
I tried this:
int main()
{
std::ifstream ifs("file.txt");
if (ifs.is_open())
{
std::string line;
while (true)
{
while (std::getline(ifs, line)) std::cout << line << "\n";
if (!ifs.eof()) break;
ifs.clear();
}
}
return 0;
}
But it reads till the end and when i add more values to my text file, it doesn't read that. But when I refresh my file, I get the o/p on the console.
I have also tried using tellg and seekg but that also doesn't help.
Related
I am learning to use TWINCAT 3 with C++ and as my first work, I decided open a .txt file and get a number inside, and put on an string or an integer.
I have read all the documentation and have many questions. I discovered that I can't use the C++ libraries, just the TWINCAT functions. Then I got lost.
First: What are the exacts steps to Open a file in TWINCAT 3 with C++?
Second: How can I read the data in the file and put in an string or integer?
I would like to do that in a CycleUpdate.
I am sorry if it is a noob question.
As a first step, you have to understand that TwinCAT is providing you a PLC with real-time capabilities. Which means each task you program will need to be executed at every cycle: your task must NOT exceed a certain duration.
Many accesses to the operating system require a lot of wait times, that you would not keep in a real-time system. For that, most of function blocks you will find are equipped with an "Execute" boolean input (or similar) and outputs like "Busy", "Done" and "Error" (even "ErrorID"). These are here in order to start a process and check periodically (i.e. in each cycle) whether the process is complete.
You cannot manage a file opening, reading, writing or closing (OS functions) within a single CycleUpdate. This is the cost for assuring real-time capabilities besides.
I have a fortran code which needs to read a series of ascii data files (which all together are about 25 Gb). Basically the code opens a given ascii file, reads the information and use it to do some operations, and then close it. Then opens another file, reads the information, do some operations, and close it again. And so on with the rest of ascii files.
Overall each complete run takes about 10h. I usually need to run several independent calculations with different parameters, and the way I do is to run each independent calculation sequentially, so that at the end if I have 10 independent calculations, the total CPU time is 100h.
A more rapid way would be to run the 10 independent calculations at the same time using different processors on a cluster machine, but the problem is that if a given calculation needs to open and read data from a given ascii file which has been already opened and it's being used by another calculation, then the code gives obviously an error.
I wonder whether there is a way to verify if a given ascii file is already being used by another calculation, and if so to ask the code to wait until the ascii file is finally closed.
Any help would be of great help.
Many thanks in advance.
Obamakoak.
Two processes should be able to read the same file. Perhaps action="read" on the open statement might help. Must the files be human readable? The I/O would very likely be much faster with unformatted (sometimes call binary) files.
P.S. If your OS doesn't support multiple-read access, you might have to create your own lock system. Create a master file that a process opens to check which files are in use or not, and to update said list. Immediately closing after a check or update. To handle collisions on this read/write file, use iostat on the open statement and retry after a delay if there is an error.
I know this is an old thread but I've been struggling with the same issue for my own code.
My first attempt was creating a variable on a certain process (e.g. the master) and accessing this variable exclusively using one-sided passive MPI. This is fancy and works well, but only with newer versions of MPI.
Also, my code seemed happy to open (with READWRITE status) files that were also open in other processes.
Therefore, the easiest workaround, if your program has file access, is to make use of an external lock file, as described here. In your case, the code might look something like this:
A process checks whether the lock file exists using the NEW statement, which fails if a file already exists. It will look something like:
file_exists = .true.
do while (file_exists)
open(STATUS='NEW',unit=11,file=lock_file_name,iostat=open_stat)
if (open_stat.eq.0) then
file_exists = .false.
open(STATUS='OLD',ACTION=READWRITE',unit=12,file=data_file_name,iostat=ierr)
if (ierr.ne.0) stop
else
call sleep(1)
end if
end do
The file is now opened exclusively by the current process. Do the operations you need to do, such as reading, writing.
When you are done, close the data file and finally the lock file
close(12,iostat=ierr)
if (ierr.ne.0) stop
close(11,status='DELETE',iostat=ierr)
if (ierr.ne.0) stop
The data file is now again unlocked for the other processes.
I hope this may be useful for other people who have the same problem.
I'm trying to work with nasty large xml and text documents: ~40GBs.
I'm using Visual Studio 2012 on Windows 7.
I'm going to use 'Xerces' to snag the header/'footer tag' from the xmls.
I want to map an area of the file, say.. 60-120MBs.
Split the Map into (3 * processors/cores) equal parts. Setting each part as a buffer and loading the buffers into an array.
Then using (#processors/cores) while statments in new threads, I will synchronously count characters/lines/xml cycles while chewing through the the buffer array. When one buffer is completed the the process will jump to the next 'available' buffer and the completed buffer will be dropped out of memory. At the end I will add the total results into a project log.
Afterwards, I will reference the log, Split the files by character count/size(Or other option) to the nearest line or cycle and drop in the header and 'footer tag' to all the splits.
I'm doing this so I can import massive data to a MySQL server over a network with multiple computers.
My Question is, how do I create the buffer array and the file map with new threads?
Can I use :
win CreateFile
win CreateFileMapping
win MapViewOfFile
with standard ifstream operations and char buffers or should I opt something else?
Futher clarification:
My thinking is that if I can have the hard drive streaming the file into memory from one place and in one direction that I can use the full processing power of the machine to chew through seperate but equal buffers.
~Flavor: It's kind of like being a Shepard trying to scoop food out from one huge bin with 3-6 Large buckets with only two arms for X sheep that need to stay inside the fenced area. But they all move at the speed of light.
A few ideas or pointers might help me along here.
Any thoughts are Most Welcome. Thanks.
while(getline(my_file, myStr))
{
characterCount += myStr.length();
lineCount++;
if(my_file.eof()){
break;
}
}
This was the only code at run time for the test. 2hours, 30+min. 45-50% total processor for the program running it on a dual core 1.6Mhz laptop with 2GB RAM. Most of the RAM loaded right now is 600+MB from ~50 tabs open in firefox, Visual Studio at 60MB, then etcs.
IMPORTANT: During the test, the program running the code, which is only a window, and a dialog box, seemed to dump it's own working and private set of ram, down to like 300K ish, and didn't respond for the length of the test. I need to make another thread for the while statement I'm sure. But this means that NONE of the file was read into a buffer. The CPU was struggling for the entire run to keep up with the tinyest effort from the hard drive.
P.S. Further proof of CPU bottlenecking. It might take me 20min to transfer than entire file to another computer over my wireless network. Which includes the read process and a socket catch to write process on the other computer.
UPDATE
I used this adorable little thing to go from the previous test time to about 15-20min which is in line with what Mats Petersson was saying.
while (my_file.read( &bufferOne[0], bufferOne.size() ))
{
int cc = my_file.gcount();
for (int i = 0; i < cc; i++)
{
if (bufferOne[i] == '\n')
lineCount++;
characterCount++;
}
currentPercent = characterCount/onePercent;
SendMessage(GetDlgItem(hDlg, IDC_GENPROGRESS), PBM_SETPOS, currentPercent, 0);
}
Granted this is a single loop and it actually behaved much more appropriately than the previous test. This test was ~800% faster than the tight loop shown above this one with Getline. I set the buffer for this loop at 20MB. I jacked this code from: SOF - Fastest Example
BUT...
I would like to point out that while polling the process in resource mon and task manager, it clearly showed the first core at 75-90% usage, the second fluxuately 25-50% (Pretty standard for some minor background stuff that I have open), and the hard drive at.. wait for it... 50%. Some 100% disk time spikes but also some lows at 25%. All of which basically means that Splitting the buffer processing between two different threads could very well be a benefit. It will use all the system resources but.. that's what I want. I'll update later today when I have the working prototype.
MAJOR UPDATE:
Finally finished my project after a bunch of learning. No File Map needed. Only a bunch of vector char's. I have successfully built a dynamically executing file stream line and character counter.
The good news, went from the previous 10-15min marker to ~3-4min on a 5.8GB file, BOOYA!~
Very short answer: Yes, you can use those functions.
For reading data, it's likely the most efficient method to map the file content into memory, since it saves having to copy the memory into a buffer in the application, just read it straight into the place it's supposed to go. So, no problem as long as you have enough address space available - 64-bit machines should certainly have plenty, in a 32-bit system it may be more of a scarce resource - but for sections of a few hundred MB, it shouldn't be a huge issue.
However, using multiple threads, I'm not at all convinced. I have a fair idea that reading more than one part of a very large file will be counter productive. This will increase the amount of head movement on the disk, which is a large portion of transfer rate. You can count on some 50-100MB/s transfer rates for "ordinary" systems. If the system has some sort of raid controller or some such, maybe around double that - very exotic raid controllers may achieve three times.
So reading 40GB will take somewhere in the order of 3-15 minutes.
The CPU is probably not going to be very busy, and running multiple threads is quite likely to worsen the overall performance of the system.
You may want to keep a thread for reading and one for writing, and only actually write out the data once you have a sufficient amount of it, again, to avoid unnecessary moves of the read/write head on the disk(s).
I'm reading a big file using fread. When I interrupt the program during it using Ctrl+C, the program hangs and is not killable, also not with kill -9. It simple sticks with 100% CPU, keeping the RAM it had already allocated. It would be great to get that fixed, but it would also be okay just to be able to kill that application from outside (the main problem being the fact that I can't restart that machine myself).
Is there a way of doing that in Unix?
Thanks!
Here is the source:
int Read_New_Format(const char* prefix,const char* folder)
{
char filename[500];
long count_pos;
//open files for reading.
sprintf(filename,"%s/%s.pos.mnc++",folder,prefix);
FILE *pos = fopen(filename,"r");
if(pos==NULL)
{
printf("Could not open pos file %s\n",filename);
}
//read the number count of entries in each of the three files.
fread(&count_pos,sizeof(long),1,pos);
printf("[...]");
//read the complete file into an array.
float *data_pos = new float[3*count_pos];
fread(data_pos,3*sizeof(float),*count_pos,pos);
printf("Read files.\n");
[...]
}
If your program cannot be interrupted by a signal, that almost surely means it's in an uninterruptable sleep state. This is normally an extremely short-lived state that only exists momentarily while waiting for the physical disk to perform a read or write, either due to an explicit read or write call that can't be satisfied by the cache, or one resulting from a page fault where a disk-backed page is not swapped into physical memory.
If the uninterruptable sleep state persists, this is almost surely indicative of either extremely high load on the storage device (a huge number of IO requests all happening at once) or, much more likely, damaged hardware.
I suspect you have a failing hard disk or scratched optical disc.
Problem wasn't reproducable after some days. Maybe a problem with the file system. As a workaround, direct use of the unix library routines instead of fread worked.
(This isn't my program, but I'll try to provide all the relevant information to the best of my knowledge.)
There is a program which reads binary files that are roughly 300MB in size, processes them and outputs some information. The program uses ifstream for file input and streams are correctly initialized and closed for each read.
The program has to read each file multiple times. Reading a file for the first time takes about 3 seconds, and each consecutive read takes about 0.1 seconds. If several files are processed, going back to the first file will still yield fast read speeds, but after some time re-reading a file becomes slow.
Additionally, if a file is copied to another location, the speed of the first read of the new file is roughly 0.1 seconds.
If you do the math, the speed of consecutive reads is roughly the advertised read speed of the hard drive.
All this looks like file locations are cached by either the OS or the hard drive, so that on consecutive reads you don't have to seek out file locations.
Does anyone know what exactly is causing the slowdown on the initial read, and if it can be prevented? Three seconds may not seem like a lot, but they add about 5 hours to the total time needed to correctly process every file.
Also, the program runs on Fedora 14 and Scientific Linux, with both OS's having their default file systems.
Any ideas would be appreciated.
Linux will try and copy the file into RAM to make the next read faster - I am guessing this is what is happening. The initial read is actual off disk - subsequent reads are out of the file cache because the entire file has been copied to RAM
The OS (Linux) has a disk cache. After you read the file once, it's in the cache.
My guess would be that maybe the first time it reads the file it takes longer because it loads some information into the cache?
After the first time, it just uses some of the information in the cache.
Yes, the data becomes cached. You might force that caching with the readahead syscall (or simply by having another process read it). If using mmap you could also use madvise