The code below creates the file, but is not writing data to it.
p=outfile("outfile.txt" "w")
fprintf(p "write to out file")
Cadence uses buffered IO for files, so to see your output you will need to explicitly flush the port like so:
drain(p)
The port will also get automatically flushed when you close it, so this is only necessary if you want to see intermediate output.
Related
We have File like Zip, txt Files in Windows SFTP Server and we use Informatica for our ETL Jobs , but our concern is the vendors who drop Files in SFTP Server they drop files in random times , and the files are of different sizes so How can we detect a File transfer is complete or Not??
Unfortunately, you can't. When your client is asking the FTP server for a list of files, it receives the current state on the server. There's no way of telling if one of the files is currently being written to or not.
So the only way here is to work out some kind of protocol with the vendor. You would need to work with some kind of lock file. If the vendor is writing to the file, they first have to create the lock file. Same goes for your ETL job when reading the file, you would first have to create the lock file and the vendor is not allowed to start a new file writing process until the lock is removed. You get the idea.
It all depends on how fool prove the solution needs to be. Another option is to let the vendor write to a temporary file first. Only when the upload process is finished, they rename the file to its final name.
Writing to temp file and renaming once write is complet is one option, as Socken23 indicated. Others might be:
create empty .ready file once write to target file is done. Your process should then check for the existence of .ready file and read data from the other one.
perform a sequence of file size checks on FTP before starting the ETL. E.g. check file size, wait one minute, check again, wait & check third time. If all sizes match, you may assume the write is complete.
My need is quite simple:
with log4cplus, I'd like to be able to write a log in a log file and to flush the log file everytime before I write in it. This way, while I run my application, I will only have one single line in my log file.
I've tryed the append=False property, but it only flush the log file at startup.
I could do it by hand in C++ but I don't want to write C++ code as the product is already in prod.
Any idea ?
Thanks,
Use the ImmediateFlush property to force the file appender to flush after each event.
I am using VC++ to generate a text file and write (via fstream) data continuously to it. I have another application2 (NOT C++) which accesses that same file which c++ appends. At the instant application2 accesses the file, new data from C++ program cannot be written to it. It seems like the new data from c++ goes to some temporary file. when application2 closes the file, new data gets updated to the file. I want the data to be written to the file in real time and be read at the same time by application2. What should I do in c++ to make new data appear in the file which is opened by application2?
C++ side:
int realTimeValues // this variable is updated continuously
FILE * pFileTXT;
while(1)
{
pFileTXT = fopen ("realTimeData.txt","a"); // Opening file in append mode
fprintf (pFileTXT, "%d\n",realTimeValues); // saving values to file
fclose (pFileTXT) // Closing the file
}
On the application2 side I can't tell how exactly its opening this file. The application is Universal Real-Time Software Oscilloscope. In the menu there is an option "read from a file"
Ugh. There are a number of things at work:
By default, fprintf output may be buffered in a private ram buffer. This buffer is flushed only as needed or when you do fclose. If you want to force the data out, there's a setbuf call you can make (read the docs) or explicitly call fflush after each fprintf.
I do not know if fopen allows for simultaneous readers. Probably not, given the buffering behavior. If you want read/write sharing, you should look at the documentation to see if there is an explicit mode parameter to enable this.
Unfortunately, this only covers the part you can control. If the reader (an application you do not control) doesn't support shared reading, then you're out of luck.
How can I check if a file is still being written? I need to wait for a file to be created, written and closed again by another process, so I can go on and open it again in my process.
In general, this is a difficult problem to solve. You can ask whether a file is open, under certain circumstances; however, if the other process is a script, it might well open and close the file multiple times. I would strongly recommend you use an advisory lock, or some other explicit method for the other process to communicate when it's done with the file.
That said, if that's not an option, there is another way. If you look in the /proc/<pid>/fd directories, where <pid> is the numeric process ID of some running process, you'll see a bunch of symlinks to the files that process has open. The permissions on the symlink reflect the mode the file was opened for - write permission means it was opened for write mode.
So, if you want to know if a file is open, just scan over every process's /proc entry, and every file descriptor in it, looking for a writable symlink to your file. If you know the PID of the other process, you can directly look at its proc entry, as well.
This has some major downsides, of course. First, you can only see open files for your own processes, unless you're root. It's also relatively slow, and only works on Linux. And again, if the other process opens and closes the file several times, you're stuck - you might end up seeing it during the closed period, and there's no easy way of knowing if it'll open it again.
You could let the writing process write a sentinel file (say "sentinel.ok") after it is finished writing the data file your reading process is interested in. In the reading process you can check for the existence of the sentinel before reading the data file, to ensure that the data file is completely written.
#blu3bird's idea of using a sentinel file isn't bad, but it requires modifying the program that's writing the file.
Here's another possibility that also requires modifying the writer, but it may be more robust:
Write to a temporary file, say "foo.dat.part". When writing is complete, rename "foo.dat.part" to "foo.dat". That way a reader either won't see "foo.dat" at all, or will see a complete version of it.
You can try using inotify
http://en.wikipedia.org/wiki/Inotify
If you know that the file will be opened once, written and then closed, it would be possible for your app to wait for the IN_CLOSE_WRITE event.
However if the behaviour of the other application doing the writing of the file is more like open,write,close,open,write,close....then you'll need some other mechanism of determining when the other app has truly finished with the file.
So I have many log files that I need to write to. They are created when program begins, and they save to file when program closes.
I was wondering if it is better to do:
fopen() at start of program, then close the files when program ends - I would just write to the files when needed. Will anything (such as other file io) be slowed down with these files being still "open" ?
OR
I save what needs to be written into a buffer, and then open file, write from buffer, close file when program ends. I imagine this would be faster?
Well, fopen(3) + fwrite(3) + fclose(3) is a buffered I/O package, so another layer of buffering on top of it might just slow things down.
In any case, go for a simple and correct program. If it seems to run slowly, profile it, and then optimize based on evidence and not guesses.
Short answer:
Big number of opened files shouldn't slow down anything
Writing to file will be buffered anyway
So you can leave those files opened, but do not forget to check the limit of opened files in your OS.
Part of the point of log files is being able to figure out what happened when/if your program runs into a problem. Quite a few people also do log file analysis in (near) real-time. Your second scenario doesn't work for either of these.
I'd start with the first approach, but with a high-enough level interface that you could switch to the second if you really needed to. I wouldn't view that switch as a major benefit of the high-level interface though -- the real benefit would normally be keeping the rest of the code a bit cleaner.
There is no good reason to buffer log messages in your program and write them out on exit. Simply write them as they're generated using fprintf. The stdio system will take care of the buffering for you. Of course this means opening the file (with fopen) from the beginning and keeping it open.
For log files, you will probably want a functional interface that flushes the data to disk after each complete message, so that if the program crashes (it has been known to happen), the log information is safe. Leaving stuff in standard I/O buffers means excavating the data from a core dump - which is less satisfactory than having the information on disk safely.
Other I/O really won't be affected by holding one - or even a few - log files open. You lose a few file descriptors, perhaps, but that is not often a serious problem. When it is a problem, you use one file descriptor for one log file - and you keep it open so you can log information. You might elect to map stderr to the log file, leaving that as the file descriptor that's in use.
It's been mentioned that the FILE* returned by fopen is already buffered. For logging, you should probably also look into using the setbuf() or setvbuf() functions to change the buffering behavior of the FILE*.
In particular, you might want to set the buffering mode to line-at-a-time, so the log file is flushed automatically after each line is written. You can also specify the size of the buffer to use.