I have writen a script that reads at real time a log file. The problem I have got is that the location of that logfile changes every midnight, same filename diferent folder.
I can regenerate the file location path but I don't know how to start/stop the script to reprocess the log file? Right now I am doing it by hand.
My suggestion would be
(a) If it's a linux box, why not run a "cron" job (which will stop and start the script)
(b) Alternatively, in your code, detect when the file can no longer be read, then switch folders
Related
I am running a local development environment from the command line with one GNU-screen terminal running a text editor and the other running a continuous build process (Expo).
I want to know if changes made to a file with a text-editor in one GNU-Screen terminal immediately take effect in another terminal with the ongoing process, or if GNU-Screen waits for the process to end before updating the file system?
If not, is there a program or GNU-screen option that will allow this to occur so I don't have to restart the build each time I make a change?
Changes to files take effect as soon as the file is written to disk.
You can verify this as follows:
In screen one, write a file change to disk.
In screen two, view your change.
We have File like Zip, txt Files in Windows SFTP Server and we use Informatica for our ETL Jobs , but our concern is the vendors who drop Files in SFTP Server they drop files in random times , and the files are of different sizes so How can we detect a File transfer is complete or Not??
Unfortunately, you can't. When your client is asking the FTP server for a list of files, it receives the current state on the server. There's no way of telling if one of the files is currently being written to or not.
So the only way here is to work out some kind of protocol with the vendor. You would need to work with some kind of lock file. If the vendor is writing to the file, they first have to create the lock file. Same goes for your ETL job when reading the file, you would first have to create the lock file and the vendor is not allowed to start a new file writing process until the lock is removed. You get the idea.
It all depends on how fool prove the solution needs to be. Another option is to let the vendor write to a temporary file first. Only when the upload process is finished, they rename the file to its final name.
Writing to temp file and renaming once write is complet is one option, as Socken23 indicated. Others might be:
create empty .ready file once write to target file is done. Your process should then check for the existence of .ready file and read data from the other one.
perform a sequence of file size checks on FTP before starting the ETL. E.g. check file size, wait one minute, check again, wait & check third time. If all sizes match, you may assume the write is complete.
I have hundreds of text files in a folder named "in/". I need to run a python script which takes one file at a time, process it and drop it in a folder named "out/". I have the python script in place to do this.
As the number of text files to be processed are very large (10000) and as all the file processing are independent I wanted to use "make -j" to get the best out of my CPU which has 8 cores. I created a make file which looks like this :
SCRIPT_DIR:=/home/xyz/abc/scriptFolder<br/>
IN_DIR:=/home/xyz/abc/data/in/in10000<br/>
OUT_DIR:=/home/xyz/abc/data/out/out10000<br/><br/>
chk:
cd $(OUT_DIR); \<br/>
python $(SCRIPT_DIR)/process_parallel.py --inFile $(IN_DIR)/\*
As mentioned process_parallel.py takes in one file at a time processes it and drops it as a text file in the current folder which is OUT_DIR. I ran htops after this and checked. I could see only one process running, where as I should have seen 8 as I ran it with -j 8. Can you please guide me where I am wrong ?
My first thought is writing a shell script to do this. Something like:
for f in in/*.txt;
do
./process_parallel.py $f &
done
wait
Your OS scheduler should take care of parallelizing the processing across the CPU cores. You can then call the script inside your Makefile.
There's also GNU Parallel https://www.gnu.org/software/parallel/
I am not sure if this is even a valid question. I am not a master at understanding the workings of system. One of my program writes logs to a text file. Another email program runs on scheduler and emails and archives the log file if found in the folder.
My question is, If at any given instant if the first program is writing information into the file and at the same time email scheduler runs what will happen? Will the email program be able to mail the file and archive it? If Yes, will the earlier program writing the file crash? How to handle this scenario without crashing either programs?
No matter what, your setup will lead to some kind of trouble.
I think the simplest solution would be to have the program that writes the log file do this e.g. 5 minutes before the emailer/archiver is scheduled to run:
start a new file for logging
copy or rename the old file to the file that the emailer/archiver uses.
I have a Native WinApi C++ Application that finds media(.wmv, .mp3 etc.) files in a specified directory & creates random playlists. The first time the application is run(& only the first time) I want to prompt the user to specify a 'home' directory that the Application will always check for media files & create a playlist from.
My Problem: I dont know of a way how I could determine when the Application is run for the 1st time?
Is there a standard way, maybe a Win32 function that I can use to detect when the Application is run for the 1st time?
Some ideas I have come up with are: (but they seem like hacks or overkill(installer idea))
The application .exe is 322kb(which is tiny & doesn't require an
installer right?) in size so I could create an installer (I was
thinking if someone is installing the application then I know its the
first run & I can prompt them then).
I could have a text file(or xml) called appData.txt & have the 1st
line where I store the home path directory. So "home_path=undefined",
on application run, I look in the text file, if the home_path ==
undefined then I prompt them to specify a home path if its not undefined then I read that directory for media files.
Any ideas of how I can determine when my Application is run for the very first time?
In the installer you could create a registry value for your program.
Then when you start your program, check the registry value.
When you run the program for the first time update that value to so you know it's been run already.
I would use the text file because you are going to have to store the user's directory somewhere anyway, might as well use it for first run detection as well. It has the added bonus that if the file is deleted, you will know that you have to prompt the user again since you no longer know what their home directory is.
You can set some registry value when your App runs for first time and check it on every run. If it is already set then App was already run. If not - set it.
Create a log file on first run. If it exists, then it's not the first time.
try
{
// open log.txt
// do second time run logic here
}
catch(file does not exist)
{
// create log.txt
// first run logic here
}