I need optimize this code. Currently, this code works. I can extract the datetimeoriginal tag in a file using the extern tool "exiftool". But this is too slow when this process is repeated. is there a way the extract this information in variables directly?
void image::add_info_creation_date(char *name_jpg){
//Execute exiftool in bash
stringstream ss;
ss << "exiftool.exe -datetimeoriginal -timezone images\\" << name_jpg << " >> temporal.txt";
error = system(ss.str().c_str());
if (error != 0){
printf("The value returned was: %d.\n", error);
ss.clear();
exit(1);
}
ss.clear();
...
}
Sorry for my grammar, i am not native english speaker.
is there a way the extract this information in variables directly?
Yes, it is possible.
Your options are to either find and use a library which can read the EXIF information for you, or to read & understand the JPEG & EXIF formats, and read the file yourself.
Since a quick search turns up several libraries that do these things, and since at least some documentation is available for both standards, it doesn't seem too hard.
If you read the file yourself, note that the JPEG file format contains multiple tagged sections, including both the image and its metadata - you just need to walk the sections until you find the EXIF data and then parse that.
Executing ExifTool once for each file can be slow, but there's a way to run it so you can feed it a stream of commands. First, create an argument file and keep it open for output; let's call it args.txt. Then launch
exiftool -stay_open True -# args.txt
For each file, write the arguments, one argument per line, something like this, to args.txt:
-datetimeoriginal
-timezone
{filename}
-execute
When finished, write the following to args.txt:
-stay_open
False
This may get you sufficient performance. Don't close args.txt till you're finished, but it will help if you flush any buffers after each -execute.
Related
I have a folder filled with hundreds of .aac files, and I'm trying to "pack" them into one file in the most efficient way that I can.
I have tried the following, but only end up with a file that's only a few bytes long or audio that sounds warbled and distorted heavily.
// Now we want to get all of the files in the folder and then repack them into an aac file as fast as possible
void Repacker(string FileName)
{
string data;
boost::filesystem::path p = "./tmp/";
boost::filesystem::ofstream aacwriter;
aacwriter.open("./" + FileName + ".aac", ios::app);
boost::filesystem::ifstream aacReader;
boost::filesystem::directory_iterator it{ p };
cout << "Repacking File!" << endl;
while (it != boost::filesystem::directory_iterator{}) {
aacReader.open(*it, std::ios::in | ios::binary);
cout << "Writing " << *it << endl;
aacReader >> data;
++it;
aacwriter << data;
aacReader.close();
}
aacwriter.close();
}
I have looked at the following questions to try and help solve this issue
Merging two files together
Merge multiple txt files into one
How do I read an entire file into a std::string in C++?
Read whole ASCII file into C++ std::string
Merging Two Files Into One
However unfortunately, none of these answer my question.
They all either have to do with text files, or functions that don't deal with hundreds of files at once.
I am trying to write Binary data, not text. The audio is either all warbled or the file is only a few bytes long.
If there's a memory efficent method to do this, please let me know. I am using C++ 20 and boost.
Thank you.
These files have an internal structure: header, blocks/frames, etc. and the simple presence of multiple headers within the concatenated file will mess up the expected result.
Take a look at the AAC file format structure, you'll see that it's not so simple.
Your best try should be to use FFMPEG, since it has a feature to concatenate media files without being forced to reencode data. It's a bit complex because FFMPEG's command line is quite complex and not always extremely intuitive, but it should works as long as all AAC files uses the same encoding and characteristics. Otherwise, you'll need to re-encode them - but it can be done automatically, too.
Check this web research to get some base informations.
Otherwise, you may use the base libraries used by FFMPEG, for example libavcodec (available at ffmpeg.org), Fraunhofer FDK AAC, etc. but you'll have way, way more work to do and, finally, you'll do exactly what FFMPEG already do, since it relies on these libraries. Other AAC libraries won't be really easier to use.
Obviously, you can also "embed" FFMPEG within your application, call tools like ffprobe to analyze files and call ffmpeg executable automatically, as a child process.
CAUTION: Take a GREAT care about licensing if you plan to distribute your program. FFMPEG licensing is really not simple, most of the time it's distributed as sources to avoid vicious cases.
I am reading a text file using C++. I am using the first line to indicate if the file has changed or not. For example, text file will look like :
0
9
After the text file has been changed I am flipping the first line. So, the new text file will look like:
1
5
C++ code will perform its functions after reading the new input from the text file. Currently, I am using infinite loop to detect changes in the text file. Is there any alternative implementation of detecting changes in the text file using C++? This is my solution so far:
do{
std::ifstream reader;
reader.open(READFILE);
if (!reader){
printf("Error opening the reader file!!");
exit(1);
}
reader >> status >> variable;
if(status != reader_file_status){
reader_file_status = status;
return true
}
} while(true);
filesystem change events are platform specific, so you'll need different solutions for different platforms.
osx: file system events api doc
freebsd/osx: kqueue doc
linux: inotify doc
windows: windows api doc
It seems some cross platform wrappers are around. A quick googling revealed:
https://github.com/emcrisostomo/fswatch (gpl3, osx/freebsd/linux/windows/solaris)
http://doc.qt.io/archives/qt-4.8/qfilesystemwatcher.html (lgpl, osx/freebsd/windows)
I have been scratching my head around this problem for a while, and I couldn't find any answer through surfing the web either.
The problem is that I call system("csvtojson someFile.csv 1> someOtherFile.json") inside my program to produce a JSON file. After this line I want to open, read, and process the JSON file. Although, I can see that the file is created, but fopen() returns NULL.
I read that system() is synchronized so I think the rest of my program will not get executed until the system call is finished, and so the file will be created.
I suspect the problem is somehow related to redirecting the output stream using "1>"; not sure, though.
Any help or hint will be much appreciated.
Thanks! :)
P.S. I don't want to use a library to convert csv to JSON, and I can't perform the conversion outside the program because there are tons of very large csv files and the only way for me is to convert each to a JSON file inside the program, run my algorithm, and move to the next csv file ( converting it to JSON and saving it in the very same JSON file). So in total I have only one JSON file, being like a buffer for my csv files. Having said that, if anyone has a better design approach that can be implemented quickly, that would be also great.
UPDATE : Actual code that exhibits the problem, copied from the OP's answer:
int main(){
system("csvtojson Test_Trace.csv 1> ~/Traces/Test_Trace.json");
FILE* traceFile = fopen("~/Traces/Test_Trace.json", "r");
if(traceFile == NULL)
perror("Error in Openning the trace file");
else
cout << "Successfull openning of the trace file!" << endl;
return 0;
}
Thank you guys for your answers. I had to be more detailed in my question as the problem seemed to be somewhere that wasn't clear from my question.
I figured out what was the problem, and would like to share it here (not a super interesting finding, but worth mentioning).
I wrote a simple program to find the problem:
int main(){
system("csvtojson Test_Trace.csv 1> ~/Traces/Test_Trace.json");
FILE* traceFile = fopen("~/Traces/Test_Trace.json", "r");
if(traceFile == NULL)
perror("Error in Openning the trace file");
else
cout << "Successfull openning of the trace file!" << endl;
return 0;
}
If you run this program you will get the error message No such file or directory, but if you replace the address string with the absolute location, i.e., /home/USER_ID/Traces/Test_Trace.json, in both system(...) and fopen(...) calls, your code will work fine. Interestingly, myself suspected that this could be the problem and I changed just the one for system(...) but still it wasn't working (though the file was being created in the location that was passed to fopen(...)).
EDIT: Thanks to #Peter's comment, this problem was because system() call takes care of ~, but fopen() does not and need an absolute path. So there is really no need to have both functions been given the absolute path.
Anyhow,
Thanks Again. :)
Perhaps the reason for this is because the system command hasn't finished executing by the time your program continues to the next instructions where it tries to read from the file that hasn't been created yet.
Although, this isn't the best way, putting in a short pause might make the difference, or at least let you know if that is the issue.
I am trying to understand how basic I/O with files is handled in c++ or c. My aim is to read file line by line and send the lines across to a remote server. If the line is sent, I want to delete it from the file.
One way I tried was that I kept a count of the lines read and called an system() system call to delete the 'count' number of lines. I used the bash command: sed -i -e 1,'count'd filename.
After that I continued reading the file and surprisingly it worked as planned.
I have two questions:
Is this way reliable?
And why does this work at all, when while
reading the file I deleted a part of it and yet it works? What if I
did a seek to a previous position, what then?
Best,Digvijay
PS:
I would be glad if somebody could suggest a better way.
Also here is the code for the program I wrote:
#include<iostream>
#include<fstream>
#include<string>
#include<sstream>
#include<cstdlib>
int main(){
std::ifstream f;
std::string line;
std::stringstream ss;
int i=0;
f.open("in.txt");
if(f.is_open()){
while(getline(f,line)){
std::cout<<line<<std::endl;
i++;
if(i==2)break;
}
ss<<"sed -i -e 1,"<<i<<"d in.txt";
system(ss.str().c_str());
while(getline(f,line)){
std::cout<<line<<std::endl;
}
}
return 0;
}
Edit:
Firstly thanks for taking the time to write answers. But here is some extra information which I missed out on earlier. The files I am dealing with are log files. So they are constantly being appended with information from devices. The reason why I want to avoid creating a copy is, because the log file themselves are very big(at times) and plus this would help to keep the log file short. Since they would be divided into parts and archived on the server.
Solution
I have found the way to deal with the problem. Apparently Thomas is right, that sed does create a new file. So the old file remains as is. Using this, I can read n lines, call the system function, close the file pointer and open it again. I do this on small chunks of the log, repeatedly until it becomes small and hence efficient to deal with. The server while archives the logs in 1gb files.
However I have a new question, due to memory constraint, I need to know if it is possible to split a log file into two efficiently. (Which possibly would be another question on SO)
Most modern file systems don't support deleting lines at the beginning of the file, so doing so would be very inefficient.
The normal solution to your actual problem is to stop writing to your log file when it reaches some size, then start writing to a new file. The code that copies the files can delete a whole file once it has been written (this is an efficient operation).
sed writes a new version of the file, while the program keeps reading the same version that it opened. This is the usual behavior of Unix and Linux when a program writes a file that another program has open.
You can see this for yourself with this small C program:
#include <stdlib.h>
#include <stdio.h>
int main(void) {
FILE *f = fopen("in.txt", "r");
while (1) {
rewind(f);
int lines = 0;
int c;
while ((c = getc(f)) != EOF)
if (c == '\n')
++lines;
printf("Number of lines in file: %d\n", lines);
}
return 0;
}
Run that program in one window, and then use sed in another window to edit the file. The number of lines printed by the program will stay the same, even if the file on disk has been edited, and this is because Unix keeps the old, open version, even if it is no longer accessible to other programs.
As to your first question, how reliable your solution is, as far as I can see it should be reliable, except with the usual caveats about the system crashing or running out of memory in the middle of an update, someone else accessing the file, and of course all the problems with the system call. It is not very efficient, though, and for large data sets you might want to do it differently.
sujin's comment about using a temporary file for the lines you want to keep seems reasonable. It would be both faster and safer. Keep the original file, so if the system crashes you'll still have your data, and wait until you have finished to rename the old file to "in.txt.bak", and then rename your temporary file to "in.txt".
First off, avoid use of system calls as much as you can (if possible, don't use it at all) as they create race conditions and other problems which drastically (and often) detrimentally affect the outcome of your program. This especially true if access to files are involved.
Given your problem, there are a number of ways to do this, each with its own caveats.
I'll cover three possible solutions:
1) If the file is small enough:
you can read in the entire thing in a data structure (vector, list, deque, etc.)
delete the original file
determine how many lines to read (and send off via server protocol)
then write the remaining lines as the name of the original file.
If you intend to parallelize your program later on, this may be a better solution, provided that the file is small. Note: small is a relative term, but is generally limited by how much memory you have available.
2) If the file is quite large or you're limited by memory constraints, you will have to get creative by using buffers. Once you've read a line and successfully sent it off via your program, you determine where the file pointer is and copy the remaining information until the end of the current file as a new file. Once done, close and delete the old file, then close and rename the new file the same name as the old file.
3) If your solution doesn't have to be in C++, you can use shell-scripting or (controversially) another language to get the job done.
1) No, it's not reliable.
2) The C++ runtime library reads your file in blocks (internally) which are then parceled out to your (higher level) input requests until the block(s) is(are) exhausted, forcing it to (internally) read more blocks from disk. Since one or more physical blocks are read in before you make any call to sed, it/they cannot be altered if sed happens to change that first part of the file.
To see your code fail, you would need to make the input file big enough that there are remaining blocks of the file that have not been read in (internally by the runtime library) before you call sed. By "fail" I mean your program would not see all the characters that were originally in the file before sed clobbered some lines.
As the other guys said, you have to make another file with the records you need after read the original file and then delete it. But in this application perhaps you will see more useful a fifo than a file. If you are on a *NIX platform check up about the makefifo statement from the console.
It is like a file with the singularity that after read a line it gets deleted.
I'm a newbie looking for a fast and easy way to parse a text file in C or C++ (wxWidgets)
The file will look something like this (A main category with "sub-objects") which will appear in a list box
[CategoryA]
[SubCat]
Str1 = Test
Str2 = Description
[SubCat] [End]
[SubCat]
Str1 = Othertest
...
[CategoryA] [End]
Any suggestions?
Sounds like you want to parse a file that's pretty close to an ini file.
There's at least a few INI parser libraries out there: minIni, iniParser, libini, for instance.
It should be fairly easy to write your own parser for this if you use streams. You can read a file using an std::ifstream:
std::ifstream ifs("filename.ext");
if(!ifs.good()) throw my_exceptions("cannot open file");
read_file(ifs);
Since it seems line-oriented, you would then first read lines, and then process these:
void read_file(std::istream& is)
{
for(;;) {
std::string line;
std::getline(is, line);
if(!is) break;
std::istringstream iss(line);
// read from iss
}
if(!is.eof()) throw my_exceptions("error reading file");
}
For the actual parsing, you could 1) first peek at the first character. If that's a [, pop it from the stream, and use std::getline(is,identifier,']') to read whatever is within '[' and ']'. If it isn't a [, use std::getline(is, key, '=') to read the left side of a key-value pair, and then std::getline(is, value) to read the right side.
Note: Stream input, unfortunately, is usually not exactly lightning fast. (This doesn't have to be that way, but in practice this often is.) However, it is really easy to do and it is fairly easy to do it right, once you know a very few patterns to work with its peculiarities (like if(strm.good()) not being the same as if(strm) and not being the opposite of if(strm.bad()) and a few other things you'll have to get used to). For something as performance-critical (har har!) as reading an ini file from disk, it should be fast enough in 999,999 out of 1,000,000 cases.
You may want to try Boost.Program_Options. However it has slightly different formatting. More close to INI files. Subcategories are done like this:
[CategoryA]
Option = Data
[CategoryB.Subcategory1]
Option = Data
[CategoryB.Subcategory2]
Option = Data
Also it has some other features so it is actually very useful IMO.
Try Configurator. It's easy-to-use and flexible C++ library for configuration file parsing (from simplest INI to complex files with arbitrary nesting and semantic checking). Header-only and cross-platform. Uses Boost C++ libraries.
See: http://opensource.dshevchenko.biz/configurator
It looks more straightforward to implement your own parser than to try to adapt an existing one you are unfamiliar with.
Your structure seems - from your example - to be line-based. This makes parsing it easy.
It generally makes sense to load your file into a tree, and then walk around it as necessary.
On Windows only, GetPrivateProfileSection does this. It's deprecated in favor of the registry but it's still here and it still works.
How about trying to make a simple XML file? There are plenty of libraries that can help you read it, and the added bonus is that a lot of other programs/languages can read it too.
If you're using wxWidgets I would consider wxFileConfig. I'm not using wxWidgets, but the class seems to support categories with sub-categories.
When you are using GTK, you are lucky.
You can use the Glib KeyFile save_to_file and load_from_file.
https://docs.gtk.org/glib/struct.KeyFile.html
Or when using Gtkmm (C++).
See: https://developer-old.gnome.org/glibmm/stable/classGlib_1_1KeyFile.html
Example in C++ with load_from_file:
#include <glibmm.h>
#include <string>
Glib::KeyFile keyfile;
keyfile.load_from_file(file_path);
std::string path = keyfile.get_string("General", "Path");
bool is_enabled = keyfile.get_boolean("General", "IsEnabled");
Saving is as easy as calling save_to_file:
Glib::KeyFile keyfile;
keyfile.set_string("General", "Path", path);
keyfile.set_boolean("General", "IsEnabled", is_enabled);
keyfile.save_to_file(file_path);