I'm trying to use libzip in a program that needs to archive several data chunks in different files. At the moment I have a code similar to the following snippet, edited from in-memory.c example in libzip examples.
The zip file is correctly saved with the files inside, but each file contains garbage.
Any help is appreciated.
bool push_files(zip_t* za) {
for (int i = 0; i < 10; i++) {
// Generate data
std::stringstream ss;
ss << "Test file #" << i;
std::string a = ss.str();
zip_source_t* source = zip_source_buffer(za, a.c_str(), a.size(), 0);
if (source == NULL) {
std::cerr << "error creating source: " << zip_strerror(za) << std::endl;
return false;
}
// Add buffer with filename
std::stringstream fname;
fname << "TEST-" << i;
a = fname.str();
if (zip_file_add(za, a.c_str(), source, ZIP_FL_ENC_UTF_8) < 0) {
std::cerr << "error adding source: " << zip_strerror(za) << std::endl;
return false;
}
}
return true;
}
int main() {
zip_source_t* src;
zip_error_t error;
zip_t* za;
zip_error_init(&error);
if ((src = zip_source_buffer_create(NULL, 0, 1, &error)) == NULL) {
std::cerr << "can't create source: " << zip_error_strerror(&error) << std::endl;
zip_error_fini(&error);
return 1;
}
if ((za = zip_open_from_source(src, ZIP_TRUNCATE, &error)) == NULL) {
std::cerr << "can't open zip from source: " << zip_error_strerror(&error) << std::endl;
zip_source_free(src);
zip_error_fini(&error);
return 1;
}
zip_error_fini(&error);
zip_source_keep(src);
if (!push_files(za))
return -1;
if (zip_close(za) < 0) {
std::cerr << "can't close zip archive" << zip_strerror(za) << std::endl;
return 1;
}
// ... omissis, save archive to file as in in-memory.c
}
zip_source_buffer does not copy the data out of the buffer - it just creates a zip_source_t which points to the same buffer. So you must keep the buffer alive until you're done adding the file.
Your code does not keep the buffer alive. The buffer you use is a.c_str() which is the data buffer of the string a. Fair enough, so far. But then before adding the file, you reassign the variable a = fname.str(); which (probably) frees that buffer and allocates a new one.
Solution: use a separate variable for the filename. Don't overwrite a until the file has been added.
Related
I have an application that must monitor some folders (in Windows) to detect if a file was created in that folder (real use is to detect incoming FTP files).
If a file is detected , it is read, then deleted .
Occasionally, I get a file reading error on a file that was detected.
Question is: Why?
To simulate the error, I created a simple program to reproduce it:
std::vector<std::filesystem::path> watch;
void main()
{
watch.push_back("D:\\test1"); //must exist
watch.push_back("D:\\test2");
watch_dir();
}
this example monitors 2 folders.
To simulate incoming files on the folder, another program copies files to that folder
continuously at configurable intervals (say 100 milliseconds).
To detect folder changes , WIN32 API functions FindFirstChangeNotification and WaitForMultipleObjects are used, based on this Microsoft example
https://learn.microsoft.com/en-us/windows/win32/fileio/obtaining-directory-change-notifications
detection function adapted from the example (Note: WaitForMultipleObjects blocks until a change is detected)
void watch_dir()
{
HANDLE handle[2];
memset(handle, 0, 2 * sizeof(HANDLE));
for (size_t idx = 0; idx < watch.size(); idx++)
{
std::string str = watch.at(idx).string();
LPTSTR path = (LPTSTR)str.c_str();
std::cout << "watch path " << path << std::endl;
handle[idx] = FindFirstChangeNotification(
path, // directory to watch
FALSE, // do not watch subtree
FILE_NOTIFY_CHANGE_FILE_NAME); // watch file name changes
if (handle[idx] == INVALID_HANDLE_VALUE)
{
assert(0);
ExitProcess(GetLastError());
}
}
while (TRUE)
{
std::cout << "Waiting for notification..." << std::endl;
DWORD wait_status = WaitForMultipleObjects(watch.size(), handle, FALSE, INFINITE);
std::cout << "Directory " << watch.at(wait_status) << " changed" << std::endl;
if (FindNextChangeNotification(handle[wait_status]) == FALSE)
{
assert(0);
ExitProcess(GetLastError());
}
std::filesystem::path path = watch.at(wait_status);
send_files_in_path(path);
}
}
Once a change is detected by the function above, then all files in the folder are listed
and read, by these functions
void send_files_in_path(const std::filesystem::path& ftp_path)
{
std::vector<std::filesystem::path> list = get_files(ftp_path);
for (size_t idx = 0; idx < list.size(); idx++)
{
std::string buf;
read_file(list.at(idx).string(), buf);
std::this_thread::sleep_for(std::chrono::milliseconds(10));
std::filesystem::remove(list.at(idx));
}
}
/////////////////////////////////////////////////////////////////////////////////////////////////////
//get_files
//get all ".txt" files inside a FTP folder
/////////////////////////////////////////////////////////////////////////////////////////////////////
std::vector<std::filesystem::path> get_files(const std::filesystem::path& base_archive_path)
{
std::vector<std::filesystem::path> list;
try
{
for (const auto& entry : std::filesystem::recursive_directory_iterator(base_archive_path))
{
std::filesystem::path path = entry.path();
if (!entry.is_regular_file())
{
continue;
}
std::string fname = entry.path().filename().string();
size_t len = fname.size();
size_t pos = len - 4;
//check if last 4 characters are ".txt"
if (fname.find(".txt", pos) == std::string::npos && fname.find(".TXT", pos) == std::string::npos)
{
continue;
}
SPDLOG_INFO("loading: " + entry.path().string());
list.push_back(path);
}//this path
} //try
catch (const std::exception& e)
{
SPDLOG_ERROR(e.what());
}
return list;
}
The function where the error happens is
int read_file(const std::string& fname, std::string& buf)
{
std::ifstream ifs;
std::ios_base::iostate mask = ifs.exceptions() | std::ios::failbit;
ifs.exceptions(mask);
std::this_thread::sleep_for(std::chrono::milliseconds(0));
std::cout << "opening : " << fname << std::endl;
try
{
ifs.open(fname);
if (!ifs.is_open())
{
std::cout << "open fail: " << fname << std::endl;
return -1;
}
}
catch (const std::exception& e)
{
std::cout << e.what() << std::endl;
return -1;
}
std::stringstream ss;
ss << ifs.rdbuf();
ifs.close();
buf = ss.str();
return 0;
}
the try/catch block, again, occasionally , is triggered with the error
ios_base::failbit set: iostream stream error
removing the try/catch block, and the open mask (just to try), then
ifs.is_open
fails.
A temporary solution was to detect the cases where the open() failed and repeat it.. which succeeds, because the file does exist.
Calling this with a small delay before the open call has the effect of reducing the open fails
std::this_thread::sleep_for(std::chrono::milliseconds(10));
ifs.open(fname);
But still would like to find out the reason for the occasional failure
I'm having trouble with libarchive version 3.3.2. I wrote a program to read selected entries in 7z archives, that look like:
file.7z
|__ file.xml
|__ file.fog
|__ file_1.fog
However, the program failed to read file_1.fog for most of my archives, and failed to read file.fog for some. I tried to use archive_error_string() to see what happens, and the errors were either corrupted archive or truncated RAR archive or Decompressing internal error.
Here's the trouble code:
void list_archive(string name) {
struct archive *a;
struct archive_entry *entry;
// create new archive struct for the file
a = archive_read_new();
archive_read_support_filter_all(a);
archive_read_support_format_all(a);
// open 7z file
int r = archive_read_open_filename(a, name.c_str(), 1024);
if (r != ARCHIVE_OK) {
cout << "cannot read file: " << name << endl;
cout << "read error: " << archive_error_string(a) << endl;
}
// looping through entries
for (;;) {
int status = archive_read_next_header(a, &entry);
// if there's no more header
if (status != ARCHIVE_OK) break;
// print some status messages to stdout
string pathname(archive_entry_pathname(entry));
cout << "working on: " << pathname << endl;
size_t entry_size = archive_entry_size(entry);
// load the entry's content
char * content;
content = (char*)malloc(entry_size);
r = archive_read_data(a, content, entry_size);
// check if archive_read_data was successful
if (r > 0) {
cout << "read " << r << " of " << entry_size << " bytes successfully\n";
// we are interested in .fog file only
if (pathname.back() == 'g') {
// do something with the .fog file
}
}
else // usually the error happens here
if (archive_errno(a) != ARCHIVE_OK) cout << "read error: " << archive_error_string(a) << endl;
// free the content and clear the entry
archive_read_data_skip(a);
free(content);
archive_entry_clear(entry);
cout << "-----" << endl;
}
// we are done with the current archive, free it
r = archive_read_free(a);
if (r != ARCHIVE_OK) {
cout << "Failed to free archive object. Error: " << archive_error_string(a) << endl;
exit(1);
}
}
I found the troublemaker and answer here if future users have the same problem.
int r = archive_read_open_filename(a, name.c_str(), 1024);
Apparently 1024 is too small for a buffer size. I increased it to 102400 and was able to read/extract all my archives.
Be aware, technically buffer size should not break functionality, it's OK to reduce speed but it's not acceptable to break the operation, therefore I think the way it's processing archives is not that reliable.
Even though the text file to which I saved all the samples contains (possibly) proper samples, the sound file generated using the same set of data contains only the noise. The code responsible for writing the wav file:
void Filter::generateFrequencySound()
{
SNDFILE * outfile;
SF_INFO sfinfo;// = {0};
memset (&sfinfo, 0, sizeof (sfinfo)) ;
//preparing output file
sfinfo.format = SF_FORMAT_WAV | SF_FORMAT_PCM_16;
sfinfo.channels = 1;
sfinfo.samplerate = 44100;
std::cout << "Trying to save samples to a file" << std::endl;
const char* path = "FilterInFrequency.wav";
outfile = sf_open(path, SFM_WRITE, &sfinfo);
if(!(outfile))
{
std::cout << "Failed to create output file" << std::endl;
sf_perror(outfile);
return;
}
unsigned long savedSamples = sf_write_double( outfile,
outputOfFrequencyFiltration,
bufferSize);
if(savedSamples > bufferSize)
{
std::cout << "Failed to save all samples into outflie. Number of sampels " << savedSamples << std::endl;
sf_close(outfile);
return;
}
sf_close(outfile);
QSound::play("FilterInFrequency.wav");
}
The code responsible for writing samples into a text file:
QFile file("finalResult_1.txt");
if(!file.open(QIODevice::WriteOnly))
{
std::cout << "something went wrong";
exit(16);
}
QTextStream outstream(&file);
for(unsigned long i = 0; i < bufferSize; i++)
{
QString line = QString::number(outputOfFrequencyFiltration[i]);
outstream << line << "\n";
}
file.close();
Comparison of divergence between wav and plotted text file can be seen in the attached image. The plots have been created using the same amount of data (~20500 samples- ~10% of the output file). The file size is same for both plots.
What could be the possible reason for the differences?
textfile
wavfile
I have sought to discover the cause of unwanted trailing end-data in a file I am writing specific data into and do not believe I have made errors in writing to the file.
The output looks like:
building room_numbr capacity
packard | 101 | 500 |
painter | 514 | 10 |
ÿÿÿÿÿÿÿÿÿÿ | Attempt to seek file pointer error
The Attempt to seek file pointer error is normal as it represents a thrown exception when attempting to move the file pointer on an invalid stream. However, the ÿÿÿÿÿÿÿÿÿÿ is not normal nor expected in a fixed size file format using either 10 or 20 bytes to write data.
Create file here:
BinarySearchFile::BinarySearchFile(std::string file_name){
// concatenate extension to fileName
file_name += ".dat";
// form complete table data filename
data_file_name = file_name;
// create or reopen table data file for reading and writing
binary_search_file.open(data_file_name, std::ios::out | std::ios::in | std::ios::app);
if(!binary_search_file.is_open()){
binary_search_file.clear();
binary_search_file.open(data_file_name, std::ios::out);
binary_search_file.close();
binary_search_file.open(data_file_name, std::ios::out | std::ios::in | std::ios::app);
}
try{
if(binary_search_file.fail()){
throw CustomException("Unspecified table data file error");
}
}
catch (CustomException &custom_exception){ // Using custom exception class
std::cout << custom_exception.what() << std::endl;
return;
}
}
Write data to file
void BinarySearchFile::writeT(std::string attribute){
try{
if(binary_search_file){
for(auto start = attribute.begin(); start != attribute.end(); ++start){
binary_search_file.put(' ');
binary_search_file.put(*start);
}
binary_search_file.flush();
/*
attribute.resize(attribute.length() * 2);
const char *write_this = attribute.data();
binary_search_file.write(write_this, attribute.length());
*/
}else if(binary_search_file.fail()){
throw CustomException("Attempt to write attribute error");
}
}
catch(CustomException &custom_exception){ // Using custom exception class
std::cout << custom_exception.what() << std::endl;
return;
}
}
Read data file here:
std::string BinarySearchFile::readT(long file_pointer_location, long size_of_data)
{
try{
if(binary_search_file){
std::string data = "";
binary_search_file.seekp(file_pointer_location);
binary_search_file.seekg(file_pointer_location);
while (size_of_data > 0 ){
binary_search_file.get();
data += binary_search_file.get();
size_of_data -= 2;
}
/*
char data[20];
binary_search_file.seekp(filePointerLocation);
binary_search_file.seekg(filePointerLocation);
binary_search_file.read(data, sizeOfData);
*/
return data;
}else if(binary_search_file.fail()){
throw CustomException("Attempt to read attribute error");
}
}
catch(CustomException &custom_exception){ // Using custom exception class
std::cout << custom_exception.what() << std::endl;
}
}
Code that reads the file and prints the result to the screen:
while(true){
//reinitialize the catalog pointer to the beginning
catalog->setPointerBegin();
//display data
do{
if (boost::iequals((domain = catalog->getAttributeDomain()), "string")){
if(dataFile->binary_search_file_status()){
std::cout << dataFile->read_data(filePointer, 20) << " | ";
if (!writer_.fail())
writer_ << dataFile->read_data(filePointer, 20) << " | ";
}
else{
std::cout << "\n";
if (!writer_.fail())
writer_ << "\n";
return true;
}
// update the file pointer
filePointer += 20;
dataFile->set_file_pointer(filePointer);
}
else{
if(dataFile->binary_search_file_status()){
std::cout << dataFile->read_data(filePointer, 10);
if (!writer_.fail())
writer_ << dataFile->read_data(filePointer, 10);
for(int i = 0; i < 5; i++){
std::cout << " ";
if (!writer_.fail())
writer_ << " ";
}
std::cout << " | ";
if (!writer_.fail()){
writer_ << " | ";
}
}
else{
std::cout << "\n";
if (!writer_.fail()){
writer_ << "\n";
}
return true;
}
// update the file pointer
filePointer += 10;
}
} while(catalog->traverseForward() != nullptr);
std::cout << "\n";
if (!writer_.fail())
writer_ << "\n";
}
}
std::ifstream::get returns std::char_traits<char>::eof on failure, which usually has the int value -1. If you interpret that blindly as a valid character and cast to char, you will get '\xff', which in ISO-8859-15 is ÿ.
You should be checking for eof and/or eofbit when you read characters from the file, and especially after seeking.
I have a problem where I try to compress a file's data. Everything works up to the compression call, but it isn't the compression call itself, as the segfault is thrown before it. Showing my code will make it much clearer:
std::cout << "FILENAME: ";
std::cin >> filename;
if(!fileExists(filename))
{
std::cout << "ERR: FILE NOT FOUND." << std::endl;
continue;
}
std::cout << "Compressing file data...";
writeFile(filename, zlib_compress(readFile(filename)));
std::cout << " Done." << std::endl;
At the function zlib_compress...
std::string zlib_compress(const std::string& str)
{
std::cout << "DEBUG" << std::endl;
z_stream zs; // z_stream is zlib's control structure
memset(&zs, 0, sizeof(zs));
if (deflateInit(&zs, 9) != Z_OK)
std::cout << "deflateInit failed while compressing." << std::endl;
zs.next_in = (Bytef*)str.data();
zs.avail_in = str.size(); // set the z_stream's input
int ret;
char outbuffer[1073741824];
std::string outstring;
// retrieve the compressed bytes blockwise
do
{
zs.next_out = reinterpret_cast<Bytef*>(outbuffer);
zs.avail_out = sizeof(outbuffer);
ret = deflate(&zs, Z_FINISH);
if (outstring.size() < zs.total_out)
{
// append the block to the output string
outstring.append(outbuffer, zs.total_out - outstring.size());
}
} while(ret == Z_OK);
deflateEnd(&zs);
if(ret != Z_STREAM_END) // an error occurred that was not EOF
{
std::ostringstream oss;
oss << "Exception during zlib compression: (" << ret << ") " << zs.msg;
std::cout << oss.str();
}
return outstring;
}
I know, I know, that function needs work, I just C&P'd from somewhere to try it out.
But the thing is this:
std::cout << "DEBUG" << std::endl; is never called. The compiler says that the seg fault is coming from here:
std::string zlib_compress(const std::string& str)
> {
But why...? It was working earlier. I just don't know what went wrong!
Edit: Debugger output.
#0 00000000 0x00402cbb in __chkstk_ms() (??:??)
#1 004013BE zlib_compress(str=...) (C:\Users\***\Documents\Work\Programming\Compressor\z.cpp:5)
#2 00401DDA _fu15___ZSt4cout() (C:\Users\***\Documents\Work\Programming\Compressor\main.cpp:80)
char outbuffer[1073741824];
That's too large to put on the stack
You are taking a constant reference to a string as a parameter in your zlib_compress - you need to make sure that memory is available (whatever is returned from your readfile) in your zlib_compress. It would be good if you can share the prototype of your readFile function too.