I'm working on what will be a pretty large and complicated system and trying to make sure it's as watertight as possible right from the start. Whilst running some memory checks, I noticed something odd when using stringstreams: they don't always seem to release all the memory when they get deleted/go out of scope.
I've tried searching the internet for answers, but most are old (so possibly out of date) and/or are more concerned with refreshing the contents than releasing the memory, so I've not really been able to tell if it's a known issue or a common mistake I'm making.
I've written a simple test to show what's going on:
#include <iostream>
#include <string>
#include <sstream>
#include <fstream>
using namespace std;
float getMemUsage(int& pid)
{
if (pid < 0)
pid = getpid();
char buf[30];
snprintf(buf, 30, "/proc/%u/statm", (unsigned)pid);
FILE* pf = fopen(buf, "r");
if (pf)
{
unsigned size; // total program size
//unsigned resident;// resident set size
//unsigned share;// shared pages
//unsigned text;// text (code)
//unsigned lib;// library
//unsigned data;// data/stack
//unsigned dt;// dirty pages (unused in Linux 2.6)
fscanf(pf, "%u" /* %u %u %u %u %u"*/, &size/*, &resident, &share, &text, &lib, &data*/);
fclose(pf);
return size/1024.0;
}
else
return -1.0;
}
int main(int argc, char* argv[])
{
if (argc < 2)
cerr << "no file specified\n";
ifstream file;
file.open(argv[1]);
int pid = -1;
const float memUseAtStart = getMemUsage(pid);
{
float memUseBefore = getMemUsage(pid);
stringstream sstream;
float memUseAfter = getMemUsage(pid);
cerr << "\tMemory use change after stringstream declaration: " << memUseAfter - memUseBefore << endl;
memUseBefore = getMemUsage(pid);
filebuf* pbuf = file.rdbuf();
memUseAfter = getMemUsage(pid);
cerr << "\tMemory use change after getting file buffer: " << memUseAfter - memUseBefore << endl;
memUseBefore = getMemUsage(pid);
sstream << pbuf;
memUseAfter = getMemUsage(pid);
cerr << "\tMemory use change after copying file contents: " << memUseAfter - memUseBefore << endl;
memUseBefore = getMemUsage(pid);
sstream.clear();
sstream.str( string() );
memUseAfter = getMemUsage(pid);
cerr << "\tMemory use change after 'clearing': " << memUseAfter - memUseBefore << endl;
}
cerr << "Overall memory use change: " << getMemUsage(pid) - memUseAtStart << endl;
file.close();
return 0;
}
Which gives me the following output when called with a file larger than around 32K:
Memory use change after stringstream declaration: 0
Memory use change after getting file buffer: 0
Memory use change after copying file contents: 0.0322266
Memory use change after 'clearing': 0
Overall memory use change: 0.00195312
I'm running on Linux (SL6.6) and compiling with gcc 4.1.2 (though I've also tried clang and ICC with similar results).
Obviously, it's not a huge leak; it's just a little annoying that I can't make it completely tidy... Is there something I can/should do to release the memory manually? Or is it just something weird (with my setup and/or stringstream itself) I'll have to live with?
NB The intended use for the stringstream is to read in some file contents above and then parse them line by line; I would try using istringstream but I couldn't figure out how to set its value from the ifstream...
Thanks!
Related
My program use a small SQLite3 database. To make sure it actually exist when the program is launched, I have a database creation script in a file, that is executed.
The script work without problem.
However, when using C++ I/O functions to read from that file, I am getting really often invalid characters at the end of my file, which result in the script containing errors and not being properly executed by the SQLite library. Here is an example when displaying the buffer content:
// Proper content from the file, then a random character is there
1
Error executing request: near "1": syntax error
Other characters also appear, whitespaces, numbers, letters...
Here is the code where I load my script :
std::cerr << "Creating database if needed...\n";
char sql_script[] = "/path/to/script.sql";
int script_length;
bool result = false;
std::ifstream script_fs(sql_script, std::fstream::binary | std::fstream::in);
if (script_fs) {
char* buffer;
char** err_msg = NULL;
script_fs.seekg(0, script_fs.end);
script_length = script_fs.tellg();
script_fs.seekg(0, script_fs.beg);
buffer = new char[script_length];
script_fs.read(buffer, script_length);
std::cout << "sql:\n" << buffer << "\n";
if (sqlite3_exec(m_db, buffer, NULL, NULL, err_msg) == SQLITE_OK){
result = true;
} else {
std::cerr << "Error executing: " << sqlite3_errmsg(m_db) << "\n" << err_msg << "\n";
}
delete buffer;
script_fs.close();
} else {
std::cerr << "Error opening script: " << strerror(errno) << "\n";
}
return result;
}
Why is this happening and how can I fix this ?
You need to make sure that you have a null-terminated string.
Allocate memory for one more character.
Assign the null character to the last element of buffer.
buffer = new char[script_length+1];
script_fs.read(buffer, script_length);
buffer[script_length] = '\0';
Also, use the array form of delete.
delete [] buffer;
Don't mix C and C++, If you want to read sql query file in C++ using ifstream then below code in C++ can be one approach in which you don't need to manage memory, take care of things like allocating one extra char of '\0' etc. :
#include <iostream>
#include <fstream>
#include <vector>
using namespace std;
int main() {
ifstream fin("test.sql", std::fstream::binary | std::fstream::in);
std::string sqlquery = std::string(std::istreambuf_iterator<char>(fin), std::istreambuf_iterator<char>());
std::cout<<sqlquery<<std::endl;
return 0;
}
I am reading binary file cmd.exe into unsigned chars array. Total bytes read into bytes_read are 153. I converted it to base64 string and then decode this string back (code from 2nd answer base64 decode snippet in c++) into vector<'BYTE>. Here BYTE is unsigned char.
decodedData.size() is also 153. But when I write this vector to file in binary mode to get my cmd.exe file again I get only 1 KB file. What thing I missed?
// Reading size of file
FILE * file = fopen("cmd.exe", "r+");
if (file == NULL) return 1;
fseek(file, 0, SEEK_END);
long int size = ftell(file);
fclose(file);
// Reading data to array of unsigned chars
file = fopen("cmd.exe", "r+");
unsigned char * myData = (unsigned char *)malloc(size);
int bytes_read = fread(myData, sizeof(unsigned char), size, file);
fclose(file);
std::string encodedData = base64_encode(&myData[0], bytes_read);
std::vector<BYTE> decodedData = base64_decode(encodedData);
////write data to file
ofstream outfile("cmd.exe", ios::out | ios::binary);
outfile.write((const char *)decodedData.data(), decodedData.size());
Update:
Thanks #chux for suggesting "r+" --> "rb+" Problem resolved.
You marked this as C++.
This is one C++ approach using fstream to read a binary file. To simplify for this example, I created a somewhat bigger m_buff than needed. From the comments, it sounds like your fopen("cmd.exe", "r+") was in error, so I'm only providing a C++ binary read.
Method tReader() a) opens a file in binary mode, b) reads the data into m_buff, and c) captures gCount for display.
It also demonstrates one possible use of chrono to measure duration.
#include <chrono>
// 'compressed' chrono access --------------vvvvvvv
typedef std::chrono::high_resolution_clock HRClk_t;
typedef HRClk_t::time_point Time_t;
typedef std::chrono::microseconds US_t;
using namespace std::chrono_literals; // suffixes 100ms, 2s, 30us
#include <iostream>
#include <fstream>
#include <cassert>
class T516_t
{
enum BuffConstraints : uint32_t {
Meg = (1024 * 1024),
END_BuffConstraints
};
char* m_buff;
int64_t m_gCount;
public:
T516_t()
: m_buff (nullptr)
, m_gCount (0)
{
m_buff = new char[Meg];
}
~T516_t() = default;
int exec()
{
tReader();
return(0);
}
private: // methods
void tReader()
{
std::string pfn = "/home/dmoen/.wine/drive_c/windows/system32/cmd.exe";
// open file in binary mode
std::ifstream sIn (pfn, std::ios_base::binary);
if (!sIn.is_open()) {
std::cerr << "UNREACHABLE: unable to open sIn " << pfn
<< " priviledges? media offline?";
return;
}
Time_t start_us = HRClk_t::now();
do
{
// perform read
sIn.read (m_buff, Meg);
// If the input sequence runs out of characters to extract (i.e., the
// end-of-file is reached) before n characters have been successfully
// read, buff contains all the characters read until that point, and
// both eofbit and failbit flags are set
m_gCount = sIn.gcount();
if(sIn.eof()) { break; } // exit when no more data
if(sIn.failbit ) {
std::cerr << "sIn.faileBit() set" << std::endl;
}
}while(1);
auto duration_us = std::chrono::duration_cast<US_t>(HRClk_t::now() - start_us);
sIn.close();
std::cout << "\n " << pfn
<< " " << m_gCount << " bytes"
<< " " << duration_us.count() << " us"
<< std::endl;
} // int64_t tReader()
}; // class T516_t
int main(int , char**)
{
Time_t start_us = HRClk_t::now();
int retVal = -1;
{
T516_t t516;
retVal = t516.exec();
}
auto duration_us = std::chrono::duration_cast<US_t>(HRClk_t::now() - start_us);
std::cout << " FINI " << duration_us.count() << " us" << std::endl;
return(retVal);
}
One typical output on my system looks like:
/home/dmoen/.wine/drive_c/windows/system32/cmd.exe 722260 bytes 1180 us
FINI 1417 us
Your results will vary.
Your ofstream use looks good (so did not replicate).
I am new at C/C++,
So basically I want to call an .exe file that displays 2 numbers and be able to grab those two numbers to use them in my code.
To call the .exe file I've used the system command, but I am still not able to grab those two numbers that are displayed by the .exe file
char *files = "MyPath\file.exe";
system (files);
I think this is better aproach:
Here you just create new process, and you read data that process gives you. I tested this on OS X 10.11 with .sh file and works like a charm. I think that this would probably work on Windows also.
FILE *fp = popen("path to exe","r");
if (fp == NULL)
{
std::cout << "Popen is null" << std::endl;
}else
{
char buff[100];
while ( fgets( buff, sizeof(buff), fp ) != NULL )
{
std::cout << buff;
}
}
You need to escapr back slashes in C++ string literals so:
// note the double "\\"
char* files = "MyPath\\file.exe";
Or just use forward slashes:
char* files = "MyPath/file.exe";
Its not very efficient but one thing you can to with std::system is redirect the output to a file and then read the file:
#include <cstdlib>
#include <fstream>
#include <iostream>
int main()
{
// redirect > the output to a file called output.txt
if(std::system("MyPath\\file.exe > output.txt") != 0)
{
std::cerr << "ERROR: calling system\n";
return 1; // error code
}
// open a file to the output data
std::ifstream ifs("output.txt");
if(!ifs.is_open())
{
std::cerr << "ERROR: opening output file\n";
return 1; // error code
}
int num1, num2;
if(!(ifs >> num1 >> num2))
{
std::cerr << "ERROR: reading numbers\n";
return 1; // error code
}
// do something with the numbers here
std::cout << "num1: " << num1 << '\n';
std::cout << "num2: " << num2 << '\n';
}
NOTE: (thnx #VermillionAzure)
Note that system doesn't always work everywhere because unicorn
environments. Also, shells can differ from each other, like cmd.exe
and bash. – VermillionAzure
When using std::system the results are platform dependant and not all shells will have redirection or use the same syntax or even exist!
I'm trying to make some experiments on disk I/O using cache and not using it. In order to perform a read directly from the disk, I open the file with the O_DIRECT flag (defining the variable DISK_DIRECT).
Now the two branches of the if beneath, should perform the same operation, with the difference that one is helped by the cache and the other not.
The files to which I try to access are stored on disk and they do not change over time.
Also the two branches access to the same files.
At some point here, when I use fread I get ferror() to be true. While when I use read everything goes fine.
I'm sure they access the same files.
Do you have any idea why this could happen?
EDIT
Ok, i'm posting here an minimal example. the code i use is:
#include <iostream>
#include <fcntl.h>
#include <sys/types.h>
#include <unistd.h>
#include <fstream>
#include <sstream>
using namespace std;
typedef float fftwf_complex [2] ;
void fetch_level(unsigned long long tid, unsigned short level, fftwf_complex* P_read, fftwf_complex* P_fread, int n_coeff_per_level, FILE** files_o_direct, fstream* & files) {
int b_read;
fseek(files_o_direct[level],(long int) (tid * sizeof(fftwf_complex)*n_coeff_per_level), SEEK_SET);
b_read = fread(reinterpret_cast<char*>(P_fread),sizeof(fftwf_complex), n_coeff_per_level,files_o_direct[level]);
if(b_read == 0){
cerr << "nothing read\n";
}
files[level].seekg((streamoff) (tid * sizeof(fftwf_complex)*n_coeff_per_level), files[level].beg);
files[level].read(reinterpret_cast<char*>(P_read),
sizeof(fftwf_complex) * n_coeff_per_level);
}
void open_files (fstream* & files){
for(int i=0; i<1;i++) {
std::ostringstream oss;
oss << "./Test_fread_read/1.txt.bin";
files[i].open(oss.str().c_str(),
std::ios::in | std::ios::out |
std::ios::binary | std::ios::ate);
if (!files[i])
{
cerr << "fstream could not open " << oss.str() << endl;
}
}
}
void open_files_o_direct (FILE** files_o_direct, int* fd){
for(unsigned int i=0;i<1; i++){
std::ostringstream oss;
oss << "./Test_fread_read/1.txt.bin";
fd[i]=open(oss.str().c_str(), O_RDONLY | O_DIRECT);
files_o_direct[i] = fdopen(fd[i], "rb");
if(!files_o_direct[i])
cerr << "Could not open " << oss.str() << endl;
}
}
int close_files(FILE** files_o_direct, int* fd, fstream* & files) {
for(unsigned int i=0; i<1; i++){
//#if defined (DISK_DIRECT)
if(files_o_direct[i])
close(fd[i]);
//#else
if(files[i].is_open())
files[i].close();
//#endif
}
return 0;
}
int main(){
FILE**files_o_direct = new FILE* [256];
fstream* files = new fstream [256];
int * fd = new int [256];
fftwf_complex * P_read = new fftwf_complex [1];
fftwf_complex * P_fread = new fftwf_complex [1];
open_files_o_direct(files_o_direct, fd);
open_files(files);
fetch_level(2, 0, P_read, P_fread, 1, files_o_direct, files);
cout << "P_read: " << P_read[0][0] << " P_fread: " << P_fread[0][0] << endl;
cout << "P_read: " << P_read[0][1] << " P_fread: " << P_fread[0][1] << endl;
fetch_level(7, 0, P_read, P_fread, 1, files_o_direct, files);
cout << "P_read: " << P_read[0][0] << " P_fread: " << P_fread[0][0] << endl;
cout << "P_read: " << P_read[0][1] << " P_fread: " << P_fread[0][1] << endl;
fetch_level(8, 0, P_read, P_fread, 1, files_o_direct, files);
cout << "P_read: " << P_read[0][0] << " P_fread: " << P_fread[0][0] << endl;
cout << "P_read: " << P_read[0][1] << " P_fread: " << P_fread[0][1] << endl;
close_files(files_o_direct, fd, files);
delete [] P_read;
delete [] P_fread;
delete [] files;
delete [] files_o_direct;
return 0;
}
and the file which is accessed is:
0.133919 0.0458176
1.67441 2.40805
0.997525 -0.279977
-2.39672 -3.076
-0.0390913 0.854464
-0.0176478 -1.3142
-0.667981 -0.486272
0.831051 0.282802
-0.638032 -0.630943
-0.669854 -1.49762
which is stored in a binary format and that can be download from here: 1.txt.bin.
The output i get is:
nothing read
P_read: 0.997525 P_fread: 0
P_read: -0.279977 P_fread: 0
nothing read
P_read: 0.831051 P_fread: 0
P_read: 0.282802 P_fread: 0
nothing read
P_read: -0.638032 P_fread: 0
P_read: -0.630943 P_fread: 0
The problem persists even if i change the type of fftwf_complex from float[2] to simple float.
If i remove the fseek line everything works correctly.
This if (b_read == 0), will be true at the end of the file, and you will enter this branch
if(ferror(this->files_o_direct[level]))
fseek(this->files_o_direct[level], 0, SEEK_END); //ftell here returns 4800000
cerr << "nothing read\n";
even if ferror returns 0, the end of the file was reached anyway
fseek(this->files_o_direct[level], 0, SEEK_END);
makes no sense, and "nothing read\n" will be output either or not ferror returns nonzero.
From the manual page
fread() does not distinguish between end-of-file and error, and callers must use feof(3) and ferror(3) to determine which occurred.
so you have to check feof and if it is false you use ferror.
For who ever may have the same problem here there is the answer:
The O_DIRECT flag may impose alignment restrictions on the length and
address of user-space buffers and the file offset of I/Os. In Linux
alignment restrictions vary by filesystem and kernel version and
might be absent entirely. However there is currently no
filesystem-independent interface for an application to discover these
restrictions for a given file or filesystem. Some filesystems
provide their own interfaces for doing so, for example the
XFS_IOC_DIOINFO operation in xfsctl(3).
Under Linux 2.4, transfer sizes, and the alignment of the user buffer
and the file offset must all be multiples of the logical block size
of the filesystem. Since Linux 2.6.0, alignment to the logical block
size of the underlying storage (typically 512 bytes) suffices. The
logical block size can be determined using the ioctl(2) BLKSSZGET
operation or from the shell using the command:
blockdev --getss
linux reference page
I want to some text to output to a file. I heard that it is better to stream the data rather than creating a large string and outputing that. Presently I am creating a large string and outputing to a file. Request to provide an sample code on how to stream a data and write to a file using C++.
Thanks!
#include <fstream>
int main()
{
std::ofstream fout("filename.txt");
fout << "Hello";
fout << 5;
fout << std::endl;
fout << "end";
}
Your file now contains this:
Hello5
end
See more info on std::ofstream for details.
HTH
File writing already uses buffering. If it is not efficient for you, you can actually modify the filebuf, eg increase its size or use a custom one.
Avoid doing unnecessary flushes of your buffer, which is done with endl. That is the most "abused" feature of file-writing.
The simplest way to create a file-stream for outputting is:
#include <fstream>
int main( int argc, char * argv[])
{
if( argc > 1 )
{
std::ofstream outputFile( argv[1] );
if( outputFile )
{
outputFile << 99 << '\t' << 158 << '\n'; // write some delimited numbers
std::vector< unsigned char > buf;
// write some data into buf
outputFile.write( &buf[0], buf.size() ); // write binary to the output stream
}
else
{
std::cerr << "Failure opening " << argv[1] << '\n';
return -1;
}
}
else
{
std::cerr << "Usage " << argv[0] << " <output file>\n";
return -2;
}
return 0;
}