I wrote this code:
ifstream f("file.txt");
char c;
std::string buffer;
buffer.reserve(1024);
bool flag = true;
while (flag) {
for (int i = 0; i < 1024; ++i) {
if (f.get(c))
buffer += c;
else
flag = false;
}
// do something with buffer
buffer.clear();
}
I need exactly 1 KB string buffer. Is there any better and efficient way to do this? Maybe some fstream or string functions which I don't know?
You don't need the for loop, you can use std::istream::read() instead. And if possible, replace the std::string with char[], eg:
ifstream f("file.txt");
char buffer[1024];
while (f.read(buffer, 1024))
{
// do something with buffer up to f.gcount() chars...
}
If, for some reason, you actually needed a std::string, then you can do this:
ifstream f("file.txt");
std::string buffer(1024);
while (f.read(&buffer[0], 1024)) // or buffer.data() in C++17 and later
{
buffer.resize(f.gcount());
// do something with buffer ...
buffer.resize(1024);
}
Related
I am trying to write a char* to a binary file.
This is what I have now.
void Write(char* fileName, char* pData)
{
ofstream binFile (fileName, ios::out | ios::binary);
if (binFile.open())
{
binFile.write((char*)&pData, sizeof(pData));
binFile.close();
}
}
void Read(char* fileName, char* pData)
{
ifstream binFile(fileName, ios::in | ios::binary);
if(binFile.open())
{
binFile.read(char*)&pData, sizeof(pData));
binFile.close
}
}
int main()
{
char* testData = "ABCdEFG"; // not real data
char* getTestData;
char* file = "C:\\testData.dat";
Write(file, testData);
Read(file, getTestData);
}
Test data will be of unknown length. May not always be the same.
When i run the program once, and write and read. I can get back the test data.
But when i stop the program and run it again, this time without writing. Just reading, i cannot get back the test data.
I don't really understand whats happening here.
Can some one explain it to me?
binFile.write((char*)&pData, sizeof(pData));
is wrong. It just writes the value of the pointer. It does not write the data.
You need to use:
binFile.write(pData, strlen(pData));
However, that won't be adequate to read the data back. To be able to read the data back, you'll need to write the size of the string first.
size_t len = strlen(pData);
binFile.write((char*)&len, sizeof(len));
binFile.write(pData, len);
And when reading the data back, you will need to use:
size_t len = 0;
binFile.read(char*)&len, sizeof(len));
binFile.read(pData, len);
and then, null terminate the string.
pData[len] = '\0';
PS
Make sure getTestData is properly initialized before using it to read the data.
char getTestData[100];
will be adequate for your test case.
Update
You can make your program a bit better by using std::string instead of char*. The size of the saved data can be more easily managed when a std::string is used.
void Write(std::string const& fileName, std::string const& data)
{
std::ofstream binFile(fileName, std::ios::out | std::ios::binary);
if (binFile.is_open())
{
size_t len = data.size();
binFile.write((char*)&len, sizeof(len));
binFile.write((char*)&data[0], len);
// No need. The file will be closed when the function returns.
// binFile.close();
}
}
void Read(std::string const& fileName, std::string& data)
{
std::ifstream binFile(fileName, std::ios::in | std::ios::binary);
if(binFile.is_open())
{
size_t len = 0;
binFile.read((char*)&len, sizeof(len));
data.resize(len);
binFile.read((char*)&data[0], len);
}
}
int main()
{
std::string file = "testData.dat";
std::string testData = "ABCdEFG";
Write(file, testData);
std::string getTestData;
Read(file, getTestData);
std::cout << getTestData << std::endl;
}
I was reading sehe's answer for fast text file reading in C++, which looks like this.
static uintmax_t wc(char const *fname)
{
static const auto BUFFER_SIZE = 16*1024;
int fd = open(fname, O_RDONLY);
if(fd == -1)
handle_error("open");
/* Advise the kernel of our access pattern. */
posix_fadvise(fd, 0, 0, 1); // FDADVICE_SEQUENTIAL
char buf[BUFFER_SIZE + 1];
uintmax_t lines = 0;
while(size_t bytes_read = read(fd, buf, BUFFER_SIZE))
{
if(bytes_read == (size_t)-1)
handle_error("read failed");
if (!bytes_read)
break;
for(char *p = buf; (p = (char*) memchr(p, '\n', (buf + bytes_read) - p)); ++p)
++lines;
}
return lines;
}
This is cool, but I was wondering if a similar approach can be taken when we aren't dealing with a character operation like counting newlines, but want to operate on each line of data. Say for instance I had a file of doubles, and already some function parse_line_to_double to use on each line.
12.44243
4242.910
...
That is, how can I read BUFFER_SIZE bytes into my buffer but avoid splitting the last line read? Effectively, can I ask "Give me BUFFER_SIZE or less bytes while ensuring that the last byte read is a newline character (or EOF)"?
Knowing extremely little about low level IO like this, ideas that came to mind were
Can I "back up" fd to the most recent newline between iterations?
Do I have to keep a second buffer holding a copy of the current line being read all the time?
Here is a comparison test. First, lets try the easy way. Just read the file with standard C++ functions:
#include <iostream>
#include <string>
#include <fstream> //std::ifstream
#include <sstream> //std::stringstream
uintmax_t test1(char const *fname)
{
std::ifstream fin(fname);
if(!fin) return 0;
uintmax_t lines = 0;
std::string str;
double value;
while(fin >> value)
{
//std::cout << value << "\n";
lines++;
}
return lines;
}
Next, with std::stringstream this is about 2.5 times faster:
uintmax_t test2(char const *fname)
{
std::ifstream fin(fname);
if(!fin) return 0;
uintmax_t lines = 0;
std::string str;
double value;
std::stringstream ss;
ss << fin.rdbuf();
while(ss >> value)
lines++;
return lines;
}
Next, lets read the whole file in to memory. This will be fine as long as the file is less than 1 GiB or so. Assuming there is a double value on each line, lets extract that value. test3 is more complicated and less flexible, and it's not any faster than test2:
uintmax_t test3(char const *fname)
{
std::ifstream fin(fname, std::ios::binary);
if(!fin) return 0;
fin.seekg(0, std::ios::end);
size_t filesize = (size_t)fin.tellg();
fin.seekg(0);
std::string str(filesize, 0);
fin.read(&str[0], filesize);
double value;
uintmax_t lines = 0;
size_t beg = 0;
size_t i;
size_t len = str.size();
for(i = 0; i < len; i++)
{
if(str[i] == '\n' || i == len - 1)
{
try
{
value = std::stod(str.substr(beg, i - beg));
//std::cout << value << "\n";
beg = i + 1;
lines++;
}
catch(...)
{
}
}
}
return lines;
}
For comparison to the wc function in the question, let's read the whole file in to memory and only count the number of lines. This runs a little faster than wc (as expected), suggesting that there is no need for additional optimizations
uintmax_t test_countlines(char const *fname)
{
std::ifstream fin(fname, std::ios::binary);
if(!fin) return 0;
fin.seekg(0, std::ios::end);
size_t filesize = (size_t)fin.tellg();
fin.seekg(0);
std::string str(filesize, 0);
fin.read(&str[0], filesize);
uintmax_t lines = 0;
for(auto &c : str)
if(c == '\n')
lines++;
return lines;
}
I have a problem, mostly because I can't really understand how to handle the situation.
I have a char* buffer of X size, it's the content of an encrypted file who's got decrypted and will be then parsed by the ifstream handler class that i can't edit.
So my idea was to create an fstream object in which, with rdbuf() assign the buffer with sputn.
fstream _handle2;
_handle2.rdbuf()->sputn(_buffer, _size);
_handle2.flush();
But of course, it's not working and buffer does not get written into the fstream object, do you have any idea of how to make it so?
I tried different methods but I clearly can't figure out what to do.
I'm trying to create a buffer type that can be parsed similarly to a ifstream.
You might try something like this (adopted from the link I provided in the comment already):
std::ifstream ifs("test.txt", std::ifstream::binary);
if (ifs)
{
ifs.seekg (0, ifs.end);
int length = ifs.tellg();
ifs.seekg (0, ifs.beg);
std::string buffer;
buffer.resize(length);
ifs.read(const_cast<char*>(buffer.data()), length);
if (ifs)
{
// de-crypt the buffer here!
// something like:
// buffer[i] = decryptChar(buffer[i]);
std::istringstream iss(buffer);
// now you can use iss just the same way as ifs,
// if the file was not encrypted...
}
else
{
std::cout << "error: only " << ifs.gcount() << " bytes could be read";
}
ifs.close();
}
Edit in response to your comment:
std::istringstream is used to convert text into binary data, e. g. int n; iss >> n; would convert the string "20102012", represented by ascii sequence 0x32, 0x30, 0x31, 0x30, 0x32, 0x30, 0x31, 0x32 into the corresponding four-byte integer value of 0x0132bb7c). But if data is already binary, std::istringstream is not suitable. Then you might rather try to write your own stream class similar to this example:
class DecryptionStream
{
std::unique_ptr<char> mBuffer;
char* mEnd;
char* mPos;
unsigned int flags;
unsigned int const eofbit = 1 << 0;
unsigned int const failbit = 1 << 1;
// other flag bits as needed
public:
// fail/eof bits as needed
DecryptionStream(char const* fileName) : mPos(nullptr)
{
std::ifstream ifs(fileName, std::ifstream::binary);
if (ifs)
{
ifs.seekg (0, ifs.end);
int length = ifs.tellg();
ifs.seekg (0, ifs.beg);
mBuffer.reset(new char[length]);
ifs.read(mBuffer.get(), length);
if (ifs)
{
// de-crypt the buffer here!
// something like:
// buffer[i] = decryptChar(buffer[i]);
mPos = mBuffer.get();
mEnd = mBuffer.get() + length;
}
else
{
flags |= failbit;
}
ifs.close();
}
}
template<typename T>
DecryptionStream& operator >>(T& t)
{
// fail, if any fail bit set already
size_t avail = mPos - mEnd;
if (avail < sizeof(t))
{
flags |= eofbit | failbit;
}
else
{
if(avail == sizeof(t))
{
flags |= eofbit;
}
memcpy(&t, mPos, sizeof(t));
mPos += sizeof(t);
}
return *this;
}
operator bool()
{
return flags == 0;
}
};
You could even use this class with complex data types - then make sure, though, that you control byte alignment of these appropriately, otherwise you might fail badly!
Have a look at boost::iostreams::array_source.
It allows you to treat an array as a std::istream. The advantage over std::istringstream is that the array will not be copied into the stream, which reduces memory usage and increases performance. The array_source will just store a pointer to the existing buffer.
#include <iostream>
#include <string>
#include <boost/iostreams/device/array.hpp>
#include <boost/iostreams/stream.hpp>
namespace io = boost::iostreams;
int main()
{
// Create an array and wrap a stream interface around it.
const char buffer[] = "hello stackoverflow";
io::stream< io::array_source > strm( buffer, sizeof(buffer) - 1 ); //-1 to strip '\0'
// Use the stream like a standard istream.
std::string s;
while( strm >> s )
std::cout << s << "\n";
}
Live Demo on Coliru.
I am trying to write a char* to a binary file.
This is what I have now.
void Write(char* fileName, char* pData)
{
ofstream binFile (fileName, ios::out | ios::binary);
if (binFile.open())
{
binFile.write((char*)&pData, sizeof(pData));
binFile.close();
}
}
void Read(char* fileName, char* pData)
{
ifstream binFile(fileName, ios::in | ios::binary);
if(binFile.open())
{
binFile.read(char*)&pData, sizeof(pData));
binFile.close
}
}
int main()
{
char* testData = "ABCdEFG"; // not real data
char* getTestData;
char* file = "C:\\testData.dat";
Write(file, testData);
Read(file, getTestData);
}
Test data will be of unknown length. May not always be the same.
When i run the program once, and write and read. I can get back the test data.
But when i stop the program and run it again, this time without writing. Just reading, i cannot get back the test data.
I don't really understand whats happening here.
Can some one explain it to me?
binFile.write((char*)&pData, sizeof(pData));
is wrong. It just writes the value of the pointer. It does not write the data.
You need to use:
binFile.write(pData, strlen(pData));
However, that won't be adequate to read the data back. To be able to read the data back, you'll need to write the size of the string first.
size_t len = strlen(pData);
binFile.write((char*)&len, sizeof(len));
binFile.write(pData, len);
And when reading the data back, you will need to use:
size_t len = 0;
binFile.read(char*)&len, sizeof(len));
binFile.read(pData, len);
and then, null terminate the string.
pData[len] = '\0';
PS
Make sure getTestData is properly initialized before using it to read the data.
char getTestData[100];
will be adequate for your test case.
Update
You can make your program a bit better by using std::string instead of char*. The size of the saved data can be more easily managed when a std::string is used.
void Write(std::string const& fileName, std::string const& data)
{
std::ofstream binFile(fileName, std::ios::out | std::ios::binary);
if (binFile.is_open())
{
size_t len = data.size();
binFile.write((char*)&len, sizeof(len));
binFile.write((char*)&data[0], len);
// No need. The file will be closed when the function returns.
// binFile.close();
}
}
void Read(std::string const& fileName, std::string& data)
{
std::ifstream binFile(fileName, std::ios::in | std::ios::binary);
if(binFile.is_open())
{
size_t len = 0;
binFile.read((char*)&len, sizeof(len));
data.resize(len);
binFile.read((char*)&data[0], len);
}
}
int main()
{
std::string file = "testData.dat";
std::string testData = "ABCdEFG";
Write(file, testData);
std::string getTestData;
Read(file, getTestData);
std::cout << getTestData << std::endl;
}
I'm trying to read a binary file and store it in a buffer. The problem is, that in the binary file are multiple null-terminated characters, but they are not at the end, instead they are before other binary text, so if I store the text after the '\0' it just deletes it in the buffer.
Example:
char * a = "this is a\0 test";
cout << a;
This will just output: this is a
here's my real code:
this function reads one character
bool CStream::Read (int * _OutChar)
{
if (!bInitialized)
return false;
int iReturn = 0;
*_OutChar = fgetc (pFile);
if (*_OutChar == EOF)
return false;
return true;
}
And this is how I use it:
char * SendData = new char[4096 + 1];
for (i = 0; i < 4096; i++)
{
if (Stream.Read (&iChar))
SendData[i] = iChar;
else
break;
}
I just want to mention that there is a standard way to read from a binary file into a buffer.
Using <cstdio>:
char buffer[BUFFERSIZE];
FILE * filp = fopen("filename.bin", "rb");
int bytes_read = fread(buffer, sizeof(char), BUFFERSIZE, filp);
Using <fstream>:
std::ifstream fin("filename.bin", ios::in | ios::binary );
fin.read(buffer, BUFFERSIZE);
What you do with the buffer afterwards is all up to you of course.
Edit: Full example using <cstdio>
#include <cstdio>
const int BUFFERSIZE = 4096;
int main() {
const char * fname = "filename.bin";
FILE* filp = fopen(fname, "rb" );
if (!filp) { printf("Error: could not open file %s\n", fname); return -1; }
char * buffer = new char[BUFFERSIZE];
while ( (int bytes = fread(buffer, sizeof(char), BUFFERSIZE, filp)) > 0 ) {
// Do something with the bytes, first elements of buffer.
// For example, reversing the data and forget about it afterwards!
for (char *beg = buffer, *end=buffer + bytes; beg < end; beg++, end-- ) {
swap(*beg, *end);
}
}
// Done and close.
fclose(filp);
return 0;
}
static std::vector<unsigned char> read_binary_file (const std::string filename)
{
// binary mode is only for switching off newline translation
std::ifstream file(filename, std::ios::binary);
file.unsetf(std::ios::skipws);
std::streampos file_size;
file.seekg(0, std::ios::end);
file_size = file.tellg();
file.seekg(0, std::ios::beg);
std::vector<unsigned char> vec;
vec.reserve(file_size);
vec.insert(vec.begin(),
std::istream_iterator<unsigned char>(file),
std::istream_iterator<unsigned char>());
return (vec);
}
and then
auto vec = read_binary_file(filename);
auto src = (char*) new char[vec.size()];
std::copy(vec.begin(), vec.end(), src);
The problem is definitievely the writing of your buffer, because you read a byte at a time.
If you know the length of the data in your buffer, you could force cout to go on:
char *bf = "Hello\0 world";
cout << bf << endl;
cout << string(bf, 12) << endl;
This should give the following output:
Hello
Hello world
However this is a workaround, as cout is foreseent to output printable data. Be aware that the output of non printable chars such as '\0' is system dependent.
Alternative solutions:
But if you manipulate binary data, you should define ad-hoc data structures and printing. Here some hints, with a quick draft for the general principles:
struct Mybuff { // special strtucture to manage buffers of binary data
static const int maxsz = 512;
int size;
char buffer[maxsz];
void set(char *src, int sz) // binary copy of data of a given length
{ size = sz; memcpy(buffer, src, max(sz, maxsz)); }
} ;
Then you could overload the output operator function:
ostream& operator<< (ostream& os, Mybuff &b)
{
for (int i = 0; i < b.size; i++)
os.put(isprint(b.buffer[i]) ? b.buffer[i]:'*'); // non printables replaced with *
return os;
}
ANd you could use it like this:
char *bf = "Hello\0 world";
Mybuff my;
my.set(bf, 13); // physical copy of memory
cout << my << endl; // special output
I believe your problem is not in reading the data, but rather in how you try to print it.
char * a = "this is a\0 test";
cout << a;
This example you show us prints a C-string. Since C-string is a sequence of chars ended by '\0', the printing function stops at the first null char.
This is because you need to know where the string ends either by using special terminating character (like '\0' here) or knowing its length.
So, to print whole data, you must know the length of it and use a loop similar to the one you use for reading it.
Are you on Windows? If so you need to execute _setmode(_fileno(stdout), _O_BINARY);
Include <fcntl.h> and <io.h>