So, here's my problem: I want to make a program that reads chunks of data from a file. Let's say, 1024 bytes per chunk.
So I read the first 1024 bytes, perform various operations and then open the next 1024 bytes, without reading the old data. The program should keep reading data untile the EOF is reached.
I'm currently using this code:
std::fstream fin("C:\\file.txt");
vector<char> buffer (1024,0); //reads only the first 1024 bytes
fin.read(&buffer[0], buffer.size());
But how can I read the next 1024 bytes? I was thinking by using a for loop, but I don't really know how. I'm totally a noob in C++, so if anyone can help me out, that would be great. Thanks!
You can do this with a loop:
std::ifstream fin("C:\\file.txt", std::ifstream::binary);
std::vector<char> buffer (1024,0); //reads only the first 1024 bytes
while(!fin.eof()) {
fin.read(buffer.data(), buffer.size())
std::streamsize s=fin.gcount();
///do with buffer
}
##EDITED
http://en.cppreference.com/w/cpp/io/basic_istream/read
Accepted answer doesn't work for me - it doesn't read last partial chunk. This does:
void readFile(std::istream &input, UncompressedHandler &handler) {
std::vector<char> buffer (1024,0); //reads only 1024 bytes at a time
while (!input.eof()) {
input.read(buffer.data(), buffer.size());
std::streamsize dataSize = input.gcount();
handler({buffer.begin(), buffer.begin() + dataSize});
}
}
Here UncompressedHandler accepts std::string, so I use constructor from two iterators.
I think you missed up that there is a pointer points to the last place you've visit in the file , so that when you read for the second time you will not start from the first , but from the last point you've visit .
Have a look to this code
std::ifstream fin("C:\\file.txt");
char buffer[1024]; //I prefer array more than vector for such implementation
fin.read(buffer,sizeof(buffer));//first read get the first 1024 byte
fin.read(buffer,sizeof(buffer));//second read get the second 1024 byte
so that how you may think about this concept .
I think that will work
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <fstream>
// Buffer size 16 Megabyte (or any number you like)
size_t buffer_size = 1 << 24; // 20 is 1 Megabyte
char* buffer = new char[buffer_size];
std::streampos fsize = 0;
std::ifstream file("c:\\file.bin", std::ios::binary);
fsize = file.tellg();
file.seekg(0, std::ios::end);
fsize = file.tellg() - fsize;
int loops = fsize / buffer_size;
int lastChunk = fsize % buffer_size;
for (int i = 0; i < loops; i++) {
file.read(buffer, buffer_size);
// DO what needs with the buffer
}
if (lastChunk > 0) {
file.read(buffer, lastChunk);
// DO what needs with the buffer
}
delete[] buffer;
Related
Can somebody tell if this is correct?
I try to read from binary file line by line and store it in a buffer? does the new line that it stores in the buffer delete the previous stored line?
ifs.open(filename, std::ios::binary);
for (std::string line; getline(ifs, line,' '); )
{
ifs.read(reinterpret_cast<char *> (buffer), 3*h*w);
}
For some reason you are mixing getline which is text-based reading, and read(), which is binary reading.
Also, it's completely unclear, what is buffer and what's it size. So, here is a simple example for you to start:
ifs.open(filename, std::ios::binary); // assume, that everything is OK
constexpr size_t bufSize = 256;
char buffer[bufSize];
size_t charsRead{ 0 };
do {
charsRead = ifs.read(buffer, bufSize)
// check if charsRead == 0, if it's ok
// do something with filled buffer.
// Note, that last read will have less than bufSize characters,
// So, query charsRead each time.
} while (charsRead == bufSize);
I am Trying to read 64000 bytes from file in binary mode in buffer at one time till end of the file. My problem is tellg() returns position in hexadecimal value, How do I make it return decimal value?
because my if conditions are not working, it is reading more than 64000 and when I am relocating my pos and size_stream(size_stream = size_stream - 63999;
pos = pos + 63999;), it is pointing to wrong positions each time.
How do I read 64000 bytes from file into buffer in binary mode at once till the end of file?
Any help would be appreciated
std::fstream fin(file, std::ios::in | std::ios::binary | std::ios::ate);
if (fin.good())
{
fin.seekg(0, fin.end);
int size_stream = (unsigned int)fin.tellg(); fin.seekg(0, fin.beg);
int pos = (unsigned int)fin.tellg();
//........................<sending the file in blocks
while (true)
{
if (size_stream > 64000)
{
fin.read(buf, 63999);
buf[64000] = '\0';
CString strText(buf);
SendFileContent(userKey,
(LPCTSTR)strText);
size_stream = size_stream - 63999;
pos = pos + 63999;
fin.seekg(pos, std::ios::beg);
}
else
{
fin.read(buf, size_stream);
buf[size_stream] = '\0';
CString strText(buf);
SendFileContent(userKey,
(LPCTSTR)strText); break;
}
}
My problem is tellg() returns position in hexadecimal value
No, it doesn't. It returns an integer value. You can display the value in hex, but it is not returned in hex.
when I am relocating my pos and size_stream(size_stream = size_stream - 63999; pos = pos + 63999;), it is pointing to wrong positions each time.
You shouldn't be seeking in the first place. After performing a read, leave the file position where it is. The next read will pick up where the previous read left off.
How do I read 64000 bytes from file into buffer in binary mode at once till the end of file?
Do something more like this instead:
std::ifstream fin(file, std::ios::binary);
if (fin)
{
unsigned char buf[64000];
std::streamsize numRead;
do
{
numRead = fin.readsome(buf, 64000);
if ((!fin) || (numRead < 1)) break;
// DO NOT send binary data using `LPTSTR` string conversions.
// Binary data needs to be sent *as-is* instead.
//
SendFileContent(userKey, buf, numRead);
}
while (true);
}
Or this:
std::ifstream fin(file, std::ios::binary);
if (fin)
{
unsigned char buf[64000];
std::streamsize numRead;
do
{
if (!fin.read(buf, 64000))
{
if (!fin.eof()) break;
}
numRead = fin.gcount();
if (numRead < 1) break;
// DO NOT send binary data using `LPTSTR` string conversions.
// Binary data needs to be sent *as-is* instead.
//
SendFileContent(userKey, buf, numRead);
}
while (true);
}
I'm using ifstream::read to read a file,
ifstream ifs("a.txt");
char buf[1024];
ifs.read(buf, 1024);
But a.txt's size might be less than 1000 bytes, so how am I supposed to know how many bytes have been read from ifs?
You can get the amount of characters extracted by the last operation with std::ifstream::gcount:
ifstream ifs("a.txt");
char buf[1024];
ifs.read(buf, 1024);
size_t extracted = ifs.gcount();
or
ifstream ifs("a.txt");
char buf[1024];
size_t extracted = ifs.read(buf, 1024).gcount();
since read(...) returns *this.
a follow up to my previous question (Reading an entire file in binary mode using C++)
After reading a jpg file in binary mode, the result of the read operation is always 4 bytes. The code is:
FILE *fd = fopen("c:\\Temp\\img.jpg", "rb");
if(fd == NULL) {
cerr << "Error opening file\n";
return;
}
fseek(fd, 0, SEEK_END);
long fileSize = ftell(fd);
int *stream = (int *)malloc(fileSize);
fseek(fd, 0, SEEK_SET);
int bytes_read = fread(stream, fileSize, 1, fd);
printf("%x\n", *stream);
fclose(fd);
The second last printf statement is always printing the first 4 bytes and not the entire file contents. How can I print the entire content of the jpg file?
Thanks.
You want it in C++? This opens a file, reads the entire contents into an array and prints the output to the screen:
#include <fstream>
#include <vector>
#include <iostream>
#include <algorithm>
using namespace std;
void hexdump(void *ptr, int buflen)
{
unsigned char *buf = (unsigned char*)ptr;
int i, j;
for (i=0; i<buflen; i+=16) {
printf("%06x: ", i);
for (j=0; j<16; j++) {
if (i+j < buflen)
printf("%02x ", buf[i+j]);
else
printf(" ");
}
printf(" ");
for (j=0; j<16; j++) {
if (i+j < buflen)
printf("%c", isprint(buf[i+j]) ? buf[i+j] : '.');
}
printf("\n");
}
}
int main()
{
ifstream in;
in.open("C:\\ISO\\ITCHOUT.txt", ios::in | ios::binary);
if(in.is_open())
{
// get the starting position
streampos start = in.tellg();
// go to the end
in.seekg(0, std::ios::end);
// get the ending position
streampos end = in.tellg();
// go back to the start
in.seekg(0, std::ios::beg);
// create a vector to hold the data that
// is resized to the total size of the file
std::vector<char> contents;
contents.resize(static_cast<size_t>(end - start));
// read it in
in.read(&contents[0], contents.size());
// print it out (for clarity)
hexdump(contents.data(), contents.size());
}
}
stream is a pointer to an int (the first element of the array you allocated1). *stream dereferences that pointer and gives you the first int.
A pointer is not an array. A pointer is not a buffer. Therefore, it carries no information about the size of the array it points to. There is no way you can print the entire array by providing only a pointer to the first element.
Whatever method you use to print that out, you'll need to provide the size information along with the pointer.
C++ happens to have a pointer + size package in its standard library: std::vector. I would recommend using that. Alternatively, you can just loop through the array yourself (which means using the size information) and print all its elements.
1Make sure the size of the file is a multiple of sizeof(int)!
Something like the following should do it. bytes_read() gives you the number of blocks read, in your case the block size is the file size so only one block can be read.
You should use a for loop to print the whole file. You're only printing one pointer address.
char *stream = (char *)malloc(fileSize);
fseek(fd, 0, SEEK_SET);
int bytes_read = fread(stream, fileSize, 1, fd);
for(int i=0; i<fileSize; i++){
printf("%d ", stream[i]);
}
I print the chars as numbers as binary data is not readable in the console. I don't know how you wanted the data to be formatted.
This is just meant as reference to your sample. You should really consider using Chad's sample. This is a far worse solution (as mixing C/C++ far too much) just for sake of completeness.
I'm trying to join two big files (like the UNIX cat command: cat file1 file2 > final) in C++.
I don't know how to do it because every method that I try it's very slow (for example, copy the second file into the first one line by line)
¿What is the best method for do that?
Sorry for being so brief, my english is not too good
If you're using std::fstream, then don't. It's intended primarily for formatted input/output, and char-level operations for it are slower than you'd expect. Instead, use std::filebuf directly. This is in addition to suggestions in other answers, specifically, using the larger buffer size.
Use binary-mode in the standard streams to do the job, don't deal with it as formatted data.
This is a demo if you want transfer the data in blocks:
#include <fstream>
#include <vector>
std::size_t fileSize(std::ifstream& file)
{
std::size_t size;
file.seekg(0, std::ios::end);
size = file.tellg();
file.seekg(0, std::ios::beg);
return size;
}
int main()
{
// 1MB! choose a conveinent buffer size.
const std::size_t blockSize = 1024 * 1024;
std::vector<char> data(blockSize);
std::ifstream first("first.txt", std::ios::binary),
second("second.txt", std::ios::binary);
std::ofstream result("result.txt", std::ios::binary);
std::size_t firstSize = fileSize(first);
std::size_t secondSize = fileSize(second);
for(std::size_t block = 0; block < firstSize/blockSize; block++)
{
first.read(&data[0], blockSize);
result.write(&data[0], blockSize);
}
std::size_t firstFilerestOfData = firstSize%blockSize;
if(firstFilerestOfData != 0)
{
first.read(&data[0], firstFilerestOfData);
result.write(&data[0], firstFilerestOfData);
}
for(std::size_t block = 0; block < secondSize/blockSize; block++)
{
second.read(&data[0], blockSize);
result.write(&data[0], blockSize);
}
std::size_t secondFilerestOfData = secondSize%blockSize;
if(secondFilerestOfData != 0)
{
second.read(&data[0], secondFilerestOfData);
result.write(&data[0], secondFilerestOfData);
}
first.close();
second.close();
result.close();
return 0;
}
Using plain old C++:
#include <fstream>
std::ifstream file1("x", ios_base::in | ios_base::binary);
std::ofstream file2("y", ios_base::app | ios_base::binary);
file2 << file1.rdbuf();
The Boost headers claim that copy() is optimized in some cases, though I'm not sure if this counts:
#include <boost/iostreams/copy.hpp>
// The following four overloads of copy_impl() optimize
// copying in the case that one or both of the two devices
// models Direct (see
// http://www.boost.org/libs/iostreams/doc/index.html?path=4.1.1.4)
boost::iostreams::copy(file1, file2);
update:
The Boost copy function is compatible with a wide variety of types, so this can be combined with Pavel Minaev's suggestion of using std::filebuf like so:
std::filebuf file1, file2;
file1.open("x", ios_base::in | ios_base::binary);
file2.open("y", ios_base::app | ios_base::binary);
file1.setbuf(NULL, 64 * 1024);
file2.setbuf(NULL, 64 * 1024);
boost::iostreams::copy(file1, file2);
Of course the actual optimal buffer size depends on many variables, 64k is just a wild guess.
As an alternative which may or may not be faster depending on your file size and memory on the machine. If memory is tight, you can make the buffer size smaller and loop over the f2.read grabbing the data in chunks and writing to f1.
#include <fstream>
#include <iostream>
using namespace std;
int main(int argc, char *argv[])
{
ofstream f1("test.txt", ios_base::app | ios_base::binary);
ifstream f2("test2.txt");
f2.seekg(0,ifstream::end);
unsigned long size = f2.tellg();
f2.seekg(0);
char *contents = new char[size];
f2.read(contents, size);
f1.write(contents, size);
delete[] contents;
f1.close();
f2.close();
return 1;
}