Here is complete example - compiles and runs, writes contents of map to the file and reads it right after:
#include <map>
#include <fstream>
#include <iostream>
using namespace std;
int main(int argc, char* argv[])
{
std::string fname("test.bin");
std::map<unsigned,unsigned> testMap;
testMap[0]=103;
testMap[1]=2;
testMap[5]=26;
testMap[22]=4;
std::ofstream output(fname.c_str(),std::ios_base::binary|std::ios_base::trunc);
for(std::map<unsigned,unsigned>::iterator iter = testMap.begin();iter != testMap.end();++iter)
{
unsigned temp = iter->first;
output.write((const char*)&temp,sizeof(temp));
unsigned temp1 = iter->second;
output.write((const char*)&temp1,sizeof(temp1));
std::cerr << temp <<" "<<temp1<<" "<<std::endl;
}
std::cerr << "wrote bytes.........."<<output.tellp()<<", map size "<<testMap.size()<<std::endl;
output.flush();
output.close();
std::ifstream input(fname.c_str());
// retrieve length of file:
input.seekg (0, input.end);
unsigned streamSize = input.tellg();
input.seekg (0, input.beg);
char* buff = new char[streamSize];
input.read(buff,streamSize);
cerr << "sizeof of input......"<<streamSize << endl;
cerr << "read bytes..........."<<input.gcount() << endl;
::getchar();
return 0;
}
It gives the following output:
0 103
1 2
5 26
22 4
wrote bytes..........32, map size 4
sizeof of input......32
read bytes...........20
The question is why bytes read does not match bytes written, and how to read/write whole map.
P.S. Online compiler gives me expected output of 32 read bytes, I'm getting wrong output while compiling with Visual Studio 2010 proffesional.
Make sure you're opening the file as a binary file.
Related
I'm using small files currently for testing and will scale up once it works.
I made a file bigFile.txt that has:
ABCDEFGHIJKLMNOPQRSTUVWXYZ
I'm running this to segment the data that is being read from the file:
#include <iostream>
#include <fstream>
#include <memory>
using namespace std;
int main()
{
ifstream file("bigfile.txt", ios::binary | ios::ate);
cout << file.tellg() << " Bytes" << '\n';
ifstream bigFile("bigfile.txt");
constexpr size_t bufferSize = 4;
unique_ptr<char[]> buffer(new char[bufferSize]);
while (bigFile)
{
bigFile.read(buffer.get(), bufferSize);
// print the buffer data
cout << buffer.get() << endl;
}
}
This gives me the following result:
26 Bytes
ABCD
EFGH
IJKL
MNOP
QRST
UVWX
YZWX
Notice how in the last line after 'Z' the character 'WX' is repeated again?
How do I get rid of it so that it stops after reaching the end?
cout << buffer.get() uses the const char* overload, which prints a NULL-terminated C string.
But your buffer isn't NULL-terminated, and istream::read() can read less characters than the buffer size. So when you print buffer, you end up printing old characters that were already there, until the next NULL character is encountered.
Use istream::gcount() to determine how many characters were read, and print exactly that many characters. For example, using std::string_view:
#include <iostream>
#include <fstream>
#include <memory>
#include <string_view>
using namespace std;
int main()
{
ifstream file("bigfile.txt", ios::binary | ios::ate);
cout << file.tellg() << " Bytes" << "\n";
file.seekg(0, std::ios::beg); // rewind to the beginning
constexpr size_t bufferSize = 4;
unique_ptr<char[]> buffer = std::make_unique<char[]>(bufferSize);
while (file)
{
file.read(buffer.get(), bufferSize);
auto bytesRead = file.gcount();
if (bytesRead == 0) {
// EOF
break;
}
// print the buffer data
cout << std::string_view(buffer.get(), bytesRead) << endl;
}
}
Note also that there's no need to open the file again - you can rewind the original one to the beginning and read it.
The problem is that you don't override the buffer's content. Here's what your code does:
It reads the beginning of the file
When reaching the 'YZ', it reads it and only overrides the buffer's first two characters ('U' and 'V') because it has reached the end of the file.
One easy fix is to clear the buffer before each file read:
#include <iostream>
#include <fstream>
#include <array>
int main()
{
std::ifstream bigFile("bigfile.txt", std::ios::binary | std::ios::ate);
int fileSize = bigFile.tellg();
std::cout << bigFile.tellg() << " Bytes" << '\n';
bigFile.seekg(0);
constexpr size_t bufferSize = 4;
std::array<char, bufferSize> buffer;
while (bigFile)
{
for (int i(0); i < bufferSize; ++i)
buffer[i] = '\0';
bigFile.read(buffer.data(), bufferSize);
// Print the buffer data
std::cout.write(buffer.data(), bufferSize) << '\n';
}
}
I also changed:
The std::unique_ptr<char[]> to a std::array since we don't need dynamic allocation here and std::arrays's are safer that C-style arrays
The printing instruction to std::cout.write because it caused undefined behavior (see #paddy's comment). std::cout << prints a null-terminated string (a sequence of characters terminated by a '\0' character) whereas std::cout.write prints a fixed amount of characters
The second file opening to a call to the std::istream::seekg method (see #rustyx's answer).
Another (and most likely more efficient) way of doing this is to read the file character by character, put them in the buffer, and printing the buffer when it's full. We then print the buffer if it hasn't been already in the main for loop.
#include <iostream>
#include <fstream>
#include <array>
int main()
{
std::ifstream bigFile("bigfile.txt", std::ios::binary | std::ios::ate);
int fileSize = bigFile.tellg();
std::cout << bigFile.tellg() << " Bytes" << '\n';
bigFile.seekg(0);
constexpr size_t bufferSize = 4;
std::array<char, bufferSize> buffer;
int bufferIndex;
for (int i(0); i < fileSize; ++i)
{
// Add one character to the buffer
bufferIndex = i % bufferSize;
buffer[bufferIndex] = bigFile.get();
// Print the buffer data
if (bufferIndex == bufferSize - 1)
std::cout.write(buffer.data(), bufferSize) << '\n';
}
// Override the characters which haven't been already (in this case 'W' and 'X')
for (++bufferIndex; bufferIndex < bufferSize; ++bufferIndex)
buffer[bufferIndex] = '\0';
// Print the buffer for the last time if it hasn't been already
if (fileSize % bufferSize /* != 0 */)
std::cout.write(buffer.data(), bufferSize) << '\n';
}
I want to read a binary file using Cpp, and its type is using MsgPack.
I'm not familiar with MsgPack and I try to read binary file fist and push in MsgPack. It doesn't work. It just gets the first number again and again. Can anyone help? Thanks a lot.
#include <bits/stdc++.h>
#include <msgpack.hpp>
using namespace std;
int main()
{
std::ifstream ifs("input.txt", std::ifstream::in);
std::stringstream buffer;
buffer << ifs.rdbuf();
msgpack::unpacked upd;
msgpack::unpack(upd, buffer.str().data(), buffer.str().size());
std::cout << upd.get() << std::endl;
return 0;
}
Which it can just get the first number "3".
I'm hoping to get number:
3
[3 6 7 5 3 5]
[6 2 9 1 2 7]
[0 9 3 6 0 6]
And here is the input binary file.
msgpack::unpack() unpacks the first MessagePack formatted data.
I think that 3 means the number of following arrays.
In this case offset is useful. See https://github.com/msgpack/msgpack-c/wiki/v2_0_cpp_unpacker#client-controls-a-buffer
Here is the code reading the first MessagePack data 3 as the counter. And then 3 times call msgpack::unpack() to unpack each array.
During that process, the variable offset is updated.
#include <iostream>
#include <fstream>
#include <msgpack.hpp>
int main()
{
std::ifstream ifs("input.txt");
std::string buffer((std::istreambuf_iterator<char>(ifs)),
std::istreambuf_iterator<char>());
msgpack::unpacked upd;
std::size_t offset = 0;
msgpack::unpack(upd, buffer.data(), buffer.size(), offset);
std::size_t count = upd.get().as<std::size_t>();
std::cout << count << std::endl;
for (std::size_t i = 0; i != count; ++i) {
msgpack::unpack(upd, buffer.data(), buffer.size(), offset);
std::cout << upd.get() << std::endl;
}
}
I believe that this is the approach based on your original code.
Your code unpacks only one msgpack message (first one, which is 3) but actually your file contains 4 messages. So you can use msgpack::unpacker class to unpack all messages one by one.
int main()
{
std::ifstream ifs("input.txt", std::ifstream::in);
std::string buffer((std::istreambuf_iterator<char>(ifs)),
std::istreambuf_iterator<char>());
msgpack::unpacker pac;
pac.reserve_buffer( buffer.size() );
std::copy( buffer.begin(), buffer.end(), pac.buffer() );
pac.buffer_consumed( buffer.size() );
msgpack::object_handle oh;
while ( pac.next(oh) ) {
msgpack::object msg = oh.get();
std::cout << msg << std::endl;
}
return 0;
}
P.S.
Read this and stop using #include <bits/stdc++.h>.
i think problem here : buffer.str().size() you can try
buffer.str().length()
or strlen(buffer.str().data().c_str());
I am trying to parse a .dat file reading it byte by byte with this code.(the name of the file is in arv[1])
std::ifstream is (arv[1], std::ifstream::binary);
if (is) {
is.seekg (0, is.end);
int length = is.tellg();
is.seekg (0, is.beg);
char * buffer = new char [length];
is.read (buffer,length);
if (is)
std::cout << "all characters read successfully.";
else
std::cout << "error: only " << is.gcount() << " could be read";
is.close();
}
Now all file is in the buffer variable. The file contains numbers represented in 32 bits, how can I iterate over the buffer reading 4 bytes at a time and convert them to integer?
first of all , you have a memory leak, you dynamically allocate character array but never delete[] them.
use std::string instead:
std::string buffer(length,0);
is.read (&buffer[0],length);
now, assuming you had written the integer correctly, and have read it correctly into buffer, you can use this character array as pointer to integer:
int myInt = *(int*)&buffer[0];
(do you understand why?)
if you have more then one integer stored:
std::vector<int> integers;
for (int i=0;i<buffer.size();i+=sizeof(int)){
integers.push_back(*(int*)&buffer[i]);
}
Instead of:
char * buffer = new char [length];
is.read (buffer,length);
You can use:
int numIntegers = length/sizeof(int);
int* buffer = new int[numIntegers];
is.read(reinterpret_cast<char*>(buffer), numIntegers*sizeof(int));
Update, in response to OP's comment
I am not seeing any problems with the approach I suggested. Here's a sample program and the output I see using g++ 4.9.2.
#include <iostream>
#include <fstream>
#include <cstdlib>
void writeData(char const* filename, int n)
{
std::ofstream out(filename, std::ios::binary);
for ( int i = 0; i < n; ++i )
{
int num = std::rand();
out.write(reinterpret_cast<char*>(&num), sizeof(int));
}
}
void readData(char const* filename)
{
std::ifstream is(filename, std::ifstream::binary);
if (is)
{
is.seekg (0, is.end);
int length = is.tellg();
is.seekg (0, is.beg);
int numIntegers = length/sizeof(int);
int* buffer = new int [numIntegers];
std::cout << "Number of integers: " << numIntegers << std::endl;
is.read(reinterpret_cast<char*>(buffer), numIntegers*sizeof(int));
if (is)
std::cout << "all characters read successfully." << std::endl;
else
std::cout << "error: only " << is.gcount() << " could be read" << std::endl;
for (int i = 0; i < numIntegers; ++i )
{
std::cout << buffer[i] << std::endl;
}
}
}
int main()
{
writeData("test.bin", 10);
readData("test.bin");
}
Output
Number of integers: 10
all characters read successfully.
1481765933
1085377743
1270216262
1191391529
812669700
553475508
445349752
1344887256
730417256
1812158119
This question already has answers here:
Question about seekg() function of ifstream in C++?
(3 answers)
Closed 8 years ago.
In the process of writing a lz4 csv to compressed binary file converter (high volume forex tick data csv) in the hope of reducing the storage/disk bandwidth requirements on my tiny vps.
self contained code to illustrate
#include <string>
#include <fstream>
#include <iostream>
#include "lz4.h"
using namespace std;
int main()
{
char szString[] = "2013-01-07 00:00:04,0.98644,0.98676 2013-01-07 00:01:19,0.98654,0.98676 2013-01-07 00:01:38,0.98644,0.98696";
const char* pchSource = szString;
int nInputSize = sizeof(szString);
cout <<"- pchSource -" << endl << pchSource << endl;
cout <<"nbytes = "<< nInputSize << endl << endl;
ofstream source("pchSource.txt");
source << pchSource;
int nbytesPassed = 0;
int nMaxCompressedSize = LZ4_compressBound(nInputSize);
char *pszDest = new char[nMaxCompressedSize];
nbytesPassed = LZ4_compress(pchSource, pszDest, nInputSize);
cout <<"- pszDest Compressed-" << endl;
cout <<"nbytesPassed = "<< nbytesPassed << endl;
cout << pszDest << endl << endl;
// pszDest garbage ?
char *pszDestUnCompressed = new char[nInputSize];
LZ4_uncompress(pszDest, pszDestUnCompressed, nInputSize);
cout <<"- pszDestUnCompressed -" << endl;
cout <<"nbytesPassed = "<< nbytesPassed << endl;
cout << pszDestUnCompressed << endl << endl;
//pszDestUnCompressed is correct ?
delete[] pszDestUnCompressed;
pszDestUnCompressed = 0;
// ok lets write compressed pszDest to pszDest.dat
ofstream outCompressedFile("pszDest.dat",std::ofstream::binary);
outCompressedFile.write (pszDest,nMaxCompressedSize);
delete[] pszDest;
pszDest = 0;
//read it back in and try to uncompress it
ifstream infile("pszDest.dat",std::ifstream::binary);
infile.seekg (0,infile.end);
int nCompressedInputSize = infile.tellg();
infile.seekg (0);
char* buffer = new char[nCompressedInputSize];
infile.read (buffer,nCompressedInputSize);
const char* pchbuffer = buffer;
char* pszUnCompressedFile = new char[nInputSize];
nbytesPassed = LZ4_uncompress(pchbuffer, pszUnCompressedFile, nInputSize);
cout <<"- pszUnCompressedFile -" << endl;
cout <<"nbytesPassed = "<< nbytesPassed << endl;
cout << pszUnCompressedFile << endl;
//write uncompressed pszDest.dat to pszUnCompressedFile.txt
ofstream outUncompressedSource("pszUnCompressedFile.txt");
outUncompressedSource << pszUnCompressedFile;
// On my system 32bit ArchLinux 3.7.10-1 - gcc 4.7.2-4
// file contains random Garbage
delete[] buffer;
buffer = 0;
delete[] pszUnCompressedFile;
pszUnCompressedFile = 0;
return 0;
}
CONSOLE OUTPUT :
- pchSource -
2013-01-07 00:00:04,0.98644 .....
nbytes = 108
- pszDest Compressed-
nbytesPassed = 63
�2013-01-07 00:
- pszDestUnCompressed -
nbytesPassed = 63
2013-01-07 00:00:04,0.98644 .....
- pszUnCompressedFile -
nbytesPassed = -17
�W��W�-07 q
Process returned 0 (0x0) execution time : 0.010 s
Press ENTER to continue.
I'm obviously missing something, apart form the samples included in the source are there any-other usage examples ?
All working now thanks, here is the code for anyone that is interested
#include <fstream>
#include <iostream>
#include "lz4.h"
using namespace std;
int main()
{
char szSource[] = "2013-01-07 00:00:04,0.98644,0.98676 2013-01-07 00:01:19,0.98654,0.98676 2013-01-07 00:01:38,0.98644,0.98696";
int nInputSize = sizeof(szSource);
// compress szSource into pchCompressed
char* pchCompressed = new char[nInputSize];
int nCompressedSize = LZ4_compress((const char *)(&szSource), pchCompressed, nInputSize);
// write pachCompressed to binary lz4.dat
ofstream outBinaryFile("lz4.dat",ofstream::binary);
outBinaryFile.write(pchCompressed, nCompressedSize);
outBinaryFile.close();
delete[] pchCompressed;
pchCompressed = 0;
//read compressed binary file (assume we pass/encode nInputSize but don't know nCompressedSize)
ifstream infCompressedBinaryFile( "lz4.dat", ifstream::binary );
//Get compressed file size for buffer
infCompressedBinaryFile.seekg (0,infCompressedBinaryFile.end);
int nCompressedInputSize = infCompressedBinaryFile.tellg();
infCompressedBinaryFile.clear();
infCompressedBinaryFile.seekg(0,ios::beg);
//Read file into buffer
char* pchCompressedInput = new char[nCompressedInputSize];
infCompressedBinaryFile.read(pchCompressedInput,nCompressedSize);
infCompressedBinaryFile.close();
// Decompress buffer
char* pchDeCompressed = new char[nInputSize]; //(nCompressedInputSize *2) +8
LZ4_uncompress(pchCompressedInput, pchDeCompressed, nInputSize);
delete[] pchCompressedInput;
pchCompressedInput = 0;
// write decompressed pachUnCompressed to
ofstream outFile("lz4.txt");
outFile.write(pchDeCompressed, nInputSize);
outFile.close();
delete[] pchDeCompressed;
pchDeCompressed = 0;
return 0;
}
I am also working on a a simple CLI csv to binary I/O example here
I'm trying to read the first line of an MP3 file (I edited this mp3 file to contain the text "I'm an MP3" right at the beginning of the file).
This is what I'm trying to do:
#include <iostream>
#include <string>
#include <fstream>
using namespace std;
int main()
{
fstream mp3;
mp3.open("05 Imagine.mp3", ios::binary | ios::in | ios::out);
/*mp3.seekg(0, ios::end);
int lof = mp3.tellg();
cout << "Length of file: " << lof << endl;
mp3.seekg(0, ios::beg);*/
//char ch;
//cout << mp3.get(ch) << endl;
char* somebuf;
while(mp3.read(somebuf, 10)) //Read the first 10 chars which are "I'm an MP3 file".
{
//cout << somebuf;
}
return 0;
}
For some reason, that is crashing. At some point it didn't crash, but it didn't print anything when I did cout << somebuf. Can someone help me with this?
You never allocated anything for somebuf:
char* somebuf;
therefore, it doesn't point anywhere.
char* somebuf = new char[11];
somebuf[10] = '\0'; // Not sure if it is necessary to null-terminate...
while(mp3.read(somebuf, 10)) // Read the first 10 chars which are "I'm an MP3 file".
{
//cout << somebuf;
}
// and free it later
delete [] somebuf;
Alternatively:
char somebuf[11];
somebuf[10] = '\0'; // Not sure if it is necessary to null-terminate...
while(mp3.read(somebuf, 10)) // Read the first 10 chars which are "I'm an MP3 file".
{
//cout << somebuf;
}
Initialize the buffer:
char somebuf[10];
while(mp3.read(somebuf, 10)) //Read the first 10 chars which are "I'm an MP3 file".
{
//cout << somebuf;
}