Read Binary file using MsgPack in C++ - c++

I want to read a binary file using Cpp, and its type is using MsgPack.
I'm not familiar with MsgPack and I try to read binary file fist and push in MsgPack. It doesn't work. It just gets the first number again and again. Can anyone help? Thanks a lot.
#include <bits/stdc++.h>
#include <msgpack.hpp>
using namespace std;
int main()
{
std::ifstream ifs("input.txt", std::ifstream::in);
std::stringstream buffer;
buffer << ifs.rdbuf();
msgpack::unpacked upd;
msgpack::unpack(upd, buffer.str().data(), buffer.str().size());
std::cout << upd.get() << std::endl;
return 0;
}
Which it can just get the first number "3".
I'm hoping to get number:
3
[3 6 7 5 3 5]
[6 2 9 1 2 7]
[0 9 3 6 0 6]
And here is the input binary file.

msgpack::unpack() unpacks the first MessagePack formatted data.
I think that 3 means the number of following arrays.
In this case offset is useful. See https://github.com/msgpack/msgpack-c/wiki/v2_0_cpp_unpacker#client-controls-a-buffer
Here is the code reading the first MessagePack data 3 as the counter. And then 3 times call msgpack::unpack() to unpack each array.
During that process, the variable offset is updated.
#include <iostream>
#include <fstream>
#include <msgpack.hpp>
int main()
{
std::ifstream ifs("input.txt");
std::string buffer((std::istreambuf_iterator<char>(ifs)),
std::istreambuf_iterator<char>());
msgpack::unpacked upd;
std::size_t offset = 0;
msgpack::unpack(upd, buffer.data(), buffer.size(), offset);
std::size_t count = upd.get().as<std::size_t>();
std::cout << count << std::endl;
for (std::size_t i = 0; i != count; ++i) {
msgpack::unpack(upd, buffer.data(), buffer.size(), offset);
std::cout << upd.get() << std::endl;
}
}
I believe that this is the approach based on your original code.

Your code unpacks only one msgpack message (first one, which is 3) but actually your file contains 4 messages. So you can use msgpack::unpacker class to unpack all messages one by one.
int main()
{
std::ifstream ifs("input.txt", std::ifstream::in);
std::string buffer((std::istreambuf_iterator<char>(ifs)),
std::istreambuf_iterator<char>());
msgpack::unpacker pac;
pac.reserve_buffer( buffer.size() );
std::copy( buffer.begin(), buffer.end(), pac.buffer() );
pac.buffer_consumed( buffer.size() );
msgpack::object_handle oh;
while ( pac.next(oh) ) {
msgpack::object msg = oh.get();
std::cout << msg << std::endl;
}
return 0;
}
P.S.
Read this and stop using #include <bits/stdc++.h>.

i think problem here : buffer.str().size() you can try
buffer.str().length()
or strlen(buffer.str().data().c_str());

Related

C++ cereal de-serialization trouble with large size vector

I hope to serialize large size vector with cereal, C++ serialization library.
But, if trying to do that, the exception "Failed to read " + std::to_string(size) + " bytes from input stream! Read " + std::to_string(readSize)" is thrown.
Does anyone know a good solution for this?
I'm using VisualStudio 2017.
The source code is shown below.
#include <iostream>
#include <fstream>
#include "include\cereal\cereal.hpp"
#include "include\cereal\archives\binary.hpp"
#include "include\cereal\types\vector.hpp"
#include "include\cereal\types\string.hpp"
void show(std::vector<int> v) {
for (auto i : v)std::cout << i << ",";
std::cout << std::endl;
}
int main(void) {
const std::string file_name = "out.cereal";
{
std::vector<int> src;
// const int STOP = 10; //OK
const int STOP = 1000; // NG
for (int i = 0; i < STOP; i++)src.push_back(i);
std::cout << "src:" << std::endl;
show(src);
std::ofstream ofs(file_name, std::ios::binary);
cereal::BinaryOutputArchive archive(ofs);
archive(src);
}
{
std::vector<int> dst;
std::fstream fs(file_name);
cereal::BinaryInputArchive iarchive(fs);
iarchive(dst);
std::cout << "dst:" << std::endl;
show(dst);
}
#ifdef _MSC_VER
system("pause");
#endif
return 0;
}
You code works fine for me in Linux, so I think it is to do with the difference between text and binary handling on Windows. Check that you pass std::ios::binary when you are constructing the input stream. Also construct it as std::ifstream rather than just std::fstream.
I think this might have to do with Windows expecting (or adding) a Unicode byte-order mark, which is confusing the serializer.

read data from text file and store it in 2D vector using c++ language

I am trying to read the data from the text file to the global 2D vector 'matrix
The file content will be like :
8, 3
1, 6, 2
9, 2, 5
1, 5, 25
7, 4, 25
I could not figure out what is my mistake. my code store just the first row.
#include <iostream>
#include<fstream>
#include<algorithm>
#include<vector>
#include <sstream>
#define EXIT_FILE_ERROR (1)
#define EXIT_UNEXPECTED_EOF (2)
#define EXIT_INVALID_FIRSTLINE (3)
#define MAXLINE (10000)
std::vector< std::vector<int> > matrix;
int main(int argc, const char * argv[])
{
FILE *fp;
std::string sFileName = "Matrix1.txt";
std::ifstream fileStream(sFileName);
if (!fileStream.is_open())
{
std::cout << "Exiting unable to open file" << std::endl;
exit(EXIT_FILE_ERROR);
}
std::string line;
while ( getline (fileStream,line) )
{
std::stringstream ss(line);
std::vector<int> numbers;
std::string v;
int value;
while(ss >> value)
{
numbers.push_back(value);
std::cout << value << std::endl;
}
matrix.push_back(numbers);
}
fileStream.close();
if ((fp = fopen(sFileName.c_str(), "r")) == NULL)
{
std::cout << "Exiting unable to open file" << std::endl;
exit(EXIT_FILE_ERROR);
}
return 0;
}
can some one tell me what is my mistake ?
Change double while loop in your code with the following piece of code:
while(getline(fileStream, line, '\n')) {
std::stringstream ss(line);
std::vector<int> numbers;
std::string in_line;
while(getline (ss, in_line, ',')) {
numbers.push_back(std::stoi(in_line, 0));
}
matrix.push_back(numbers);
}
Reason of Failure: You are messing things up with the parsing of the ss stream, you need to introduce delimiters.
However, I wouldn't recommend such kind of parsing. C++11 supports regular expressions that make parsing smooth sailing.

Missing data when reading/writing to stream

Here is complete example - compiles and runs, writes contents of map to the file and reads it right after:
#include <map>
#include <fstream>
#include <iostream>
using namespace std;
int main(int argc, char* argv[])
{
std::string fname("test.bin");
std::map<unsigned,unsigned> testMap;
testMap[0]=103;
testMap[1]=2;
testMap[5]=26;
testMap[22]=4;
std::ofstream output(fname.c_str(),std::ios_base::binary|std::ios_base::trunc);
for(std::map<unsigned,unsigned>::iterator iter = testMap.begin();iter != testMap.end();++iter)
{
unsigned temp = iter->first;
output.write((const char*)&temp,sizeof(temp));
unsigned temp1 = iter->second;
output.write((const char*)&temp1,sizeof(temp1));
std::cerr << temp <<" "<<temp1<<" "<<std::endl;
}
std::cerr << "wrote bytes.........."<<output.tellp()<<", map size "<<testMap.size()<<std::endl;
output.flush();
output.close();
std::ifstream input(fname.c_str());
// retrieve length of file:
input.seekg (0, input.end);
unsigned streamSize = input.tellg();
input.seekg (0, input.beg);
char* buff = new char[streamSize];
input.read(buff,streamSize);
cerr << "sizeof of input......"<<streamSize << endl;
cerr << "read bytes..........."<<input.gcount() << endl;
::getchar();
return 0;
}
It gives the following output:
0 103
1 2
5 26
22 4
wrote bytes..........32, map size 4
sizeof of input......32
read bytes...........20
The question is why bytes read does not match bytes written, and how to read/write whole map.
P.S. Online compiler gives me expected output of 32 read bytes, I'm getting wrong output while compiling with Visual Studio 2010 proffesional.
Make sure you're opening the file as a binary file.

Need help loading simple text data with c++

I need help loading a custom file format into my program made in c++...
I know there's a simple way of doing this but I think I'm using the
wrong terms to search for it online...
My custom format for 3d objects is as follows:
NumVerts 6
//verts (float)
-1
-1
0
1
-1
0
-1
1
0
1
-1
0
1
1
0
-1
1
0
//texture (float)
0
0
1
0
0
1
1
0
1
1
0
1
//index (int)
0
1
2
1
3
2
And that is a quad... (yeas; I know... horrible format... but it's what I'm using for an android game).
I want to make a function in c++ for my editor (SDL + OpenGL for windows) that loads these files into data... Unfortunately though I know how to export this format with C++, I can't figure out how to import them... I wish to use the fstream commands...
If someone could quickly write out a simple version I'd be really thankful.
I just it to do the following:
Find text file from input string
read "NumVerts" and grab the integer written after it
loop through the next (NumVerts*3) lines and grab each number as a float
loop though the next (NumVerts*2) lines and grab each number as a float
loop through the next (NumVerts*1) lines and grab each number as an Int
(skip any line that starts with "//")
close file.
Thank you for reading and any help would be really good right now... or a relivent link that is quite simple and reads strings from files and grabs numbers from them to be placed into memory...
I really just want to finish this game and it's getting really stressful trying to locate helpful tutorials.
Update: updated the script... I accidently forgot to seperate 1's and 0's...
Hope this helps. Incidentally, you have the wrong number of vertex components - you need 18 of them.
#include <algorithm>
#include <exception>
#include <fstream>
#include <iostream>
#include <string>
#include <vector>
#include <boost/algorithm/string/trim.hpp>
#include <boost/lexical_cast.hpp>
using boost::lexical_cast;
int load_3d_object(const std::string& filename,
std::vector<float>& vertComponents,
std::vector<float>& texComponents,
std::vector<int>& indices)
{
std::ifstream fs(filename.c_str());
std::string line;
if(!std::getline(fs, line))
{
throw std::runtime_error("The input file is empty");
}
if(line.substr(0,8) != "NumVerts")
{
throw std::runtime_error("The first line must start with NumVerts");
}
// Extract the number of vertices.
int numVerts = lexical_cast<int>(line.substr(line.find(' ') + 1));
// Read in the vertex components, texture components and indices.
while(std::getline(fs, line))
{
boost::trim(line);
if(line.substr(0,2) == "//") continue;
if((int)vertComponents.size() < numVerts * 3) vertComponents.push_back(lexical_cast<float>(line));
else if((int)texComponents.size() < numVerts * 2) texComponents.push_back(lexical_cast<float>(line));
else indices.push_back(lexical_cast<int>(line));
}
return numVerts;
}
int main()
{
std::vector<float> vertComponents;
std::vector<float> texComponents;
std::vector<int> indices;
try
{
int numVerts = load_3d_object("input.txt", vertComponents, texComponents, indices);
}
catch(std::exception& e)
{
std::cout << e.what() << '\n';
}
return 0;
}
Hopefully this might help (minimal error checking):
int _get_num_verts_value(std::ifstream& a_in)
{
char buf[128];
int result = -1;
a_in.getline(buf, sizeof(buf));
while (a_in.good())
{
if (a_in.gcount() > 9)
{
string s(buf);
if (0 == s.find("NumVerts "))
{
result = atoi(s.substr(8).c_str());
break;
}
}
a_in.getline(buf, sizeof(buf));
}
return result;
}
template <typename T>
void _get_values(std::ifstream& a_in, std::vector<T>& a_values)
{
char buf[128];
a_in.getline(buf, sizeof(buf));
int i = 0;
while (a_in.good())
{
string line(buf);
if (0 != line.find("//"))
{
a_values[i++] = boost::lexical_cast<T>(line);
// All read ?
if (i == a_values.capacity())
{
break;
}
}
a_in.getline(buf, sizeof(buf));
}
}
int main(int /*a_argc*/, char** /*a_argv*/)
{
ifstream in("test.txt");
const int num_verts_value = _get_num_verts_value(in);
std::vector<float> verts(num_verts_value * 3);
_get_values<float>(in, verts);
std::vector<float> textures(num_verts_value * 2);
_get_values<float>(in, textures);
std::vector<int> indexes(num_verts_value);
_get_values<int>(in, indexes);
in.close();
return 0;
}
using a flat file for this purpose could be a bad idea. it's pretty fragile. I prefer to a binary file or a self-described file (e.g xml).
std::ifstream in("in");
char buf[256];
std::string line;
int NumVerts;
in >> buf >> NumVerts;
std::vector<float> verts(NumVerts * 3);
std::getline(in, line);//read the rest of the line
std::getline(in, line);//skip the comment line
std::for_each(verts.begin(), verts.end(), [&in](float& par){
in >> par;
});
std::vector<float> texture(NumVerts * 2);
std::getline(in, line);//read the rest of the line
std::getline(in, line);//skip the comment line
std::for_each(texture.begin(), texture.end(), [&in](float& par){
in >> par;
});
std::vector<int> index(NumVerts);
std::getline(in, line);//read the rest of the line
std::getline(in, line);//skip the comment line
std::for_each(index.begin(), index.end(), [&in](int& par){
in >> par;
});
For the links on how to DIY:
Tutorial: http://www.cplusplus.com/doc/tutorial/files/
Reference: http://new.cplusplus.com/reference/iostream/fstream/
Example code:
#include <string>
#include <iostream> // needed for printing stuff out
#include <fstream> // Needed to read the file
#include <stdexcept>
#include <vector>
using namespace std;
struct Vertice {
double x,y,z; // 3 part stuff to read
double a,b; // 2 part stuff to read
int i; // 1 int thing to read
};
/// Reads a number ignoring comment lines (and any other line that isn't a number)
template <typename T>
T readNumber(istream& data) {
char lastChar;
while (data.good()) {
data >> lastChar; // Don't use peek as that won't skip whitespace
data.putback(lastChar);
if (( (lastChar >= '0') && (lastChar <= '9') ) || (lastChar == '-')) {
// If we're looking at a number read and return it
T result;
data >> result;
return result;
} else {
// If it's not part of a number .. assume it's a comment line and skip the whole line
string commentLine;
getline(data, commentLine);
// TODO: Maybe just skip '//' lines and throw an exception for everything else..
}
}
throw exception("Couldn't read file");
}
double readDouble(istream& data) { return readNumber<double>(data); }
int readInt(istream& data) { return readNumber<int>(data); }
int main(int argc, char** argv) {
if (argc != 2)
cout << "Usage: " << argv[0] << " [DATA_FILE_NAME]" << endl;
else {
fstream data(argv[1], ios_base::in);
data >> skipws; // Skip whitespace
string lastString;
long numVerts = -1;
// Read in words ignoring everything until we hit a 'NumVerts'
while (numVerts < 0) {
data >> lastString;
if (lastString == "NumVerts")
data >> numVerts;
}
// We know how many to get now
typedef vector<Vertice> Verts;
Verts verts(numVerts);
// Read in the triples
for (Verts::iterator i=verts.begin(); i != verts.end(); ++i) {
i->x = readDouble(data);
i->y = readDouble(data);
i->z = readDouble(data);
}
// Read in the pairs (some other data)
for (Verts::iterator i=verts.begin(); i != verts.end(); ++i) {
i->a = readDouble(data);
i->b = readDouble(data);
}
// Read in the single integer value
for (Verts::iterator i=verts.begin(); i != verts.end(); ++i) {
i->i = readInt(data);
}
// Print out all we found
for (Verts::iterator i=verts.begin(); i != verts.end(); ++i) {
cout << "Vertice" << endl
<< " x: " << i->x << endl
<< " y: " << i->y << endl
<< " z: " << i->z << endl
<< " a: " << i->a << endl
<< " b: " << i->b << endl
<< " i: " << i->i << endl
<< endl;
}
}
}
The example code throws an error on the provided files as there isn't enough data to fill out the 'NumVerts'.

print out the last 10 lines of a file

I want to have the option to print out the last 10 lines of a textfile . with this program I've been able to read the whole textfile, but I can't figure out how to manipulate the array in which the textfile is saved, any help?
// Textfile output
#include<fstream>
#include<iostream>
#include<iomanip>
using namespace std;
int main() {
int i=1;
char zeile[250], file[50];
cout << "filename:" << flush;
cin.get(file,50); ///// (1)
ifstream eingabe(datei , ios::in); /////(2)
if (eingabe.good() ) { /////(3)
eingabe.seekg(0L,ios::end); ////(4)
cout << "file:"<< file << "\t"
<< eingabe.tellg() << " Bytes" ////(5)
<< endl;
for (int j=0; j<80;j++)
cout << "_";
cout << endl;
eingabe.seekg(0L, ios::beg); ////(6)
while (!eingabe.eof() ){ ///(7)
eingabe.getline(zeile,250); ///(8)
cout << setw(2) << i++
<< ":" << zeile << endl;
}
}
else
cout <<"dateifehler oder Datei nicht gefunden!"
<< endl;
return 0;
}
Try this:
#include <list>
#include <string>
#include <iostream>
#include <fstream>
#include <algorithm>
#include <iterator>
// A class that knows how to read a line using operator >>
struct Line
{
std::string theLine;
operator std::string const& () const { return theLine; }
friend std::istream& operator>>(std::istream& stream, Line& l)
{
return std::getline(stream, l.theLine);
}
};
// A circular buffer that only saves the last n lines.
class Buffer
{
public:
Buffer(size_t lc)
: lineCount(lc)
{}
void push_back(std::string const& line)
{
buffer.insert(buffer.end(),line);
if (buffer.size() > lineCount)
{
buffer.erase(buffer.begin());
}
}
typedef std::list<std::string> Cont;
typedef Cont::const_iterator const_iterator;
typedef Cont::const_reference const_reference;
const_iterator begin() const { return buffer.begin(); }
const_iterator end() const { return buffer.end();}
private:
size_t lineCount;
std::list<std::string> buffer;
};
// Main
int main()
{
std::ifstream file("Plop");
Buffer buffer(10);
// Copy the file into the special buffer.
std::copy(std::istream_iterator<Line>(file), std::istream_iterator<Line>(),
std::back_inserter(buffer));
// Copy the buffer (which only has the last 10 lines)
// to std::cout
std::copy(buffer.begin(), buffer.end(),
std::ostream_iterator<std::string>(std::cout, "\n"));
}
Basically, you are not saving the file contents to any array. The following sample will give you a head start:
#include <iostream>
#include <vector>
#include <string>
int main ( int, char ** )
{
// Ask user for path to file.
std::string path;
std::cout << "filename:";
std::getline(std::cin, path);
// Open selected file.
std::ifstream file(path.c_str());
if ( !file.is_open() )
{
std::cerr << "Failed to open '" << path << "'." << std::endl;
return EXIT_FAILURE;
}
// Read lines (note: stores all of it in memory, might not be your best option).
std::vector<std::string> lines;
for ( std::string line; std::getline(file,line); )
{
lines.push_back(line);
}
// Print out (up to) last ten lines.
for ( std::size_t i = std::min(lines.size(), std::size_t(10)); i < lines.size(); ++i )
{
std::cout << lines[i] << std::endl;
}
}
It would probably be wiser to avoid storing the whole file into memory, so you could re-write the last 2 segments this way:
// Read up to 10 lines, accumulating.
std::deque<std::string> lines;
for ( std::string line; lines.size() < 0 && getline(file,line); )
{
lines.push_back(line);
}
// Read the rest of the file, adding one, dumping one.
for ( std::string line; getline(file,line); )
{
lines.pop_front();
lines.push_back(line);
}
// Print out whatever is left (up to 10 lines).
for ( std::size_t i = 0; i < lines.size(); ++i )
{
std::cout << lines[i] << std::endl;
}
The eof() function does not do what you and it seems a million other C++ newbies think it does. It does NOT predict if the next read will work. In C++ as in any other language, you must check the status of each read operation, not the state of the input stream before the read. so the canonical C++ read line loop is:
while ( eingabe.getline(zeile,250) ) {
// do something with zeile
}
Also, you should be reading into a std::string, and get rid of that 250 value.
Do a circular buffer with 10 slots and while reading the file lines, putting them into this buffer. When you finish thr file, do a position++ to go to the first element and print them all.
Pay attention for null values if the file has less than 10 lines.
Have an array of strings with size 10.
Read the first line and store into the array
Continue reading till the array is full
Once the array is full delete the first entry so that you can enter new line
Repeate step 3 and 4 till the file is finished reading.
I investigate proposed approaches here and describe all in my blog post. There is a better solution but you have to jump to the end and persist all needed lines:
std::ifstream hndl(filename, std::ios::in | std::ios::ate);
// and use handler in function which iterate backward
void print_last_lines_using_circular_buffer(std::ifstream& stream, int lines)
{
circular_buffer<std::string> buffer(lines);
std::copy(std::istream_iterator<line>(stream),
std::istream_iterator<line>(),
std::back_inserter(buffer));
std::copy(buffer.begin(), buffer.end(),
std::ostream_iterator<std::string>(std::cout));
}