I hope to serialize large size vector with cereal, C++ serialization library.
But, if trying to do that, the exception "Failed to read " + std::to_string(size) + " bytes from input stream! Read " + std::to_string(readSize)" is thrown.
Does anyone know a good solution for this?
I'm using VisualStudio 2017.
The source code is shown below.
#include <iostream>
#include <fstream>
#include "include\cereal\cereal.hpp"
#include "include\cereal\archives\binary.hpp"
#include "include\cereal\types\vector.hpp"
#include "include\cereal\types\string.hpp"
void show(std::vector<int> v) {
for (auto i : v)std::cout << i << ",";
std::cout << std::endl;
}
int main(void) {
const std::string file_name = "out.cereal";
{
std::vector<int> src;
// const int STOP = 10; //OK
const int STOP = 1000; // NG
for (int i = 0; i < STOP; i++)src.push_back(i);
std::cout << "src:" << std::endl;
show(src);
std::ofstream ofs(file_name, std::ios::binary);
cereal::BinaryOutputArchive archive(ofs);
archive(src);
}
{
std::vector<int> dst;
std::fstream fs(file_name);
cereal::BinaryInputArchive iarchive(fs);
iarchive(dst);
std::cout << "dst:" << std::endl;
show(dst);
}
#ifdef _MSC_VER
system("pause");
#endif
return 0;
}
You code works fine for me in Linux, so I think it is to do with the difference between text and binary handling on Windows. Check that you pass std::ios::binary when you are constructing the input stream. Also construct it as std::ifstream rather than just std::fstream.
I think this might have to do with Windows expecting (or adding) a Unicode byte-order mark, which is confusing the serializer.
Related
I wanted to create simple todo like program in console where you can input your task ((name) (level) (interesting level)) and it will save it from the vector to the binary file. I have this program, but when I try to save tasks to the file and then read from it, it gives me an error Segmentation fault (core dumped) and I have no ideas why... I tried to debug, bug I steel have no ideas why this is not working. Here is the git-hub link (here). What do I do?
As mentioned on the comments you cannot do what you are trying to do with a dynamic object such as a std::string. You will never know how big such an object is when you are loading it. Because of that, your program may start loading part of another object on to the std::string.
Your simplest solution in my opinion is to change the std::string for a fix sized array of char and limit the name size a task can have. Bellow is a fully working example.
#include <fstream>
#include <iostream>
#include <vector>
struct data
{
int m_value = 0;
int m_priority = 0;
char m_task[28] = {"\0"};
};
int main()
{
std::vector<data> dat;
data datt;
for (int i = 0; i < 4; i++)
{
sprintf( datt.m_task, "task %d", i);
datt.m_priority = i;
datt.m_value = i*4;
dat.push_back(datt);
}
std::ofstream ofile;
ofile.open("data.dat", std::ios::binary);
if (ofile.is_open())
{
for (const auto& d : dat)
ofile.write(reinterpret_cast<const char *>(&d), sizeof( d));
ofile.close();
}
std::ifstream ifile;
ifile.open("data.dat", std::ios::binary);
if (ifile.is_open())
{
while (!ifile.eof()){
ifile.read(reinterpret_cast<char *>(&datt), sizeof( datt));
std::cout << datt.m_value << " " << datt.m_priority << " " << datt.m_task << std::endl;
}
ifile.close();
}
}
I am trying to move the map array but for this I need to put everything in buffer, here is my code.
#include <iostream>
#include <map>
#include <string>
std::map<std::int, std::string> maphack;
maphack.emplace(1, "CS");
maphack.emplace(2, "PBG");
maphack.emplace(3, "Maicraft");
maphack.insert_or_assign(3, "GTA5");
for (auto &a : maphack) {
std::cout << a.first << " : " << a.second << '\n';
}
How to put everything above in the buffer?
char buffer[64];
send(sock, buffer, AmountToSend, 0);
The simple answer is to use std::ostringstream and std::istringstream which allows to use std streams machinery with a string storage. Which can be converted to char* with c_str() method if needed later, full example is below.
The better answer can be to use more heavy lifting library for serialization, the choice of the format will depend on your program. Do you need binary or text format? Do you want other programs (in other programming languages) be able to read it? Do you need to support different endian system? Do you need to handle error when deserializing malformed input? Boost, Qt, google protobuf, and other ones are out there.
#include <iostream>
#include <map>
#include <sstream>
#include <string>
void write_map(const std::map<int, std::string> &m, std::ostream &out) {
for (auto &a : m) {
out << a.first << " " << a.second << '\n';
}
}
std::map<int, std::string> read_map(std::istream &in) {
std::map<int, std::string> m;
int i;
std::string s;
while (in >> i >> s) {
m.emplace(i, s);
}
return m;
}
int main() {
std::map<int, std::string> m;
m.emplace(1, "CS");
m.emplace(2, "");
m.emplace(3, "Maicraft");
m.insert_or_assign(3, "GTA5");
std::ostringstream out;
write_map(m, out);
std::string data = out.str();
std::cout << "Data:\n" << data << std::endl;
// send data over socket
// ...
std::istringstream in(data);
auto m1 = read_map(in);
std::cout << "Read:\n";
write_map(m1, std::cout);
std::cout << std::endl;
}
I am pretty new to c++. I am trying to read a file in line by line and store the input into several arrays.
Because I don't know the size of input file, I have this to get the number of lines in the file
while (std::getline(inputFile, line)){
++numOfLines;
std::cout << line << std::endl;
}
Now I want to use the numOfLines as the size of arrays, but i cannot get it run by having this
std::string *firstName= new std::string[numOfLines];
std::string *lastName= new std::string[numOfLines];
for (int i = 0; i < numOfLines; ++i)
{
line >> firstName[i];
}
I guess it is because it has reached the end of the file after the while loop. But I do not know how to solve this problem. Is there a way to scan the input file in and store the value into array at the same time?
If you use std::vector you don't need to know ahead the lines count. You can use vector method push_back to insert new elements into it. Try use something like this:
#include <fstream>
#include <iostream>
#include <vector>
#include <string>
using namespace std;
int main()
{
std::vector<std::string> first_names;
std::string line;
ifstream input_file;
while (std::getline(input_file, line)){
first_names.push_back(line);
}
for (size_t i = 0; i < first_names.size(); i++) {
std::cout << first_names[i] << std::endl;
}
return 0;
}
I don't know if you have ever taken a course related to Data Structures & Algorithms,
in which you will learn to use Containers (such as:
vector,
deque,
list, etc.) instead of Primitive Data Structures.
Please notice that although the follow example chooses vector as its container, it could vary according to different contexts. Say you are handling gigantic mount of data, you might want to use list instead`1,2,3.
#include <fstream>
#include <iostream>
#include <vector>
#include <string>
// alias long type
// #see: https://en.cppreference.com/w/cpp/language/type_alias
using NameVector = std::vector<std::string>;
int handleLine(std::string line, NameVector &firstNames)
{
// TODO implement your line handler here
firstNames.push_back(line);
return 0;
}
int handleFile(std::ifstream inputFile, NameVector &firstNames)
{
std::string line;
for (int lineNum = 1;
// invoke `good` to check if there is any error
inputFile.good()
&&
std::getline(inputFile, line);
lineNum++)
{
std::cout << "Current line number : (" << lineNum << ")" << std::endl;
std::cout << "Current line content: (" << line << ")" << std::endl;
handleLine(line, &firstNames);
}
return 0;
}
int main()
{
std::string path; // = R"(HERE GOES YOUR FILE PATH)";
// Using **Raw string**
std::ifstream inputFile { path }; // Initialize `inputFile`
NameVector firstNames;
handleFile(inputFile, firstNames);
for (auto firstName : firstNames)
{
std::cout << firstName << std::endl;
}
return 0;
}
I want to create some text file in C++. For example: I will run a loop from 1 to 5 and create the following files:
1.txt
2.txt
3.txt
4.txt
5.txt
is it possible? I have made a sample code:
#include<iostream>
#include<cstdio>
#include<cstdlib>
using namespace std;
main()
{
FILE *fp;
int i;
for(i=1;i<=5;i++)
{
//fp=fopen("%d.txt","r",i); //what will go here??
}
}
I am confused about what I will write inside the loop. how can I create those files?
char i;
char fileName[] = "0.txt";
for(i='1';i<='5';i++)
{
fileName[0]=i;
fp=fopen(fileName,"r"); //what will go here??
//...
}
You can use sprintf if this is too simple for your case;
Since you tag c++, I think fstream string is the thing to use.
A simple c++ example
#include <fstream>
#include <string>
using namespace std;
int main(){
string base(".txt");
for(int i=1;i<=5;++i){
ofstream(to_string(i)+base);// to_string() need c++11
}
}
If you still don't have to_string (you don't have c++11 or your compiler just don't have this) you can use this simple version for now. (better put this in your own namespace)
#include <string>
#include <sstream>
std::string to_string(int i){
std::stringstream s;
s << i;
return s.str();
}
You can use a std::stringstream to compose the file name before passing it to the std::ofstream constructor as a std::string.
#include <iostream>
#include <fstream>
#include <string>
#include <sstream>
#include <iomanip>
int main()
{
std::cout << "How many files do you want to create? ";
int n;
std::cin >> n;
std::cout << "How many digits do you want to display? ";
int n_digits;
std::cin >> n_digits; // i.e. zeroes == 3 -> 001.txt
std::cout << "Enter a common prefix for all the files: ";
std::string prefix;
std::cin.ignore();
std::getline(std::cin, prefix); // i.e. prefix == "file" -> file001.txt
std::string ext(".txt");
for ( int i = 1; i <= n; ++i )
{ // use a stringstream to create a file names like: prefix001.txt
std::stringstream ss;
ss << prefix << std::setfill('0') << std::setw(n_digits) << i << ext;
// open the file. If not c++11 use ss.str().c_str() instead
std::ofstream file( ss.str() );
if ( !file )
{
std::cerr << "Error: failed to create file " << ss.str() << '\n';
break;
}
// write something to the newly created file
file << "This is file: " << ss.str() << "\n\nHello!\n";
if ( !file )
{
std::cerr << "Error: failed to write to file " << ss.str() << '\n';
break;
}
}
}
#include <iostream>
#include <fstream>
int main(void)
{
std::ofstream out; // you must call out.close() inside loop to be able to open another file for writting otherwise you'll get only the first one "a.txt"
std::string sFileName;
for(char c('a'); c < 'f'; c++)
{
sFileName = c;
sFileName += ".txt";
out.open(sFileName.c_str(), std::ios::out);
// std::ofstream out(sFileName.c_str(), std::ios::out); // here you are not obliged to call out.close() because the first out is not the very second and so on...
out.close(); // very important if you use the same ofstream to open another file
}
std::cout << std::endl;
return 0;
}
*** to be able to use one ostream object in opening many files you must close the precedent file to be able to open the next otherwise it fails trying creating the next one.
I have a string (of some fixed length), which I need to compress and then compare the compressed lengths (as a proxy for redundancy in the data or as a rough approximation to the Kolmogorov complexity). Currently, I am using boost::iostreams for compression, which seems working well. However, I don't know how to obtain the size of the compressed data. Can someone help, please?
The code snippet is
#include <boost/iostreams/filtering_streambuf.hpp>
#include <boost/iostreams/filtering_stream.hpp>
#include <boost/iostreams/copy.hpp>
#include <boost/iostreams/filter/gzip.hpp>
#include <boost/iostreams/device/file_descriptor.hpp>
#include <boost/filesystem.hpp>
#include <string>
#include <sstream>
namespace io = boost::iostreams;
int main() {
std::string memblock;
std::cout << "Input the string to be compressed:";
std::cin >> memblock;
std::cout << memblock << std::endl;
io::filtering_ostream out;
out.push(io::gzip_compressor());
out.push(io::file_descriptor_sink("test.gz"));
out.write (memblock.c_str(), memblock.size());
std::cout << out.size() << std::endl;
return 0;
}
You can try adding boost::iostreams::counter to you chain between the compressor and sink and then calling it's characters() member to get number of bytes that went through it.
This works for me:
#include <boost/iostreams/filter/counter.hpp>
...
io::filtering_ostream out;
out.push(io::counter());
out.push(io::gzip_compressor());
out.push(io::counter());
out.push(io::file_descriptor_sink("test.gz"));
out.write (memblock.c_str(), memblock.size());
io::close(out); // Needed for flushing the data from compressor
std::cout << "Wrote " << out.component<io::counter>(0)->characters() << " bytes to compressor, "
<< "got " << out.component<io::counter>(2)->characters() << " bytes out of it." << std::endl;
I figured out yet another (and slightly slicker) way to achieve the compressed length of a string. I thought sharing it here, but basically it is simply passing the uncompressed string to a filtered buffer and copying the output back to a string:
template<typename T>
inline std::string compressIt(std::vector<T> s){
std::stringstream uncompressed, compressed;
for (typename std::vector<T>::iterator it = s.begin();
it != s.end(); it++)
uncompressed << *it;
io::filtering_streambuf<io::input> o;
o.push(io::gzip_compressor());
o.push(uncompressed);
io::copy(o, compressed);
return compressed.str();
}
Later one can easily get the size of the compressed string as
compressIt(uncompressedString).size()
I feel it is better for it does not required me to create an output file as previously.
cheers,
Nikhil
one other way would be
stream<array_source> input_stream(input_data,input_data_ize);
stream<array_sink> compressed_stream(compressed_data,alloc_compressed_size);
filtering_istreambuf out;
out.push(gzip_compressor());
out.push(input_stream);
int compressed_size = copy(out,compressed_stream);
cout << "size of compressed_stream" << compressed_size << endl;