I have some data in a buffer pointed to by a const char* pointer. The data is just an ASCII string. I know its size. I would like to be able to read it in the same way data is read from streams. I'm looking for a solution that would allow me to write code like this:
// for example, data points to a string "42 3.14 blah"
MemoryStreamWrapper in(data, data_size);
int x;
float y;
std::string w;
in >> x >> y >> w;
Important condition: the data must not be copied or altered in any way (otherwise I'd just use a string stream. To my best knowledge, it isn't possible to create a string stream from a const char pointer without copying the data.)
The way to do this is to create a suitable stream buffer. This can, e.g., be done like this:
#include <streambuf>
#include <istream>
struct membuf: std::streambuf {
membuf(char const* base, size_t size) {
char* p(const_cast<char*>(base));
this->setg(p, p, p + size);
}
};
struct imemstream: virtual membuf, std::istream {
imemstream(char const* base, size_t size)
: membuf(base, size)
, std::istream(static_cast<std::streambuf*>(this)) {
}
};
The only somewhat awkward thing is the const_cast<char*>() in the stream buffer: the stream buffer won't change the data but the interface still requires char* to be used, mainly to make it easier to change the buffer in "normal" stream buffers. With this, you can use imemstream as a normal input stream:
imemstream in(data, size);
in >> value;
The only way would be to subclass std::istream (which also requires subclassing std::streambuf) to create your own stream class that reads from constant memory.
It's not as easy as it sounds because the the C++ standard library stream classes are pretty messy and badly designed. I don't think it's worth it unless you need it to scale a lot.
Related
I have data in stl containers (vector). Each node in the vector is a structure which also contains stl strings.
struct record
{
string name;
string location;
int salary;
}
vector< record > employees;
I want to serialize employees but I also want to encrypt it before serializing.
my encryption function looks like this:
Encode(const char * inBfr, const int in_size, char ** outBfr, int& out_size )
By searching it looks like the stl standard doesn't require the memory of my structure to be contiguous so I can't just grab the memory of employees variable. Is there any other smart way that I can use this encoding function with my stl based structures/container? It is good for me that Encode function works in plain char * buffers so I know exactly what goes in and out but stl structures are not and I am tring to find a nice way so I can use stl with this function.
I am also opening to using any other stl containers if that helps.
Although the element in the std::vector<T> are guaranteed to be laid out contiguously, this doesn't really help: the record you have may include padding and, more importantly, will store the std::string's content external to the std::string object (in case the small string optimization is used, the value may be embedded inside the std::string but it will also contain a couple of bytes which are not part of the std::strings value). Thus, you best option is to format your record and encrypt the formatted string.
The formatting is straight forward but personally I would encapsulate the encoding function into a simple std::streambuf so that the encryption can be done by a filtering stream buffer. Given the signature you gave, this could look something like this:
class encryptbuf
: public std::streambuf {
std::streambuf* d_sbuf;
char d_buffer[1024];
public:
encryptbuf(std::streambuf* sbuf)
: d_sbuf(sbuf) {
this->setp(this->d_buffer, this->d_buffer + sizeof(this->d_buffer) - 1);
}
int overflow(int c) {
if (c != std::char_traits<char>::eof()) {
*this->pptr() = std::char_traits<char>::to_char_type(c);
this->pbump(1);
}
return this->pubsync()? std::char_traits<char>::eof(): std::char_traits<char>::not_eof(c);
}
int sync() {
char* out(0);
int size(0);
Encode(this->pbase(), this->pptr() - this->pbase(), &out, size);
this->d_sbuf->sputn(out, size);
delete[] out; // dunno: it seems the output buffer is allocated but how?
this->setp(this->pbase(), this->epptr());
return this->d_sbuf->pubsync();
}
};
int main() {
encryptbuf sbuf(std::cout.rdbuf());
std::ostream eout(&sbuf);
eout << "print something encoded to standard output\n" << std::flush;
}
Now, creating an output operator for your records just printing to an std::ostream can be used to create an encoded
It's probably easiest to serialize your structure into a string, then encrypt the string. For example:
std::ostringstream buffer;
buffer << a_record.name << "\n" << a_record.location << "\n" << a_record.salary;
encode(buffer.str().c_str(), buffer.str().length(), /* ... */);
If it were me, I'd probably write encode (or at least a wrapper for it) to take input (and probably produce output) in a vector, string, or stream though.
If you want to get ambitious, there are other possibilities. First of all, #MooingDuck raises a good point that it's often worthwhile to overload operator<< for the class, instead of working with the individual items all the time. This will typically be a small function similar to what's above:
std::ostream &operator<<(std::ostream &os, record const &r) {
return os << r.name << "\n" << r.location << "\n" << r.salary;
}
Using this, you'd just have:
std::ostringstream os;
os << a_record;
encode(os.str().c_str(), os.str().length(), /* ... */);
Second, if you want to get really ambitious, you can put the encryption into (for one example) a codecvt facet, so you can automatically encrypt all the data as you write it to a stream, and decrypt it as you read it back in. Another possibility is to build the encryption into a filtering streambuf object instead. The codecvt facet is probably the method that should theoretically be preferred, but the streambuf is almost certainly easier to implement, with less unrelated "stuff" involved.
Is there a C/C++ way to read data from a socket using read() and having the receiving buffer be a file (ofstream) or a similar self-extending object (vector e.g.)?
EDIT: The question arose while I contemplated how to read a stream socket that may receive the contents of a, say 10000+ byte file. I just never did like putting 20000 or 50000 bytes (large enough for now) on the stack as a buffer where the file could be stored temporarily till I could stick in into a file. Why not just stream it directly into the file to star with.
Much like you can get at the char* inside a std:string, I thought of something like
read( int fd, outFile.front(), std::npos ); // npos = INT_MAX
or something like that.
end edit
Thanks.
This is simplistic, and off the top of my fingers, but I think something along these lines would work out:
template <unsigned BUF_SIZE>
struct Buffer {
char buf_[BUF_SIZE];
int len_;
Buffer () : buf_(), len_(0) {}
int read (int fd) {
int r = read(fd, buf_ + len_, BUF_SIZE - len_);
if (r > 0) len_ += r;
return r;
}
int capacity () const { return BUF_SIZE - len_; }
}
template <unsigned BUF_SIZE>
struct BufferStream {
typedef std::unique_ptr< Buffer<BUF_SIZE> > BufferPtr;
std::vector<BufferPtr> stream_;
BufferStream () : stream_(1, BufferPtr(new Buffer<BUF_SIZE>)) {}
int read (int fd) {
if ((*stream_.rbegin())->capacity() == 0)
stream_.push_back(BufferPtr(new Buffer<BUF_SIZE>));
return (*stream_.rbegin())->read(fd);
}
};
In a comment, you mentioned you wanted to avoid creating a big char buffer. When using the read system call, it is generally more efficient to perform a few large reads rather than many small ones. So most implementations will opt for large input buffers to gain that efficiency. You could implement something like:
std::vector<char> input;
char in;
int r;
while ((r = read(fd, &in, 1)) == 1) input.push_back(in);
But that would involve a system call and at least one byte copied for every byte of input. In contrast, the code I put forth avoids extra data copies.
I don't really expect the code I put out to be the solution you would adopt. I just wanted to provide you with an illustration of how to create a self-extending object that was fairly space and time efficient. Depending on your purposes, you may want to extend it, or write your own. Off the top of my head, some improvements may be:
use std::list instead, to avoid vector resizing
allow API a parameter to specify how many bytes to read
use readv to always allow at least BUF_SIZE bytes (or more than BUF_SIZE bytes) to be read at a time
Take a look at stream support in boost::asio.
Every one know stringstream.str() need a string variable type to store the content of stringstream.str() into it .
I want to store the content of stringstream.str() into char variable or char array or pointer.
Is it possible to do that?
Please, write a simple example with your answer.
Why not just
std::string s = stringstream.str();
const char* p = s.c_str();
?
Edit: Note that you cannot freely give the p outside your function: its lifetime is bound to the lifetime of s, so you may want to copy it.
Edit 2: as #David suggests, copy above means copying of the content, not the pointer itself. There are several ways for that. You can either do it manually (legacy way "inherited" from C) -- this is done with the functions like std::strcpy. This way is quite complicated, since it involves manual resources management, which is usually discouraged, since it leads to a more complicated and error-prone code. Or you can use the smart pointers or containers: it can be either std::vector<char> or std::unique_ptr/std::shared_ptr.
I personally would go for the second way. See the discussion to this and #Oli's answer, it can be useful.
If you want to get the data into a char buffer, why not put it there immediately anyway? Here is a stream class which takes an array, determines its size, fills it with null characters (primarily to make sure the resulting string is null terminated), and then sets up an std::ostream to write to this buffer directly.
#include <iostream>
#include <algorithm>
struct membuf: public std::streambuf {
template <size_t Size> membuf(char (&array)[Size]) {
this->setp(array, array + Size - 1);
std::fill_n(array, Size, 0);
}
};
struct omemstream: virtual membuf, std::ostream {
template <size_t Size> omemstream(char (&array)[Size]):
membuf(array),
std::ostream(this)
{
}
};
int main() {
char array[20];
omemstream out(array);
out << "hello, world";
std::cout << "the buffer contains '" << array << "'\n";
}
Obviously, this stream buffer and stream would probably live in a suitable namespace and would be implemented in some header (there isn't much point in putting anything of it into a C++ file because all the function are templates needing to instantiated). You could also use the [deprecated] class std::ostrstream to do something similar but it is so easy to create a custom stream that it may not worth bothering.
You can do this if you want an actual copy of the string (vital if the stringstream object is going to go out of scope at some point):
const char *p = new char[ss.str().size()+1];
strcpy(p, ss.str().c_str());
...
delete [] p;
As discussed in comments below, you should be wary of doing it like this (manual memory management is error-prone, and very non-idiomatic C++). Why do you want a raw char array?
I figured it out. Using namespace std and replacing tstingstreamwith stringstream. Next step is:
stringstream strstream;
strstream.imbue(std::locale("C"));
string str = strstream.str();
const char *sql= str .c_str();
Now you can execute sql statement.
sqlite3_exec(db, sql, callback, (void*)data, &zErrMsg);
Maybe it helps to somebody.
I have my struct:
struct a
{
int x;
float f;
double d;
char c;
char s[50];
};
and I wish append each time into my timer schedule into a binary file.
// declaration
std::ofstream outFile;
// constructor:
outFile.open( "save.dat", ios::app );
// tick:
outFile << a << endl;
but inside the save.dat appears only this:
0C3A0000..0C3A0000..0C3A0000..0C3A0000..0C3A0000..0C3A0000..0C3A0000..0C3A0000..0C3A0000..
thanks in advance
What you're currently doing is writing the address of the struct definition.
What you want to do is use ostream::write
outfile.write(reinterpret_cast<char*>(&myStruct), sizeof(a));
This will work as long as your struct is a POD (Plain Old Data) type (which your example is). POD type means that all members are of fixed size.
If you on the other hand have variable sized members then you would need to write out each member one by one.
A sensible way to serialize custom objects is to overload your own output stream operator:
std::ostream & operator<<(std::ostream & o, const a & x)
{
o.write(reinterpret_cast<char*>(&x.x), sizeof(int));
o.write(reinterpret_cast<char*>(&x.f), sizeof(float));
/* ... */
return o;
}
a x;
std::ofstream ofile("myfile.bin", std::ios::binary | std::ios::app);
ofile << a;
This is still platform-dependent, so to be a bit safer, you should probably use fixed-width data types like int32_t etc.
It might also not be the best idea semantically to use << for binary output, since it's often used for formatted output. Perhaps a slightly safer method would be to write a function void serialize(const a &, std::ostream &);
Hello! My problem can be described the following way:
I have some data which actually is an array and could be represented as char* data with some size
I also have some legacy code (function) that takes some abstract std::istream object as a param and uses that stream to retrieve data to operate.
So, my question is the following - what would be the easy way to map my data to some std::istream object so that I can pass it to my function? I thought about creating a std::stringstream object from my data, but that means copying and (as I assume) isn't the best solution.
Any ideas how this could be done so that my std::istream operates on the data directly?
Thank you.
If you're looking at actually creating your own stream, I'd look at the Boost.Iostreams library. It makes it easy to create your own stream objects.
Definitely not the easiest way but just in case someone wanted to understand how std streams work inside, this seems to be a very nice introduction about how you can roll your own:
http://www.mr-edd.co.uk/blog/beginners_guide_streambuf
Use string stream:
#include <sstream>
int main()
{
char[] data = "PLOP PLOP PLOP";
int size = 13; // PS I know this is not the same as strlen(data);
std::stringstream stream(std::string(data, size));
// use stream as an istream;
}
If you want to be real effecient you can muck with the stream buffer directly. I have not tried this and do not have a compiler to test with, but the folowing should work:
#include <sstream>
int main()
{
char[] data = "PLOP PLOP PLOP";
int size = 13; // PS I know this is not the same as strlen(data);
std::stringstream stream;
stream.rdbuf()->pubsetbuf(data, size);
// use stream as an istream;
}