Related
Hi i have a client program and a server program.
I created a struct like this
struct player
{
Int x,y;
Int score;
Std::string name;
}
I created the object with:
player p;
And i initialized it:
p.x = 100; p.y = 200; p.score = 281;
p.name = "DiCri";
I sent the object to the server.
SDLNet_TCP_Send(client, (char*)&p, sizeof(p));
All works fine.
The server gets the object with data.
The only problem is the name.
Server says that x is 100, y is 200, score is 281, and name is a sort of strange random symbols.
I don't know why.
Help. How to fix that?
This happens also if that was a char* and not a string.
Thanks
EDIT1:
I found a question similar to mine: Serialization of an object.
And the user who asked this question wants to send this object over the network too.
I'll try to follow the answers
EDIT2:
using char name[100] it works.
EDIT3:
Thanks now all works fine! Sorry for bad english because i'm italian!
Player contains 3 ints and a std::string This ints can easily be written to a stream with write (watch out for the differing byte orders, endian, used by processors) , but the string is too complex an object. Option 1 is to replace the string with a fixed size char array to simplify the structure, but this adds more pain than it's worth. The char array can easily be overflowed, and always writes the full size of the array whether you used it or not.
The better approach is to establish a communication protocol and serialize the data.
First establish the endian to be used by the protocol so that both sides know exactly which byte order is used. Traditionally Big Endian is used for network communications, but there are fewer and fewer big endian devices, so this is increasingly becoming a case of, "Your call."
Let's stick with big endian because the tools for this are ancient, well known, and well established. If you are using a socket library odds are very good you have access to tools to perform the operations required.
Next, explicitly size the integer data types. Different implementations of C++ may have different sizes for fundamental types. Solution for this one is simple: Don't use the fundamental types. Instead, use the fixed width integers defined in in cstdint
Right now our structure looks something like
struct player
{
int x,y;
int score;
std::string name;
}
It needs to look more like
struct player
{
int32_t x,y;
int32_t score;
std::string name;
}
Now there are no surprises about the size of an integer. The code either compiles or does not support int32_t. Stop laughing. There are still 8 bit micro controllers out there.
Now that we know what the integers look like, we can make a pair of functions to handle reading and writing:
bool sendInt32(int32_t val)
{
int32_t temp = htonl(val);
int result = SDLNet_TCP_Send(client, (char*)&temp, sizeof(temp));
return result == sizeof(temp); // this is simplified. Could do a lot more here
}
bool readInt32(int32_t & val)
{
int32_t temp;
if (readuntil(client, (char*)&temp, sizeof(temp)))
{
val = ntohl(temp);
return true;
}
return false;
}
Where readuntil is a function that keeps reading data from the socket until the socket either fails or all of the requested data has been read. Do NOT assume that all of the data you wanted will arrive all to the same time. Sometimes you have to wait. Even if you wrote it all in one chunk it won't necessarily arrive all in one chunk. Dealing with this is another question, but it gets asked just about weekly so you should be able to find a good answer with a bit of searching.
Now for the string. You go the C route and send a null terminated string, but I find this makes the reader much more complicated than it needs to be. Instead I prefix the string data with the length so all the reader has to do is read the length and then read length bytes.
Basically this:
bool sendstring(const std::string & str)
{
if (sendInt32(str.size()))
{
int result = SDLNet_TCP_Send(client, str.c_str(), str.size());
return result == str.size();
}
return false;
}
bool readstring(std::string & str)
{
int32 len;
if (readInt32(len))
{
str.resize(len);
return readuntil(client, str.data(), len) == len;
}
return false;
}
All together, a writer looks something like
bool writePlayer(const Player & p)
{
return sendInt32(p.x) &&
sendInt32(p.y) &&
sendInt32(p.score) &&
sendString(p.name);
}
and a reader
bool readPlayer(Player & p)
{
return readInt32(p.x) &&
readInt32(p.y) &&
readInt32(p.score) &&
readString(p.name);
}
First of all, Std::string should be replaced by array of char.
You also aware about network repsentation of Int on different OS. htons() and ntohs must be used for sending a 16bits integer over Networking.
I have a binary file with some layout I know. For example let format be like this:
2 bytes (unsigned short) - length of a string
5 bytes (5 x chars) - the string - some id name
4 bytes (unsigned int) - a stride
24 bytes (6 x float - 2 strides of 3 floats each) - float data
The file should look like (I added spaces for readability):
5 hello 3 0.0 0.1 0.2 -0.3 -0.4 -0.5
Here 5 - is 2 bytes: 0x05 0x00. "hello" - 5 bytes and so on.
Now I want to read this file. Currently I do it so:
load file to ifstream
read this stream to char buffer[2]
cast it to unsigned short: unsigned short len{ *((unsigned short*)buffer) };. Now I have length of a string.
read a stream to vector<char> and create a std::string from this vector. Now I have string id.
the same way read next 4 bytes and cast them to unsigned int. Now I have a stride.
while not end of file read floats the same way - create a char bufferFloat[4] and cast *((float*)bufferFloat) for every float.
This works, but for me it looks ugly. Can I read directly to unsigned short or float or string etc. without char [x] creating? If no, what is the way to cast correctly (I read that style I'm using - is an old style)?
P.S.: while I wrote a question, the more clearer explanation raised in my head - how to cast arbitrary number of bytes from arbitrary position in char [x]?
Update: I forgot to mention explicitly that string and float data length is not known at compile time and is variable.
If it is not for learning purpose, and if you have freedom in choosing the binary format you'd better consider using something like protobuf which will handle the serialization for you and allow to interoperate with other platforms and languages.
If you cannot use a third party API, you may look at QDataStream for inspiration
Documentation
Source code
The C way, which would work fine in C++, would be to declare a struct:
#pragma pack(1)
struct contents {
// data members;
};
Note that
You need to use a pragma to make the compiler align the data as-it-looks in the struct;
This technique only works with POD types
And then cast the read buffer directly into the struct type:
std::vector<char> buf(sizeof(contents));
file.read(buf.data(), buf.size());
contents *stuff = reinterpret_cast<contents *>(buf.data());
Now if your data's size is variable, you can separate in several chunks. To read a single binary object from the buffer, a reader function comes handy:
template<typename T>
const char *read_object(const char *buffer, T& target) {
target = *reinterpret_cast<const T*>(buffer);
return buffer + sizeof(T);
}
The main advantage is that such a reader can be specialized for more advanced c++ objects:
template<typename CT>
const char *read_object(const char *buffer, std::vector<CT>& target) {
size_t size = target.size();
CT const *buf_start = reinterpret_cast<const CT*>(buffer);
std::copy(buf_start, buf_start + size, target.begin());
return buffer + size * sizeof(CT);
}
And now in your main parser:
int n_floats;
iter = read_object(iter, n_floats);
std::vector<float> my_floats(n_floats);
iter = read_object(iter, my_floats);
Note: As Tony D observed, even if you can get the alignment right via #pragma directives and manual padding (if needed), you may still encounter incompatibility with your processor's alignment, in the form of (best case) performance issues or (worst case) trap signals. This method is probably interesting only if you have control over the file's format.
Currently I do it so:
load file to ifstream
read this stream to char buffer[2]
cast it to unsigned short: unsigned short len{ *((unsigned short*)buffer) };. Now I have length of a string.
That last risks a SIGBUS (if your character array happens to start at an odd address and your CPU can only read 16-bit values that are aligned at an even address), performance (some CPUs will read misaligned values but slower; others like modern x86s are fine and fast) and/or endianness issues. I'd suggest reading the two characters then you can say (x[0] << 8) | x[1] or vice versa, using htons if needing to correct for endianness.
read a stream to vector<char> and create a std::string from this vector. Now I have string id.
No need... just read directly into the string:
std::string s(the_size, ' ');
if (input_fstream.read(&s[0], s.size()) &&
input_stream.gcount() == s.size())
...use s...
the same way read next 4 bytes and cast them to unsigned int. Now I have a stride.
while not end of file read floats the same way - create a char bufferFloat[4] and cast *((float*)bufferFloat) for every float.
Better to read the data directly over the unsigned ints and floats, as that way the compiler will ensure correct alignment.
This works, but for me it looks ugly. Can I read directly to unsigned short or float or string etc. without char [x] creating? If no, what is the way to cast correctly (I read that style I'm using - is an old style)?
struct Data
{
uint32_t x;
float y[6];
};
Data data;
if (input_stream.read((char*)&data, sizeof data) &&
input_stream.gcount() == sizeof data)
...use x and y...
Note the code above avoids reading data into potentially unaligned character arrays, wherein it's unsafe to reinterpret_cast data in a potentially unaligned char array (including inside a std::string) due to alignment issues. Again, you may need some post-read conversion with htonl if there's a chance the file content differs in endianness. If there's an unknown number of floats, you'll need to calculate and allocate sufficient storage with alignment of at least 4 bytes, then aim a Data* at it... it's legal to index past the declared array size of y as long as the memory content at the accessed addresses was part of the allocation and holds a valid float representation read in from the stream. Simpler - but with an additional read so possibly slower - read the uint32_t first then new float[n] and do a further read into there....
Practically, this type of approach can work and a lot of low level and C code does exactly this. "Cleaner" high-level libraries that might help you read the file must ultimately be doing something similar internally....
I actually implemented a quick and dirty binary format parser to read .zip files (following Wikipedia's format description) just last month, and being modern I decided to use C++ templates.
On some specific platforms, a packed struct could work, however there are things it does not handle well... such as fields of variable length. With templates, however, there is no such issue: you can get arbitrarily complex structures (and return types).
A .zip archive is relatively simple, fortunately, so I implemented something simple. Off the top of my head:
using Buffer = std::pair<unsigned char const*, size_t>;
template <typename OffsetReader>
class UInt16LEReader: private OffsetReader {
public:
UInt16LEReader() {}
explicit UInt16LEReader(OffsetReader const or): OffsetReader(or) {}
uint16_t read(Buffer const& buffer) const {
OffsetReader const& or = *this;
size_t const offset = or.read(buffer);
assert(offset <= buffer.second && "Incorrect offset");
assert(offset + 2 <= buffer.second && "Too short buffer");
unsigned char const* begin = buffer.first + offset;
// http://commandcenter.blogspot.fr/2012/04/byte-order-fallacy.html
return (uint16_t(begin[0]) << 0)
+ (uint16_t(begin[1]) << 8);
}
}; // class UInt16LEReader
// Declined for UInt[8|16|32][LE|BE]...
Of course, the basic OffsetReader actually has a constant result:
template <size_t O>
class FixedOffsetReader {
public:
size_t read(Buffer const&) const { return O; }
}; // class FixedOffsetReader
and since we are talking templates, you can switch the types at leisure (you could implement a proxy reader which delegates all reads to a shared_ptr which memoizes them).
What is interesting, though, is the end-result:
// http://en.wikipedia.org/wiki/Zip_%28file_format%29#File_headers
class LocalFileHeader {
public:
template <size_t O>
using UInt32 = UInt32LEReader<FixedOffsetReader<O>>;
template <size_t O>
using UInt16 = UInt16LEReader<FixedOffsetReader<O>>;
UInt32< 0> signature;
UInt16< 4> versionNeededToExtract;
UInt16< 6> generalPurposeBitFlag;
UInt16< 8> compressionMethod;
UInt16<10> fileLastModificationTime;
UInt16<12> fileLastModificationDate;
UInt32<14> crc32;
UInt32<18> compressedSize;
UInt32<22> uncompressedSize;
using FileNameLength = UInt16<26>;
using ExtraFieldLength = UInt16<28>;
using FileName = StringReader<FixedOffsetReader<30>, FileNameLength>;
using ExtraField = StringReader<
CombinedAdd<FixedOffsetReader<30>, FileNameLength>,
ExtraFieldLength
>;
FileName filename;
ExtraField extraField;
}; // class LocalFileHeader
This is rather simplistic, obviously, but incredibly flexible at the same time.
An obvious axis of improvement would be to improve chaining since here there is a risk of accidental overlaps. My archive reading code worked the first time I tried it though, which was evidence enough for me that this code was sufficient for the task at hand.
I had to solve this problem once. The data files were packed FORTRAN output. Alignments were all wrong. I succeeded with preprocessor tricks that did automatically what you are doing manually: unpack the raw data from a byte buffer to a struct. The idea is to describe the data in an include file:
BEGIN_STRUCT(foo)
UNSIGNED_SHORT(length)
STRING_FIELD(length, label)
UNSIGNED_INT(stride)
FLOAT_ARRAY(3 * stride)
END_STRUCT(foo)
Now you can define these macros to generate the code you need, say the struct declaration, include the above, undef and define the macros again to generate unpacking functions, followed by another include, etc.
NB I first saw this technique used in gcc for abstract syntax tree-related code generation.
If CPP is not powerful enough (or such preprocessor abuse is not for you), substitute a small lex/yacc program (or pick your favorite tool).
It's amazing to me how often it pays to think in terms of generating code rather than writing it by hand, at least in low level foundation code like this.
You should better declare a structure (with 1-byte padding - how - depends on compiler). Write using that structure, and read using same structure. Put only POD in structure, and hence no std::string etc. Use this structure only for file I/O, or other inter-process communication - use normal struct or class to hold it for further use in C++ program.
Since all of your data is variable, you can read the two blocks separately and still use casting:
struct id_contents
{
uint16_t len;
char id[];
} __attribute__((packed)); // assuming gcc, ymmv
struct data_contents
{
uint32_t stride;
float data[];
} __attribute__((packed)); // assuming gcc, ymmv
class my_row
{
const id_contents* id_;
const data_contents* data_;
size_t len;
public:
my_row(const char* buffer) {
id_= reinterpret_cast<const id_contents*>(buffer);
size_ = sizeof(*id_) + id_->len;
data_ = reinterpret_cast<const data_contents*>(buffer + size_);
size_ += sizeof(*data_) +
data_->stride * sizeof(float); // or however many, 3*float?
}
size_t size() const { return size_; }
};
That way you can use Mr. kbok's answer to parse correctly:
const char* buffer = getPointerToDataSomehow();
my_row data1(buffer);
buffer += data1.size();
my_row data2(buffer);
buffer += data2.size();
// etc.
I personally do it this way:
// some code which loads the file in memory
#pragma pack(push, 1)
struct someFile { int a, b, c; char d[0xEF]; };
#pragma pack(pop)
someFile* f = (someFile*) (file_in_memory);
int filePropertyA = f->a;
Very effective way for fixed-size structs at the start of the file.
Use a serialization library. Here are a few:
Boost serialization and Boost fusion
Cereal (my own library)
Another library called cereal (same name as mine but mine predates theirs)
Cap'n Proto
The Kaitai Struct library provides a very effective declarative approach, which has the added bonus of working across programming languages.
After installing the compiler, you will want to create a .ksy file that describes the layout of your binary file. For your case, it would look something like this:
# my_type.ksy
meta:
id: my_type
endian: be # for big-endian, or "le" for little-endian
seq: # describes the actual sequence of data one-by-one
- id: len
type: u2 # unsigned short in C++, two bytes
- id: my_string
type: str
size: 5
encoding: UTF-8
- id: stride
type: u4 # unsigned int in C++, four bytes
- id: float_data
type: f4 # a four-byte floating point number
repeat: expr
repeat-expr: 6 # repeat six times
You can then compile the .ksy file using the kaitai struct compiler ksc:
# wherever the compiler is installed
# -t specifies the target language, in this case C++
/usr/local/bin/kaitai-struct-compiler my_type.ksy -t cpp_stl
This will create a my_type.cpp file as well as a my_type.h file, which you can then include in your C++ code:
#include <fstream>
#include <kaitai/kaitaistream.h>
#include "my_type.h"
int main()
{
std::ifstream ifs("my_data.bin", std::ifstream::binary);
kaitai::kstream ks(&ifs);
my_type_t obj(&ks);
std::cout << obj.len() << '\n'; // you can now access properties of the object
return 0;
}
Hope this helped! You can find the full documentation for Kaitai Struct here. It has a load of other features and is a fantastic resource for binary parsing in general.
I use ragel tool to generate pure C procedural source code (no tables) for microcontrollers with 1-2K of RAM. It did not use any file io, buffering, and produces both easy to debug code and .dot/.pdf file with state machine diagram.
ragel can also output go, Java,.. code for parsing, but I did not use these features.
The key feature of ragel is the ability to parse any byte-build data, but you can't dig into bit fields. Other problem is ragel able to parse regular structures but has no recursion and syntax grammar parsing.
I have a struct and I would like to write it to a binary file (c++ / visual studio 2008).
The struct is:
struct DataItem
{
std::string tag;
std::vector<int> data_block;
DataItem(): data_block(1024 * 1024){}
};
I am filling tha data_block vector with random values:
DataItem createSampleData ()
{
DataItem data;
std::srand(std::time(NULL));
std::generate(data.data_block.begin(), data.data_block.end(), std::rand);
data.tag = "test";
return data;
}
And trying to write the struct to file:
void writeData (DataItem data, long fileName)
{
ostringstream ss;
ss << fileName;
string s(ss.str());
s += ".bin";
char szPathedFileName[MAX_PATH] = {0};
strcat(szPathedFileName,ROOT_DIR);
strcat(szPathedFileName,s.c_str());
ofstream f(szPathedFileName, ios::out | ios::binary | ios::app);
// ******* first I tried to write this way then one by one
//f.write(reinterpret_cast<char *>(&data), sizeof(data));
// *******************************************************
f.write(reinterpret_cast<const char *>(&data.tag), sizeof(data.tag));
f.write(reinterpret_cast<const char *>(&data.data_block), sizeof(data.data_block));
f.close();
}
And the main is:
int main()
{
DataItem data = createSampleData();
for (int i=0; i<5; i++) {
writeData(data,i);
}
}
So I expect a file size at least (1024 * 1024) * 4 (for vector)+ 48 (for tag) but it just writes the tag to the file and creates 1KB file to hard drive.
I can see the contents in while I'm debugging but it doesn't write it to file...
What's wrong with this code, why can't I write the strcut to vector to file? Is there a better/faster or probably efficient way to write it?
Do I have to serialize the data?
Thanks...
Casting a std::string to char * will not produce the result you expect. Neither will using sizeof on it. The same for a std::vector.
For the vector you need to use either the std::vector::data method, or using e.g. &data.data_block[0]. As for the size, use data.data_block.size() * sizeof(int).
Writing the string is another matter though, especially if it can be of variable length. You either have to write it as a fixed-length string, or write the length (in a fixed-size format) followed by the actual string, or write a terminator at the end of the string. To get a C-style pointer to the string use std::string::c_str.
Welcome to the merry world of C++ std::
Basically, vectors are meant to be used as opaque containers.
You can forget about reinterpret_cast right away.
Trying to shut the compiler up will allow you to create an executable, but it will produce silly results.
Basically, you can forget about most of the std::vector syntactic sugar that has to do with iterators, since your fstream will not access binary data through them (it would output a textual representation of your data).
But all is not lost.
You can access the vector underlying array using the newly (C++11) introduced .data() method, though that defeats the point of using an opaque type.
const int * raw_ptr = data.data_block.data();
that will gain you 100 points of cool factor instead of using the puny
const int * raw_ptr = &data.data_block.data[0];
You could also use the even more cryptic &data.data_block.front() for a cool factor bonus of 50 points.
You can then write your glob of ints in one go:
f.write (raw_ptr, sizeof (raw_ptr[0])*data.data_block.size());
Now if you want to do something really too simple, try this:
for (int i = 0 ; i != data.data_block.size() ; i++)
f.write (&data.data_block[i], sizeof (data.data_block[i]));
This will consume a few more microseconds, which will be lost in background noise since the disk I/O will take much more time to complete the write.
Totally not cool, though.
I need to create a byte array that is needed to be stream to another device through UART. There are some fixed parameters that I can fill in before hand but variables such as string is dynamically sized. Right up till now, I've been doing:
unsigned char buffer[255];
unsigned char wr_head = 0;
buffer[wr_head++] = 0x01; // and so on
memcpy(&buffer[wr_head], &some_chararray, sizeof(some_chararray));
wr_head += some_chararray;
I've experimented with other methods like std::string and std::vector but I felt that there is much manageable way of writing byte array for streams. Suggestions?
edit: Please advice on performance as well because is threaded.
edit2: Sorry for lacking of details the first time around. The device is indeed an embedded device. Though some suggested some solution, its not really what I want. Maybe a snippet of my current implementation will clear some confusion:
unsigned char buffer[255];
unsigned char wr_head = 0;
buffer[wr_head++] = 0x01; // Set message type
buffer[wr_head++] = 0x30; // message length
memcpy(&buffer[wr_head], &some_chararray, sizeof(some_chararray));
wr_head += some_chararray;
buffer[wr_head++] = CalChecksum;
UartSend(&buffer, wr_head); // Send array to stream out from UART
The configuration and setting value is known before hand, provided by the device documentation. This question is related to what I've asked in here
Thanks for the effort so far.
A ring buffer is a typical solution for problems like these.
I have no idea what kind of device you're on, but I'll just suppose that you're writing for some kind of embedded device. Let's assume that there's some interrupt moving data from the ring buffer to the UART. This interrupt will call getc, other code will call putc and puts.
class RingBuffer {
private:
static unsigned BUFSZ = 256;
volatile unsigned char buf[BUFSZ];
volatile unsigned char read, write;
public:
RingBuffer() : read(0), write(0) { }
// Blocks until space is available
void putc(unsigned int c) {
while (((write - read) & (BUFSZ - 1)) == 1)
sleep();
buf[write++ & (BUFSZ - 1)] = c;
}
// Returns -1 if empty
int getc() {
if (read == write)
return -1;
return buf[read++ & (BUFSZ - 1)];
}
// There are faster ways to write this.
void puts(char *str) {
for (; *str; ++str)
putc(*str);
}
};
Typically, you don't want to make the buffer dynamically grow for something like this. There's lots of room for improvement in the above code, and there are also libraries available for this kind of thing.
This particular implementation also never lets you fill the buffer completely, but the code is simpler as a result. I probably wouldn't put this code in production, but hopefully it's a step in the right direction.
If UartSend is a blocking function then you can do just this:
void UartSend(byte b) { UartSend(&b, 1); } // sends one byte
UartSend(0x01); // Set message type
UartSend(0x30); // message length
UartSend(some_chararray,sizeof(some_chararray));
I have a binary file that was created on a unix machine. It's just a bunch of records written one after another. The record is defined something like this:
struct RECORD {
UINT32 foo;
UINT32 bar;
CHAR fooword[11];
CHAR barword[11];
UNIT16 baz;
}
I am trying to figure out how I would read and interpret this data on a Windows machine. I have something like this:
fstream f;
f.open("file.bin", ios::in | ios::binary);
RECORD r;
f.read((char*)&detail, sizeof(RECORD));
cout << "fooword = " << r.fooword << endl;
I get a bunch of data, but it's not the data I expect. I'm suspect that my problem has to do with the endian difference of the machines, so I've come to ask about that.
I understand that multiple bytes will be stored in little-endian on windows and big-endian in a unix environment, and I get that. For two bytes, 0x1234 on windows will be 0x3412 on a unix system.
Does endianness affect the byte order of the struct as a whole, or of each individual member of the struct? What approaches would I take to convert a struct created on a unix system to one that has the same data on a windows system? Any links that are more in depth than the byte order of a couple bytes would be great, too!
As well as the endian, you need to be aware of padding differences between the two platforms. Particularly if you have odd length char arrays and 16 bit values, you may well find different numbers of pad bytes between some elements.
Edit: if the structure was written out with no packing, then it should be fairly straightforward. Something like this (untested) code should do the job:
// Functions to swap the endian of 16 and 32 bit values
inline void SwapEndian(UINT16 &val)
{
val = (val<<8) | (val>>8);
}
inline void SwapEndian(UINT32 &val)
{
val = (val<<24) | ((val<<8) & 0x00ff0000) |
((val>>8) & 0x0000ff00) | (val>>24);
}
Then, once you've loaded the struct, just swap each element:
SwapEndian(r.foo);
SwapEndian(r.bar);
SwapEndian(r.baz);
Actually, endianness is a property of the underlying hardware, not the OS.
The best solution is to convert to a standard when writing the data -- Google for "network byte order" and you should find the methods to do this.
Edit: here's the link: http://www.gnu.org/software/hello/manual/libc/Byte-Order.html
Don't read directly into struct from a file! The packing might be different, you have to fiddle with pragma pack or similar compiler specific constructs. Too unreliable. A lot of programmers get away with this since their code isn't compiled in wide number of architectures and systems, but that doesn't mean it's OK thing to do!
A good alternative approach is to read the header, whatever, into a buffer and parse from three to avoid the I/O overhead in atomic operations like reading a unsigned 32 bit integer!
char buffer[32];
char* temp = buffer;
f.read(buffer, 32);
RECORD rec;
rec.foo = parse_uint32(temp); temp += 4;
rec.bar = parse_uint32(temp); temp += 4;
memcpy(&rec.fooword, temp, 11); temp += 11;
memcpy(%red.barword, temp, 11); temp += 11;
rec.baz = parse_uint16(temp); temp += 2;
The declaration of parse_uint32 would look like this:
uint32 parse_uint32(char* buffer)
{
uint32 x;
// ...
return x;
}
This is a very simple abstraction, it doesn't cost any extra in practise to update the pointer as well:
uint32 parse_uint32(char*& buffer)
{
uint32 x;
// ...
buffer += 4;
return x;
}
The later form allows cleaner code for parsing the buffer; the pointer is automatically updated when you parse from the input.
Likewise, memcpy could have a helper, something like:
void parse_copy(void* dest, char*& buffer, size_t size)
{
memcpy(dest, buffer, size);
buffer += size;
}
The beauty of this kind of arrangement is that you can have namespace "little_endian" and "big_endian", then you can do this in your code:
using little_endian;
// do your parsing for little_endian input stream here..
Easy to switch endianess for the same code, though, rarely needed feature.. file-formats usually have a fixed endianess anyway.
DO NOT abstract this into class with virtual methods; would just add overhead, but feel free to if so inclined:
little_endian_reader reader(data, size);
uint32 x = reader.read_uint32();
uint32 y = reader.read_uint32();
The reader object would obviously just be a thin wrapper around pointer. The size parameter would be for error checking, if any. Not really mandatory for the interface per-se.
Notice how the choise of endianess here was done at COMPILATION TIME (since we create little_endian_reader object), so we invoke the virtual method overhead for no particularly good reason, so I wouldn't go with this approach. ;-)
At this stage there is no real reason to keep the "fileformat struct" around as-is, you can organize the data to your liking and not necessarily read it into any specific struct at all; after all, it's just data. When you read files like images, you don't really need the header around.. you should have your image container which is same for all file types, so the code to read a specific format should just read the file, interpret and reformat the data & store the payload. =)
I mean, does this look complicated?
uint32 xsize = buffer.read<uint32>();
uint32 ysize = buffer.read<uint32>();
float aspect = buffer.read<float>();
The code can look that nice, and be a really low-overhead! If the endianess is same for file and architecture the code is compiled for, the innerloop can look like this:
uint32 value = *reinterpret_cast<uint32*>)(ptr); ptr += 4;
return value;
That might be illegal on some architectures, so that optimization might be a Bad Idea, and use slower, but more robust approach:
uint32 value = ptr[0] | (static_cast<uint32>(ptr[1]) << 8) | ...; ptr += 4;
return value;
On a x86 that can compile into bswap or mov, which is reasonably low-overhead if the method is inlined; the compiler would insert "move" node into the intermediate code, nothing else, which is fairly efficient. If alignment is a problem the full read-shift-or sequence might get generated, outch, but still not too shabby. Compare-branch could allow the optimization, if test the address LSB's and see if can use the fast or slow version of the parsing. But this would mean penalty for the test in every read. Might not be worth the effort.
Oh, right, we are reading HEADERS and stuff, I don't think that is a bottleneck in too many applications. If some codec is doing some really TIGHT innerloop, again, reading into a temporary buffer and decoding from there is well-adviced. Same principle.. no one reads byte-at-time from file when processing a large volume of data. Well, actually, I seen that kind of code very often and the usual reply to "why you do it" is that the file systems do block reads and that the bytes come from memory anyway, true, but they go through a deep call stack which is high-overhead for getting a few bytes!
Still, write the parser code once and use zillion times -> epic win.
Reading directly into struct from a file: DON'T DO IT FOLKS!
It affects each member independently, not the whole struct. Also, it does not affect things like arrays. For instance, it just makes bytes in an ints stored in reverse order.
PS. That said, there could be a machine with weird endianness. What I just said applies to most used machines (x86, ARM, PowerPC, SPARC).
You have to correct the endianess of each member of more than one byte, individually. Strings do not need to be converted (fooword and barword), as they can be seen as sequences of bytes.
However, you must take care of another problem: aligmenent of the members in your struct. Basically, you must check if sizeof(RECORD) is the same on both unix and windows code. Compilers usually provide pragmas to define the aligment you want (for example, #pragma pack).
You also have to consider alignment differences between the two compilers. Each compiler is allowed to insert padding between members in a structure the best suits the architecture. So you really need to know:
How the UNIX prog writes to the file
If it is a binary copy of the object the exact layout of the structure.
If it is a binary copy what the endian-ness of the source architecture.
This is why most programs (That I have seen (that need to be platform neutral)) serialize the data as a text stream that can be easily read by the standard iostreams.
I like to implement a SwapBytes method for each data type that needs swapping, like this:
inline u_int ByteSwap(u_int in)
{
u_int out;
char *indata = (char *)∈
char *outdata = (char *)&out;
outdata[0] = indata[3] ;
outdata[3] = indata[0] ;
outdata[1] = indata[2] ;
outdata[2] = indata[1] ;
return out;
}
inline u_short ByteSwap(u_short in)
{
u_short out;
char *indata = (char *)∈
char *outdata = (char *)&out;
outdata[0] = indata[1] ;
outdata[1] = indata[0] ;
return out;
}
Then I add a function to the structure that needs swapping, like this:
struct RECORD {
UINT32 foo;
UINT32 bar;
CHAR fooword[11];
CHAR barword[11];
UNIT16 baz;
void SwapBytes()
{
foo = ByteSwap(foo);
bar = ByteSwap(bar);
baz = ByteSwap(baz);
}
}
Then you can modify your code that reads (or writes) the structure like this:
fstream f;
f.open("file.bin", ios::in | ios::binary);
RECORD r;
f.read((char*)&detail, sizeof(RECORD));
r.SwapBytes();
cout << "fooword = " << r.fooword << endl;
To support different platforms you just need to have a platform specific implementation of each ByteSwap overload.
Something like this should work:
#include <algorithm>
struct RECORD {
UINT32 foo;
UINT32 bar;
CHAR fooword[11];
CHAR barword[11];
UINT16 baz;
}
void ReverseBytes( void *start, int size )
{
char *beg = start;
char *end = beg + size;
std::reverse( beg, end );
}
int main() {
fstream f;
f.open( "file.bin", ios::in | ios::binary );
// for each entry {
RECORD r;
f.read( (char *)&r, sizeof( RECORD ) );
ReverseBytes( r.foo, sizeof( UINT32 ) );
ReverseBytes( r.bar, sizeof( UINT32 ) );
ReverseBytes( r.baz, sizeof( UINT16 )
// }
return 0;
}