How to represent irregular, but ordered, data in objects? - c++

Question

I have a set of files with binary data. Each file is composed of blocks and each block has a header and then a set of events. Each event has a header and then a sequence of fields. My problem is with the sequence of fields.
These fields contain different lengths of ordered/structured data but the fields do not come in any particular order. For example, in one event I might have in one event 3 fields looking as follows:
Event Header (12 bytes, always, made of things like number of fields, size, etc)
Field Header (2 bytes, always, field type in the top 4 bits, size in the bottom 12)
Field Data (4299-4298(VDC) data, Signals from various wires in a vertical drift chamber)
Field Header ( '' )
Field Data (ADC-LAS data, Signals from various photo multiplier tubes)
Field Header ( '' )
Field Data (FERA data, Signals from a fast encoding readout adc system)
In another event I might have the same fields plus a few more, or a field removed and another added in, etc. It all depends on which pieces of the DAQ hardware had data to be recorded when the readout system triggered.
I have thought about a few possible solutions and honestly, none of them seem palatable to me.
Method 1:
Make an abstract base class Field and then for each field type (there are only 13) inherit from that.
Pros: Reading the data in from the file is easy, simply get the region id, allocate the appropriate type of field, read the data, and store a Field*. Also, this method appeals to my sense of a place for everything and everything in its place.
Cons: When I process the fields in an event to convert the data to the information that the analysis actually uses I am continuously needing to dynamic_cast<>() everything to the derived class. This is a bit tedious and ugly and I remember reading somewhere (a while ago) that if you are having to use dynamic_cast<>() then you are using polymorphism 'wrong'. Also this makes having object pools for the fields tricky as I would need a pool for every subclass of Field. Finally, if more field types are added later then in addition to modifying the processing code, additional subclasses of field need to be created.
Method 2:
Simply have a big array of bytes to store all the field headers and data. Then leave it up to the processing code to extract the structure as well as process the data.
Pros: This means that if data formats change in the future then the only changes that need to occur are in the event processing code. It's a pretty simple solution. It's more flexible.
Cons: Jobs in the processing/reading code are less compartmentalized. Feels less elegant.
I recognize that there is probably not a solution that is going to be 'elegant' in every way, and from the standpoint of KISS I am leaning towards method 2. Should I choose Method 1, Method 2, or is there some Method 3 that I have not thought of?

Answer 1

You are trying to choose between struct or tuple or MSRA safeprotocole handler
` // Example program
#include
#include
#include
#include
// start ABI Protocole signature
const int EVENT_HEADER_SZ = 12;
const int FIELD_HEADER_SZ = 2;
const int FIELD_DATA_SIZE = 2^12;
// end ABI Protocole
#ifdef WINDOWS
#define __NOVT __declspec(novtable
#else
#define __NOVT
#endif
struct Protocole_Header __NOVT {
union {
char pbody[EVENT_HEADER_SZ+1];
unsigned ptype : 32;
unsigned psize : 32;
unsigned pmisc : 32;
};
};
struct Field_Header __NOVT {
union {
char fbody[FIELD_HEADER_SZ+1];
unsigned ftype : 4; // type of data 0...15
unsigned fsize : 12; // size of field data to follow up 0..4096 max size
};
};
struct Field_Data {
std::string _content;
};
typedef std::tuple<uint_fast32_t, int_fast32_t,uint_fast32_t> Protocole_Header2;
enum PHR{
TPL_TYPE,
TPL_SIZE,
TPL_ETC
};
std::istream& operator >>(std::istream &is, std::tuple<uint_fast32_t, int_fast32_t,uint_fast32_t>& tpl)
{
is >> std::get<TPL_TYPE>(tpl) >> std::get<TPL_SIZE>(tpl) >> std::get<TPL_ETC>(tpl);
return is;
}
union Field_Header2 {
char fbody[FIELD_HEADER_SZ];
unsigned ftype : 4; // type of data 0...15
unsigned fsize : 12; // size of field data to follow up 0..4096 max size
};
int main()
{
Protocole_Header ph;
Field_Header fh;
Field_Data fd;
long i;
char protocole_buffer[FIELD_DATA_SIZE+1];
std::cin.get(ph.pbody,EVENT_HEADER_SZ);
std::cin.get(fh.fbody,FIELD_HEADER_SZ);
for(i=0;i<ph.psize;++i)
{
std::cin.get(protocole_buffer,fh.fsize);
fd._content = protocole_buffer; // push somewhere else
std::cin.get(fh.fbody,FIELD_HEADER_SZ);
}
// ...
// ...
Protocole_Header2 ph2;
Field_Header2 fh2;
std::cin >> ph2;
std::cin.get(fh2.fbody,FIELD_HEADER_SZ);
for(i=0;i<std::get<TPL_SIZE>(ph2);++i)
{
std::cin.get(protocole_buffer,fh.fsize);
fd._content = protocole_buffer; // push somewhere else
std::cin.get(fh2.fbody,FIELD_HEADER_SZ);
}
}`
Here , you have both of your answer ...
Note , using metastructure over structure is as much a burden than find back the code and recompile it in case of rupture of protocole.
Usually you do not define ABI for protocole structure, and that why C++ Spirit was made.
A parser must be used to handle protocole ( always, because protocole is a grammar on its own, define a EBNF and your code will run for decades without people to recompile it ... )
There is only exception for not using a parser , its when you need to pass MSRA or Heatlh Care or any Regulated sector. Rest of time, don't bind external data to ABI structure with C or C++ , it's a 100% cause of bugs .