I'm trying to write a simple TGA image file saver as a learning exercise in C++. I'm basing my code on an example TGA loader that declares a struct for the header and then uses fread() to load the entire header in one go.
My program isn't working right now, and it seems like there are two extra bytes being written to the file. I printed the sizeof my struct and it's two bytes too large (20 instead of the correct 18). After a little reading I think the problem is related to data alignment and padding (I'm not very familiar with how structs get stored).
My question is what's a good solution for this? I guess I could write the struct's components byte-by-byte, instead of using fwrite() to write the entire struct at once, which is what I'm going now. I assumed that if it worked when loading the header, it would also work when writing it. Was my assumption incorrect?
Compilers are allowed to, and frequently do, insert padding bytes into structures so that fields are aligned on appropriate memory addresses.
The simplest solution is to instruct the compiler to "pack" the structure, which means to not insert any padding bytes. However this will make data access to the structure slower, and the means of doing it are compiler-dependent. If you want to be portable and efficient, the only way to go is writing the fields individually.
It did not work even when loading it. You have a couple of options:
Use compiler specific directives (#pragma packed, etc.) to force your structure to be 18 bytes.
Write the code more portable by using offsets and pointers to get/set the buffer fields.
Elements of a struct are generally arranged on 4byte boundaries.
if you have shorts, or chars in your struct, the struct is larger than the sum of the individual element sizes.
Related
I am a beginning C++ student. I have a structure array that holds employee info.
I can put values into the structure, write those values to a binary dat file and
read the values back into the program so that it can be displayed to the console.
Here is my problem. Once I close the program, I can't get the file to read the data from the file back into memory - instead it reads "garbage."
I tried a few things and then read this in my book:
NOTE: Structures containing pointers cannot be correctly stored to
disk using the techniques of this section. This is because if the
structure is read into memory on a subsequent run of the program, it
cannot be guaranteed that all program variables will be at the same
memory locations.
I am pretty sure this is what is going on when I try to open a .dat file with previously stored information and try to read it into a structure array.
I can send my code examples if that would help clarify my question.
Any suggestions would be appreciated.
Speaking generally (since I don't have your code) there's two reasons you generally shouldn't just write the bytes of a struct or class to a file:
As your book mentioned, writing a pointer to disk is pointless, since you're just storing a random address and not the data at that address. You need to write out the data itself.
You especially should not attempt to write a struct/class all at once with something like
fwrite(file, myStruct, sizeof(myStruct)). Compilers sometimes put empty bytes between variables in structs in order to let the processor read them faster - this is called padding. Using a different compiler or compiling for a different computer architecture can pad structures differently, so a file that opens correctly on one computer might not open correctly on another.
There's lots of ways you can write out data to a file, be it in a binary format or in some human-readable format like XML. Regardless of what method you choose to use (each has strengths and weaknesses), every one of them involves writing each piece of data you care to save one by one, then reading them back in one by one. Higher level languages like Java or C# have ways to do this automagically by storing metadata about objects, but this comes at the cost of more memory usage and slower program execution.
I've read that the ideal size of a structure for performance, that's going to be used in a large collection, is 32 bytes. Is this true and why? Does this effect 64bit processors or is it not applicable?
This is in context of modern (2008+) home Intel-based systems.
The ideal size of a struct is enough to hold the information it needs to contain.
The optimal size for a struct is usually the minimum size needed to store whatever data it's supposed to contain without requiring any hacks like bit twiddling/misaligned accesses to make it fit.
The ideal size of a structure is likely to be one cache line (or a sub-multiple thereof). Level one cache lines are typically 32 or 64 bytes. Splitting an element of a data structure across a cache line boundary will require two main memory accesses to read or write it instead of one.
I don't think there is a reasonable answer to your question. Without any information on the context of the application, the "ideal size of a structure" is way, way underspecified.
As an aside, 32 bits is the space of one modern integer -- it isn't large enough for a "struct" except of a couple of characters or bitfields.
The title is a bit confusing, so i will explain a bit more using examples. Just a note: i am parsing a file format. Say we have a structure like this:
struct example
{
typeA a;
typeB b;
typeX x;
typeY y;
typeZ z;
};
So far its ok. Now the problem is, that typeX, typeY and typeZ can vary in size. Depending on flags in the file header (metadata) they can be either two or four bytes large. Another thing is, that there are several more structures like this (about 40). Each of them use the typeX, typeY, typeZ. Some all of them, some just one or two. And finally, most of them are optional, so there might be just four or five structures used, or 20 or 30...
I would like to know if anyone has some idea how to store such a varying set of data. I thought about using templates, but dont know if its the right way.
EDIT: to clarify more: memory is not a big issue, so i probably can afford wasting a bit of space. If typeX is four bytes then it is so for all structures. Hovewer they are not synced, so typeX can be 4 bytes, typeZ can be 2. Most structures might be used multiple times, so there can be 50 example1 structures, 10 example2 structures etc.
The issue for me isn't so much dealing with allocating some space, in concept we can do this:
byte *pA = new byte[the size this time];
but rather what you do with these typeA objects. What does
pA->getValue()
return? Is the intent that it's always, say, a 32 bit numeric? Or do we really have
pA->get16bitValue()
in some cases and
pA->get32bitValue()
in others?
I'd be seeking a way to encapsulate that difference first, and the way of doing that very much depends on how you use the values. The storage problem is probably solvable with a bit of dynamic allocation. It's the actual beahviour of typeA that I see as tricky.
One factor that you haven't mentioned, since you are parsing a file, is whether the software is intended to be CPU-agnostic. Some CPUs are little-endian, which means that an item of data is stored with the least-significant byte first and the most-significant byte last. Other CPUs are big-endian and the byte order is the other way around. When you are parsing a file you have to take this into account if the file might have been written using a CPU with the opposite endianness. The robust way to do this is to define the endianness that the file format requires, and then read the file one byte at a time and construct the data using appropriate shift operators, e.g.
intVal = (buffer[0] << 24) | (buffer[1] << 16) | (buffer[2] << 8) | buffer[3];
So you see that reading the file directly into a struct is probably not a good idea.
You should really treat the file as a stream (which is what it is) and define stream operations to transfer data into your internal memory structures one item at a time.
If you accept this, then the file format becomes decoupled from your internal memory structures. You can then store the data internally however you like. In your case it sounds like an ideal application for polymorphism. You can declare a subclass for each variant of typeX/Y/Z. Or you could use a single class and let it allocate a variable amount of memory in its constructor. It all depends on your circumstances. The key is to decouple the file format from your internal memory structures.
I have a structure called log that has 13 chars in it. after doing a sizeof(log) I see that the size is not 13 but 16. I can use the __attribute__((packed)) to get it to the actual size of 13 but I wonder if this will affect the performance of the program. It is a structure that is used quite frequently.
I would like to be able to read the size of the structure (13 not 16). I could use a macro, but if this structure is ever changed ie fields added or removed, I would like the new size to be updated without changing a macro because I think this is error prone. Have any suggestion?
Yes, it will affect the performance of the program. Adding the padding means the compiler can use integer load instructions to read things from memory. Without the padding, the compiler must load things separately and do bit shifting to get the entire value. (Even if it's x86 and this is done by the hardware, it still has to be done).
Consider this: Why would compilers insert random, unused space if it was not for performance reasons?
Don't use __attribute__((packed)). If your data structure is in-memory, allow it to occupy its natural size as determined by the compiler. If it's for reading/writing to/from disk, write serialization and deserialization functions; do not simply store cpu-native binary structures on disk. "Packed" structures really have no legitimate uses (or very few; see the comments on this answer for possible disagreeing viewpoints).
Yes, it can affect the performance. In this case, if you allocate an array of such structures with the ((packed)) attribute, most of them must end up unaligned (whereas if you use the default packing, they can all be aligned on 16 byte boundaries). Copying such structures around can be faster if they are aligned.
Yes, it can affect performance. How depends on what it is and how you use it.
An unaligned variable can possibly straddle two cache lines. For example, if you have 64-byte cache lines, and you read a 4-byte variable from an array of 13-byte structures, there is a 3 in 64 (4.6%) chance that it will be spread across two lines. The penalty of an extra cache access is pretty small. If everything your program did was pound on that one variable, 4.6% would be the upper bound of the performance hit. If logging represents 20% of the program's workload, and reading/writing to the that structure is 50% of logging, then you're already at a small fraction of a percent.
On the other hand, presuming that the log needs to be saved, shrinking each record by 3 bytes is saving you 19%, which translates to a lot of memory or disk space. Main memory and especially the disk are slow, so you will probably be better off packing the log to reduce its size.
As for reading the size of the structure without worrying about the structure changing, use sizeof. However you like to do numerical constants, be it const int, enum, or #define, just add sizeof.
As with all other performance optimizations, you'll need to profile your code to find the right answer. The right answer will vary by architecture --- and how your use your structure.
If you're creating gigantic arrays the space savings from packing might mean the difference between fitting and not fitting in cache. Or your data might already fit into your cache, in which case it will make no difference. If you're allocating large numbers of the structures in an STL associative container that allocates the storage for your struct with operator new it might not matter at all --- operator new might round your storage up to something that's aligned anyway.
If most of your structures live on the stack the extra storage might already be optimized away anyway.
For a change this simple to test, I suggest building a timing rig and then trying things both ways. For further optimizations I suggest using a profiler to identify your bottlenecks and go from there.
I'm creating my first real binary parser (a tiff reader) and have a question regarding how to allocate memory. I want to create a struct within my TiffSpec class for the IFD entries. These entries will always be 12 bytes, but depending upon the type specified in that particular entry, the values at the end could be of different types (or maybe just an address to another location in the file). What would be the best way to go about casting this sort of data? The smallest size memory I believe I would be dealing with would be 1 byte.
In C++ you should use a union.
This is a mechanism by which you can define several, overlapping data types, possibly with a common header.
See this article for how to use unions for exactly your problem -- a common header with different data underneath.