C++ Custom Binary Resource File - c++

Question

I have spent countless hours searching for information about a topic like this. I am writing my own custom game engine for fun using SDL in C++. I'm trying to create a custom binary file which will manage my in game resources. So far I've not been able to get vectors to play nice when it comes to storing each 'type' of object I place in the file. So I dropped the idea of using vectors and went to arrays. I have both examples below where I use both a vector or an array. So, first I create a header for the file. Here is the struct:
struct Header
{
const char* name; // Name of Header file
float version; // Resource version number
int numberOfObjects;
int headerSize; // The size of the header
};
Then after creating the header, I have another struct which defines how an object is stored in memory. Here it is:
struct ObjectData{
int id;
int size;
const char* name;
// std::vector<char> data; // Does not work very well
// unsigned char* data; // Also did not
// Also does not work, because I do not know the size yet until I have the data.
// char data[]
};
The major issue with this struct is that the vector does not play well, an unsigned char pointer kept giving me issues, and an array of char data (for hexadecimal storage) was not working because my compiler does not like variable arrays.
The final struct is my resource file structure.
struct ResourceFile
{
Header header;
int objectCount;
// Again, vectors giving me issues because of how they are constructed internally
// std::vector<ObjectData> objectList;
// Below does not work because, again, no variable data types;
// ObjectData objects[header.numberOfObjects]
};
My goal is to be able to write out a single struct to a binary file. Like so:
Header header;
header.name = "Resources.bin";
header.version = 1.0f;
header.headerSize = sizeof(header);
//vector<char> Object1 = ByteReader::LoadFile("D:\\TEST_FOLDER\\test.obj");
//vector<char> Object2 = ByteReader::LoadFile("D:\\TEST_FOLDER\\test.obj");
ObjectData cube;
cube.id = 0;
cube.name = "Evil Cubie";
cube.data = ByteReader::LoadFile("D:\\TEST_FOLDER\\test.obj");
cube.size = sizeof(cube.id) + sizeof(cube.name) + cube.data.size();
ofstream resourceFile("D:\\TEST_FOLDER\\Resources.bin", ios::out|ios::app|ios::binary);
resourceFile << header.name << header.version << header.headerSize;;
resourceFile << cube.id << cube.name << cube.size;
for each (char ch in cube.data)
{
resourceFile << ch;
}
resourceFile.close();
/*
ObjectData cube2;
cube.id = 1;
cube.name = "Ugle Cubie";
for each (char ch in Object1)
{
cube.object.push_back(ch);
}
*/
//resourceFile.data.push_back(cube);
//resourceFile.data.push_back(cube2);
//resourceFile.header.numberOfObjects = resourceFile.data.size();
//FILE* dat = fopen(filename, "wb");
//fwrite(&resourceFile, sizeof(resourceFile), 1, dat); // <-- write to resource file
//fclose(dat);
As you noticed above, I tried two different ways. The first way I tried it was using good old fwrite. The second way was not even writing it in binary even though I told the computer to do so through the flags accepted by ofstream.
My goal was to get the code to work fluently like this:
ResourceFile resourceFile;
resourceFile.header.name = "Resources.bin";
resourceFile.header.version = 1;
resrouceFile.header.numberOfObjects = 2;
resourceFile.header.headerSize = sizeof(resourceFile.header);
ObjectData cube;
ObjectData cube2;
resourceFile.data.push_back(cube);
resourceFile.data.push_back(cube2);
resourceFile.header.numberOfObjects = resourceFile.data.size();
FILE* dat = fopen(filename, "wb");
fwrite(&resourceFile, sizeof(resourceFile), 1, dat); // <-- write to resource file
fclose(dat);
Still no cigar. Any one have any pointers (no pun intended) or a proper example of a resource manager?

Answer 1

This is one of the things I specialize in, so here you go. There is a whole school of programming around this, but the basic rules I follow are:
1) Use FIXED-LENGTH structures for things with a "constant" layout.
These are things like the flag bits of the file, bytes indicating the # of sub-records, etc. Put as much of the file contents into these structures as you can- they are very efficient especially when combined with a good I/O system.
You do this using the pre-processor macro "#pragma pack(1)" to align a struct to byte boundaries:
#ifdef WINDOWS
#pragma pack(push)
#endif
#pragma pack(1)
struct FixedSizeHeader {
uint32 FLAG_BYTES[1]; // All Members are pointers for a reason
char NAME[20];
};
#ifdef WINDOWS
#pragma pack(pop)
#endif
#ifdef LINUX
#pragma pack()
#endif
2) Create a base class, pure interface with a name like "Serializable". He is your high-level API for staging entire file objects into and out of raw memory.
class Serializable { // Yes, the name comes from Java. The idea, however, predates it
public:
// Choose your buffer type- char[], std::string, custom
virtual bool WriteToBinary(char* buffer) const = 0;
};
NOTE: To support a static "Load" you will need all your "Serializable"s to have an additional static function. There are several (very different) ways to support that, none of which the language alone will enforce since C++ doesn't have "virtual static".
3) Create your aggregate classes for managing each file type. They should have the same name as the file type. Depending on file structure, each may in turn contain more "aggregator" classes before you get down to the fixed structures.
Here's an example:
class GameResourceFile : public Serializable
{
private:
// Operator= and the copy ctor should point to the same data for files,
// since that is what you get with FILE*
protected:
// Actual member variables- allows specialized (derived) file types direct access
FixedSizeHeader* hdr; // You don't have to use pointers here
ContentManager* innards; // Another aggregator- implements "Serializable"
GameResourceFile(FixedSizeHeader* hdr, ContentManager* innards)
: hdr(hdr), innards(innards) {}
virtual ~GameResourceFile() { delete hdr; delete innards; }
public:
virtual bool WriteToBinary(char* outBuffer) const
{
// For fixed portions, use this
memcpy(outBuffer, hdr, sizeof(FixedSizeHeader)); // This is why we 'pack'
outBuffer += sizeof(FixedSizeHeader); // Improve safety...
return innards->WriteToBinary(outBuffer);
}
// C++ doesn't enforce this, but you can via convention
static GameResourceFile* Load(const char* filename)
{
// Load file into a buffer- You'll want your own code here
// Now that's done, we have a buffer
char* srcContents;
FixedSizeHeader* hdr = new FixedSizeHeader();
memcpy(hdr, srcContents, sizeof(FixedSizeHeader));
srcContents += sizeof(FixedSizeHeader);
ContentManager* innards = ContentManager::Load( srcContents); // NOT the file
if(!innards) {
return 0;
}
return new GameResourceFile(hdr, innards);
}
};
Notice how this works- each piece is responsible for serializing itself into the buffer, until we get to "primitive" structures that we can add via memcpy() (you can make ALL the components 'Serializable' classes). If any piece fails to add, the call returns "false" and you can abort.
I STRONGLY recommend using a pattern like "referenced object" to avoid the memory management issues. However, even if you don't you now provide users a nice, one-stop shopping method to load data objects from files:
GameResourceFile* resource = GameResourceFile::Load("myfile.game");
if(!resource) { // Houston, we have a problem
return -1;
}
The best thing yet is to add all low-level manipulation and retrieval APIs for that kind of data to "GameResourceFile". Then any low-level state machine coordination for committing changes to disk & such is all localized to 1 object.