I wrote a simple thread and its task is just to read the data from hard drive , put it in a container and tag it with timestamp and an unique Id. After that I will write the newly structured data into a memory mapped file.
The thing is I don't care about the internal structure of the data, I mean it can be in a Wav format (because in the real situtaion I will be dealing with some audio datas which are avergely 3 mb each) bu I won't do any operaions on that data. After inserting it in my struct I will be just dealing with the UniqueId and Data Tag. Sample structure would be something like:
Struct SampleData
{
long UniqueID;
... MyData; // the data which I am trying to read from hard drive
Time insertionTime;
}
So the question is how am i going to read a Wav data into this struct without knowing (because i don't need to) the internal structure of it? What will ve the ... part for instance. Is there any container type for a large data chunk?
For reading tha data may I use ifstream or any other method?
Try to keep it in a TLV format:
http://en.wikipedia.org/wiki/Type-length-value
EDIT:
A very simple container for TLV.
You'll be able to store the raw data as it is and you'll know which field you're reading and what's its size.
class TlvContainer
{
public:
unsigned long Type; // Maybe we have billions of types of objects?
unsigned long Size; // The size of the object.
unsigned char* Bytes; // This will hold the raw data.
};
When you write your data to the file, you'll have to know how many bytes it is, allocate the "Bytes" array and update the "Size" field.
When you read it from the file, you'll know how you have written it in. (You'll have to read the fields in the same order you have written them.)
For example, if you wrote it as: Type, Size, Bytes:
You'll first read sizeof(unsigned long) from the file to know the the type of the element.
Then you'll need to read another sizeof(unsigned long) to know how big is your real data.
and then you'll be able to read "Size" bytes from the file, knowing that after them there is a new element built the same way.
How about storing MyData as vector<unsigned char>?
You can use file streams to read but remember to use ios::binary mode. See http://www.cplusplus.com/doc/tutorial/files/
Here is sample code. You may want to add error checking. Also, I didn't try to compile this so there may be mistakes.
std::vector<unsigned char> data;
ifstream file("sample.wav", ios::binary);
while(!file.eof()) {
unsigned char byte;
file >> byte;
data.push_back(byte);
}
Related
How to read from binary file (bytes, also known as unsigned charin C++) to vector skipping the first value which is number as unsigned int 32, because the first value is also the size of vector?
First value is also the size of the whole file.
You could try something like this:
uint32_t data_size = 0;
data_file.read((char *) &data_size, sizeof(data_size));
std::vector<uint8_t> data(data_size);
data_file.read((char *) &data[0], data_size);
The above code fragment first reads the size or quantity of the data from the file.
A std::vector is created, using the quantity value that was read in.
Finally, the data is read into the vector.
Edit 1: Memory-mapped files
You may want to consider opening the data file as a memory mapped file. This is where the Operating System treats the file as memory. You don't have to store the data in memory nor read it in. Since memory mapped file APIs vary among operating systems, you'll have to search your Operating System API to find out how to use the feature.
I would like to write a struct to a file as binary. The struct has two members, one is POD-only but the problem is the second member is a string:
struct ToWrite
{
std::string _str;
PODType _pod;
};
If I was simply writing the POD type as binary I would do:
file.write((char*)&_pod, sizeof(_pod));
and to read back:
const PODType& pod = *reinterpret_cast<const PODType*>(&bytes[position]);
However, I'm aware the string is more-complicated because you need to record the size. If I were to add a third class member, being an int containing the size of the string, how do you write and read the struct?
You need to do three things:
Define a storage format for the structure as a stream of bytes.
Write code to convert the structure into an array of bytes in the format you defined in step 1.
Write code to parse the array of bytes you defined in step 1 and fill in the structure.
If you want to find more information, the best search keyword to use is probably "serialization". There are lots of serialization libraries that you can use to save you from having to go through this complexity every time you need to serialize/deserialize a data structure. I personally like protocol buffers and boost serialization, but there are lots of options.
I'm using Arduino Uno to save several variables into the EEPROM. My solution to this is to create a custom struct in which to store all the variables needed then use EEPROM.put to store them into the built-in EEPROM.
My problem arises not when putting data, but when getting data. Whenever I would change something in my code and/or use the Arduino in a short amount of time, the data stored in the EEPROM gets corrupted.
I check the data through Serial.println() function. From time to time the String data get corrupted but not the other data types as far as I can remember. Maybe an issue with String data?
struct EEPROMDATA
{
String customMessage, emergencyMessage;
String emergencyContact[3];
String ownerContact;
String idleMessage;
int travelThreshold;
int idleThreshold;
char password[6];
location locationList[3];
};
EEPROMDATA eepromstruct;
void loadReset()
{
EEPROM.get(100, eepromstruct);
}
void saveReset()
{
EEPROM.put(100, eepromstruct);
}
These are the functions I use to save (put) and load (get) the data from the EEPROM. I've read somewhere to avoid writing data at the 0th address that's why the address is set to 100.
Do remember that this is not a writing to EEPROM issue as my program currently doesn't have any references to EEPROM.put nor to saveReset(). Somehow during the course of the program the EEPROM data gets corrupted. I don't know if this is a hardware or a software problem.
EDIT: I forgot to note that when troubleshooting this, whenever a corruption occurs, I re-initialize then save the data in the EEPROM again. After that it works fine for a short while (turning on/off continuously then testing the data, resetting the device, etc.) until it corrupts again.
You need to use char[] instead of using String which is a class object when you declare at some point in your code. The object (an instance of the class) is supposed to refer the other area of the memory in case of dynamic memory allocation to store its data by its methods. Even though you're assuming that your data would be stored in EEPROM, but actually, it was stored in some area of RAM. That's why you have corrupted data only with String data.
First of all, if you change the definition of class EEPROMDATA, this will "corrupt" your eeprom data. Not in the sense that the data in the eeprom is somehow changed, but in the sense that you're trying to read data written in an old format into a new format.
Next, strings are variable size char arrays. Structs assume a static memory layout. The String class achieves variable length storage through indirection (pointers, essentially), which cannot be trivially translated from RAM to EEPROM.
You need to decide whether you want static layout in EEPROM (in which case you have to stop messing with the struct, and dedicate a specific number of chars to your strings) or dynamic (in which case you need to write an "intelligent" function to parse the data into/out of EEPROM)
I have a struct. I would like to have an array in this struct, and then write this to a binary file, and then read it. However this array should be dynamically allocated. I'm not sure how should I approach this. My current guess is this:
I define and then write the struct to the file like this:
struct map {
int *tiles;
};
int main() {
map sample;
sample.tiles = new int[2];
sample.tiles[0]=1;
sample.tiles[1]=2;
ofstream file("sample.data", ios::binary);
file.write((char *)&sample, sizeof(sample));
file.close();
return 0;
}
Then read it like this in another program:
map test;
ifstream file("sample.data", ios::binary);
file.read((char *)&test, sizeof(test));
When I want to check the results with
cout << test.tiles[0];
I get a weirdly huge number, but clearly not the number I originally wrote to the file.
What is the right way to do this? How can I read an array without knowing its size?
file.write((char *)&sample, sizeof(sample));
This writes to the file the following structure:
struct map {
int *tiles;
};
That is, a structure that contains a single pointer. A pointer is an address in memory. So, what you end up writing to the file is one, meaningless, raw, memory address.
Which, obviously, is of absolutely no use, whatsoever, when it gets read back later. You did not write any of your integers to your file, just what their address in memory was, in a program that long ago terminated.
In order to do this correctly, you will need to:
Record how many int-egers the tiles pointer is pointing to.
fwrite() not the structure itself, but the integers that the tiles pointer is pointing to.
You will also need to write, to the file, how many integers there are in this array. If the file contains only these integers, this is not really needed, since you can simply read the entire contents of the file. But, if you expect to write some additional data in the file, it obviously becomes necessary to also record the size of the written array, in the file itself, so that it's possible to figure it out how many integers there are, when reading them back. The simplest way to do this is to write a single int, the size of the array first, followed by the contents of the array itself. And do the reverse process, when reading everything back.
Your sample object contains a pointer tiles and it's the pointer value that you're writing to the file. What you want to write is what the pointer points at.
In your example, you'd want to do file.write(sample.tiles, 2*sizeof(*sample.tiles));. It's 2* because you did new int[2]. In general, you'd also want to save the size (2 in this case) in the file so you know how many ints to read back in. In this simple case you could infer the 2 from the size of the file.
I'm trying to write a .bmp in C++ but unsigned char can't hold a value high enough for my needs and other data types add padding because of specific byte alignment that makes a mess of things. So is there an alternative data type that can hold a higher value that won't add padding (or a way to make unsigned char hold a higher value)?
If you write your data in an array first, there is no padding.
It sounds as if you are misunderstanding how data is stored.
If you have a bmp image in memory it may be kept in an internal structure (struct/class) however you can serialize that to an array of unsigned chars (bytes) which you write to a file. A file opened in binary mode doesn't add anything more than what you write.
see