Currently I read arrays in C++ with ifstream, read and reinterpret_cast by making a loop on values. Is it possible to load for example an unsigned int array from a binary file in one time without making a loop ?
Thank you very much
Yes, simply pass the address of the first element of the array, and the size of the array in bytes:
// Allocate, for example, 47 ints
std::vector<int> numbers(47);
// Read in as many ints as 'numbers' has room for.
inFile.read(&numbers[0], numbers.size()*sizeof(numbers[0]));
Note: I almost never use raw arrays. If I need a sequence that looks like an array, I use std::vector. If you must use an array, the syntax is very similar.
The ability to read and write binary images is non-portable. You may not be able to re-read the data on another machine, or even on the same machine with a different compiler. But, you have that problem already, with the solution that you are using now.
Related
UPDATE: Considerations: this is on an embedded platform, I am memory limited and not wanting to have to use any extra Libs to perfom things like hashing..
I am running a program that uses a multi dimensional array.
uint8_t array[8][32]
This array is constantly updating, I need to run a comparison to determine if in the last 3 steps, there is a repetition.
It occoured to me that the array is just memory, so rather than iterate through the array, and create a comparison temp array (memory and processor wasted) why not approach this almost like making a hash of the array. I dont need any unique crypto so why not just read the memory location of the whole array as an integer?
Thoughts? How would I go about addressing the array to read in an integer which I could store in another simple array for my much easier comparison
unsigned long arrayTemp[3]
This question already has answers here:
Segmentation fault when trying to create a buffer of 100MB
(5 answers)
Closed 1 year ago.
I am writing a function that requires me to turn a file into an array of 0s and 1s. I figured this was most easily accomplished using a bool array. However, the below code fails and crashes for files larger than ~1MB. My machine has 8GB RAM, so I see no reason for the crash.
string file_name;
cin >> file_name;
string text = read_file(file_name); //I have defined this function as returning a string containing the file's contents and it works fine when tested separately
int length = text.size();
bool bin_arr[8*length]; //to store 0s and 1s
The initialisation of bin_arr fails, and the program simply exits.
I will be handling files larger than 1GB or so. However, I have no idea why this is happening. I am fairly new to C++.
In case it is relevant, I am on Windows 10, using GCC version 6.3.0.
I am writing a function that requires me to turn a file into an array of 0s and 1s. I figured this was most easily accomplished using a bool array.
I don't know why you figured that, the easiest way is to store a file as bytes, exactly as the file contains. vector<bool> does this, for example, it doesn't store each bit in an individual byte like your code, and is thus 8 times more memory efficient.
To get individual bits, either use the aforementioned vector<bool> or use regular bit fiddling. Remember that b & (1 << bit_number) returns a non-zero value if that bit is set in your byte.
I will be handling files larger than 1GB or so.
Then don't store the entire file in memory, stream it in small chunks.
However, I have no idea why this is happening.
Oh, that's easy. You are trying to allocate a needlessly gigantic array needlessly on the stack, instead of on the heap. Your stack has very strict limits, by default 1MB I believe in Windows.
Variable-Length Arrays like bool bin_arr[8*length]; are not in the C++ standard and it has risk of stack overflow. You should use std::vector.
Note that std::vector<bool> has a special implementation. To avoid this, using std::vector<char> or std::deque<bool> should be better.
The vector<bool> class in the C++ STL is optimized for memory to allocate one bit per bool stored, rather than one byte. Every time I output sizeof(x) for vector<bool> x, the result is 40 bytes creating the vector structure. sizeof(x.at(0)) always returns 16 bytes, which must be the allocated memory for many bool values, not just the one at position zero. How many elements do the 16 bytes cover? 128 exactly? What if my vector has more or less elements?
I would like to measure the size of the vector and all of its contents. How would I do that accurately? Is there a C++ library available for viewing allocated memory per variable?
I don't think there's any standard way to do this. The only information a vector<bool> implementation gives you about how it works is the reference member type, but there's no reason to assume that this has any congruence with how the data are actually stored internally; it's just that you get a reference back when you dereference an iterator into the container.
So you've got the size of the container itself, and that's fine, but to get the amount of memory taken up by the data, you're going to have to inspect your implementation's standard library source code and derive a solution from that. Though, honestly, this seems like a strange thing to want in the first place.
Actually, using vector<bool> is kind of a strange thing to want in the first place. All of the above is essentially why its use is frowned upon nowadays: it's almost entirely incompatible with conventions set by other standard containers… or even those set by other vector specialisations.
I have a CSV file which contains 250 lines and each line contains 12 items separated by commas. I am going to store this in a 2D array of which the dimension is [250][12].
My question is : " Is it a bad programming practice to use such a huge array ?
I am going to pass this array to a method which takes a 2D array as the argument. It comes with openCV.
will there be a memory overflow ? "
Well, if you don't have to use it, it would be better. For example, read the file line by line and enter each line into the csv parser. That way each line is dealt with, and you rely on the (hopefully professional and optimized) memory management.
However, if it works it works. If you don't need this in a production environment, I don't see why you should have to change it, other than good practice.
First, you have to be clear about how you'll break a line of text into 12 fields typed as expected by openCV. You may want that to be the central area of the design.
No problem using a static array if the size 250x12 will never change and memory consumption is suitable for the hardware your program is supposed to run on. You face a trade-off between memory usage and complexity of code: if memory is a concern or if you have flexibility in mind then you should process line by line or even token by token, provided openCV implements those modes.
If you know the size of the array is going to be limited to 250*12 then that is not a huge array, assuming you are using a reasonable machine. Even with long double type elements your array is going to take 36 MB of space. If, however, your underlying elements are objects with sub-elements then you may want to re-think your approach e.g., processing the array row-by-row or element-by-element instead of reading it into the memory all at once.
As for passing the array to the function, you will not pass the array by value, you will pass a pointer to the array so it should not be a big overhead.
I need to implement a Boolean data container that will store fairly large amount of variables. I suppose I could just use char* and implement C-style macro accessors but I would prefer to have it wrapped in an std:: structure. std::bitset<size_t> does not seem practical as it has a fixed-during-compilation size.
So that leaves me with std::vector<bool> which is optimized for space; and it has a nice bool-like accessor.
Is there a way to do something like directly feed a pointer from it to fwrite()?
And how would one do file input into such a vector?
And lastly, is it a good data structure when a lot of file I/O is needed?
What about random file access (fseek etc)?
EDIT: I've decided to wrap a std::vector<unsigned int> in a new class that has functionality demanded by my requirements.
Is there a way to do something like directly feed a pointer from it to fwrite()?
No, but you can do it with std::fstream
std::ofstream f("output.file");
std::copy(vb.begin(), vb.end(), std::ostream_iterator<bool>(f, " "));
And how would one do file input into such a vector?
Using std::fstream
std::ifstream f("input.file");
std::copy(std::istream_iterator<bool>(f), {}, std::back_inserter(vb));
And lastly, is it a good data structure when a lot of file I/O is needed?
No, vector<bool> is rarely a good data structure for any purpose. See http://howardhinnant.github.io/onvectorbool.html
What about random file access (fseek etc)?
What about it?
You could use a std::vector<char>, resize it to the size of the file (or other size, say you want to process fixed length blocks), then you can pass the contents of it to a function like fread() or fwrite() in the following way:
std::vector<char> fileContents;
fileContents.resize(100);
fread(&fileContents[0], 1, 100, theFileStream);
This really just allows you to have a resizable char array, in C++ style. Perhaps it is a useful starting point? The point being that you can directly access the memory behind the vector, since it is guaranteed to be laid out sequentially, just like an array.
The same concept would work for a std::vector<bool> - I'd just be careful when freading into this as off the top of my head I can't tell you how big (sizeof wise) a bool is, as it depends on the platform (8bit vs 16bit vs 32bit, if you were working on a microcontroller for example).
It seems std::vector<bool> can be optimised to store each bool in a single bit, so, definitely do not attempt to use the memory behind a vector<bool> directly unless you know it is going to work!