How can I write a series of numbers (in binary) preceded by its length, without knowing the length beforehand?
Can I write the number series to a temporary buffer (e.g. buf.write()), determine the buffer length, and copy the buffer to an ofstream afterwards (e.g. ofs << buf)?
Since you are writing in binary mode to the file, reserve a location for the quantity.
For example, let's have an 32-bit value for the quantity. We could write that at the beginning of the file:
uint32_t number_size = 0U;
// Open output file for read & write.
// Reserve space for the quantity by writing dummy value.
output_file.write((char *)&number_size, sizeof(number_size));
// Perform calculations & writing to file, incrementing "number_size"
// Write the number size to the top of the file:
output_file.seekp(0, SEEK_BEG);
output_file.write((char *)&number_size, sizeof(number_size));
You can place quantity variable anywhere, just remember its file position. When finished, seek to that position and write it in.
Related
Let's just say I have an .ini file that looks like:
[material#3]
Diffuse=sometexture.jpg
Normal=someNormalMap.jpg
Specular=NULL
And I want to add a specular map value to the 'Specular' label, if I had this string buffer in a byte array I would do something like:
void assignSpecularMap(const char* name)
{
// Seek to position after 'Specular='
int oldValueStrLen = strlen("NULL");
int newValueStrLen = strlen(name);
int shiftAmount = newValueStrLen - oldValueStrLen;
// Shift the entire buffer from after 'NULL' by the shiftAmount
// Write the new value after 'Specular='
}
The shifting of the buffer would be done with a memcpy. This is my naive understanding of how to write new values into a text file. If each entry or section isn't padded to a uniform byte size, which it never is for human readable and editable files, then any change in the entries for these files had to be accompanied by a shifting of the entire buffer, correct?
I'm wondering how to do this with a file stream in C++. It's not the case that the entire stream has to be loaded into a char buffer and do it the way I showed?
Right so I am working on reading a large text file (100K+ Characters) and storing it into a large character array.
I have been brainstorming to find the best possible algorithm to do this. Now I know that file reading is an expensive operation in terms of time complexity.
So far I have come up with this using C++:
//Opening the text file from the path given
char * dataset;
int dataLength;
ifstream in(path);
if (!in) {
cout << "Could not open the file." << endl;
exit(-1);
}
//Calculating the number of characters in the file
in.ignore(numeric_limits<streamsize>::max());
dataLength = in.gcount();
//Clearing since EOF bit is set after ignore method above
in.clear();
in.seekg(0, ifstream::beg);
//Initializing the dataset array and copying the data
dataset = new T[dataLength + 1];
in.read(dataset, dataLength);
dataset[dataLength] = '\0';
I am not reading character by character as that renders a very costly operation. Now further trying to improve it, I thought if there would be a way to not read the file twice.
In my algo, I have to read it first (using .ignore) to count the number of characters so I can make a dynamic character array. And then I have to read it to store it into the character array.
Is there any way that when I read the file the first time, I could store the data temporarily into a stream, initialize an array and then ask the stream to copy data into my data structure (array)
I have 640*480 numbers. I need to write them into a file. I will need to read them later. What is the best solution? Numbers are between 0 - 255.
For me the best solution is to write them binary(8 bits). I wrote the numbers into txt file and now it looks like 1011111010111110 ..... So there are no questions where the number starts and ends.
How am I supposed to read them from the file?
Using c++
It's not good idea to write bit values like 1 and 0 to text file. The file size will bigger in 8 times. 1 byte = 8 bits. You have to store bytes, 0-255 - is byte. So your file will have size 640*480 bytes instead of 640*480*8. Every symbol in text file has size of 1 byte minimum. If you want to get bits, use binary operators of programming language that you use. To read bytes much easier. Use binary file for saving your data.
Presumably you have some sort of data structure representing your image, which somewhere inside holds the actual data:
class pixmap
{
public:
// stuff...
private:
std::unique_ptr<std::uint8_t[]> data;
};
So you can add a new constructor which takes a filename and reads bytes from that file:
pixmap(const std::string& filename)
{
constexpr int SIZE = 640 * 480;
// Open an input file stream and set it to throw exceptions:
std::ifstream file;
file.exceptions(std::ios_base::badbit | std::ios_base::failbit);
file.open(filename.c_str());
// Create a unique ptr to hold the data: this will be cleaned up
// automatically if file reading throws
std::unique_ptr<std::uint8_t[]> temp(new std::uint8_t[SIZE]);
// Read SIZE bytes from the file
file.read(reinterpret_cast<char*>(temp.get()), SIZE);
// If we get to here, the read worked, so we move the temp data we've just read
// into where we'd like it
data = std::move(temp); // or std::swap(data, temp) if you prefer
}
I realise I've assumed some implementation details here (you might not be using a std::unique_ptr to store the underlying image data, though you probably should be) but hopefully this is enough to get you started.
You can print the number between 0-255 as the char value in the file.
See the below code. in this example I am printing integer 70 as char.
So this result in print as 'F' on the console.
Similarly you can read it as char and then convert this char to integer.
#include <stdio.h>
int main()
{
int i = 70;
char dig = (char)i;
printf("%c", dig);
return 0;
}
This way you can restrict the file size.
I am taking input from a file in binary mode using C++; I read the data into unsigned ints, process them, and write them to another file. The problem is that sometimes, at the end of the file, there might be a little bit of data left that isn't large enough to fit into an int; in this case, I want to pad the end of the file with 0s and record how much padding was needed, until the data is large enough to fill an unsigned int.
Here is how I am reading from the file:
std::ifstream fin;
fin.open('filename.whatever', std::ios::in | std::ios::binary);
if(fin) {
unsigned int m;
while(fin >> m) {
//processing the data and writing to another file here
}
//TODO: read the remaining data and pad it here prior to processing
} else {
//output to error stream and exit with failure condition
}
The TODO in the code is where I'm having trouble. After the file input finishes and the loop exits, I need to read in the remaining data at the end of the file that was too small to fill an unsigned int. I need to then pad the end of that data with 0's in binary, recording enough about how much padding was done to be able to un-pad the data in the future.
How is this done, and is this already done automatically by C++?
NOTE: I cannot read the data into anything but an unsigned int, as I am processing the data as if it were an unsigned integer for encryption purposes.
EDIT: It was suggested that I simply read what remains into an array of chars. Am I correct in assuming that this will read in ALL remaining data from the file? It is important to note that I want this to work on any file that C++ can open for input and/or output in binary mode. Thanks for pointing out that I failed to include the detail of opening the file in binary mode.
EDIT: The files my code operates on are not created by anything I have written; they could be audio, video, or text. My goal is to make my code format-agnostic, so I can make no assumptions about the amount of data within a file.
EDIT: ok, so based on constructive comments, this is something of the approach I am seeing, documented in comments where the operations would take place:
std::ifstream fin;
fin.open('filename.whatever', std::ios::in | std::ios::binary);
if(fin) {
unsigned int m;
while(fin >> m) {
//processing the data and writing to another file here
}
//1: declare Char array
//2: fill it with what remains in the file
//3: fill the rest of it until it's the same size as an unsigned int
} else {
//output to error stream and exit with failure condition
}
The question, at this point, is this: is this truly format-agnostic? In other words, are bytes used to measure file size as discrete units, or can a file be, say, 11.25 bytes in size? I should know this, I know, but I've got to ask it anyway.
Are bytes used to measure file size as discrete units, or can a file be, say, 11.25 bytes in size?
No data type can be less than a byte, and your file is represented as an array of char meaning each character is one byte. Thus it is impossible to not get a whole number measure in bytes.
Here is step one, two, and three as per your post:
while (fin >> m)
{
// ...
}
std::ostringstream buffer;
buffer << fin.rdbuf();
std::string contents = buffer.str();
// fill with 0s
std::fill(contents.begin(), contents.end(), '0');
I am serializing some data to a file like this:
vector<ByteFeature>::iterator it = nByteFeatures.Content().begin();
for (;it != nByteFeatures.Content().end(); ++it)
{
for ( int i = 0; i < 52; i++)
{
fwrite( &it->Features[i], sizeof(unsigned char), 1, outfile);
}
}
But I would like to know in advance how much bytes that will be in the file.
I would like to write this number in front of the actual data.
Because in some situations I will have to skip loading this data, and I need to know how many bytes I have to skip.
There is more data written to the disk, and it would be crucial to me that I can write the number of bytes directly before the actual data. I do not want to store this number in a separate file or so.
.Content.size() would only tell me how many "items" are in there, but not the actual size of the data.
Thank you.
I've had to do this before myself.
The approach I took was to write a 4-byte placeholder, then the data, then fseek() back to the placeholder to write the length.
Write a 4-byte placeholder to the file.
ftell() to get the current file position.
Write the data to the file.
ftell() to get the new position.
Compute the length: the difference between the two ftell() values.
Then fseek() back to the placeholder and write the length.
You are writing 52 unsigned chars to a file for every ByteFeature. So the total number of bytes you are writing is 52 * nByteFeatures.Contents().size(), assuming that one char equals one byte.