this is my first Post as I came along with using the search function so far. But now I wasted a whole day on the following issue:
I recorded a 12bit (written as 16bit) grayscale video and wrote it directly into a binary stream-file (no headers or such).
Now the task is to read the file and output every frame as a 16bit pgm.
The following snipped illustrates what i tried. The output is a valid pgm, but whith "white noise on it".
...
imageBufferShort = new short[imageWidth*imageHeight* sizeof(short)];
...
streamFileHandle.read(reinterpret_cast<char*>(imageBufferShort),2*imageWidth*imageHeight); //double amount because 8bit chars!
// As .read only takes chars, I thought, that I just read the double amount of char-bytes and when it is interpreted as short (=16bit) everything is ok?!?
...now the pgm output:
std::ofstream f_hnd(fileName,std::ios_base::out |std::ios_base::binary |std::ios_base::trunc);
// write simple header
f_hnd.write("P5\n",3);
f_hnd << imageWidth << " " << imageHeight << "\n4095\n"; //4095 should tell the pgm to use 2 bytes for each pixel
f_hnd.write(reinterpret_cast<char*>(imageBufferShort),2*imageWidth*imageHeight);
f_hnd.close();
Again, the file is produced and viewable but contains rubbish. Is the initial guess ok? read 2 "chars" and handle them as one "short"? I also trieb a whitespace after every line but this changes nothing, so I decided to post this shorter code.
Thanks for any help!
As #Domi and #JoeZ pointed out: Your endianness is probably screwed up. Meaning, the order of your bytes is wrong.
To fix your problem, you will have to iterate over every pixel and swap it's bytes before writing it back to file.
Problem Solved. Thank you all very much. The Endianess was indeed the Problem. The solution is given below:
f_hnd << "P5" << " " << imDimensions.GetWidth() << " " << imDimensions.GetHeight() << " " << "4095\n";
// convert imageBufferShort to Big-Endian format
unsigned short imageBufferShortBigEndian[imDimensions.GetWidth()*imDimensions.GetHeight()];
for (int k=0 ; k<imDimensions.GetWidth()*imDimensions.GetHeight() ; k++)
{
imageBufferShortBigEndian[k] = ( (imageBufferShort[k] << 8) | (imageBufferShort[k] >> 8) );
}
f_hnd.write(reinterpret_cast<char*>(imageBufferShortBigEndian),2*imDimensions.GetWidth()*imDimensions.GetHeight());
f_hnd.close();
imageBufferShort has also to an unsigned short-array. If signed types are used, the bitshift-conversion gets slightly more difficult.
Thanks again!
Related
A bit of confusion here: I'm trying to do this:
QBuffer _ourLogMessageBuffer;
QByteArray theLogMessage;
...
qDebug() << "Writing " << theLogMessage.size() << " bytes to buffer\n";
qint64 numberOfBytes - _ourLogMessagesBuffer.write(theLogMessage);
qDebug() << "Wrote " << numberOfBytes << " bytes to buffer\n";
qDebug() << "Buffer has " << _ourLogMessagesBuffer.bytesAvailable()
<< " bytes available to read (after write)\n";
This outputs the following:
Writing 196 bytes to buffer
Wrote 196 bytes to buffer
Buffer has 0 bytes available to read (after write)
That last line really confuses me. I thought the return value from the .write() method was supposed to say how many bytes were written? Why would they not be available?
And, later, I attempt the following:
qDebug() << "Buffer has " << _ourLogMessagesBuffer.bytesAvailable()
<< " bytes available to read (before read)\n";
char logMessageBytes[565];
qint64 numberOfBytes = _ourLogMessagesBuffer.read(logMessageBytes, 565);
qDebug() << "Read " << numberOfBytes << " bytes from buffer\n";
Considering the previous bytesAvailable result, the output of these calls aren't too surprising. They output the following:
Buffer has 0 bytes available to read (before read)
Read 0 bytes from buffer
So I feel like I'm missing a step, and that you have to do something between writing and the data being available to read. Perhaps some sort of seek or something? But I seem to be missing where it says that in the documentation.
Any tips would be appreciated. Thank you!
You need to seek back to the position you want to read from:
_ourLogMessagesBuffer.seek(0);
Then you will be able to see an appropriate amount of bytesAvailable. If you think about as a (physical) pointer to a position on a tape, it makes sense. As you write, the pointer moves to the end where it can write more data. Any tape ahead of the pointer is "blank"; there's nothing to read (for a "blank" tape, a new or empty buffer).
When just writing, the position is automatically updated for you. But if you want to read data you already wrote, you need to tell it to go back.
An exception to this is with, say, a file format. If we are modifying an existing file, we could update a fixed-length timestamp in one part, then immediately read a couple bytes denoting the length of an "author" string, and then read that string in. For that, we would not need a seek as all the data is contiguous, and the write and read functions handle moving the position within the file (buffer) automatically.
If you have non-contiguous reads/writes, you need to seek. Otherwise, it can't read your mind on where you want to read from.
When I read in information via fstream, it has ocurred twice in two different programs, that the input given to my program isn't stable, even if a given file doesn't change.
In my most recent program, which is concerned with audio-reading. I'm doing a simple check on the first four letters in the file. These letters are supposed to be RIFF, which they also are - I checked.
So, in order to check the format of a given binary file, I buffer the first four letters and see if they are equal to 'RIFF'.
char buffer[4];
std::ifstream in(fn,std::ios::binary);
in.read(buffer,4);
if(buffer!="RIFF"){//Always wrong atm
std::cout << "INVALID WAV FILE: " << buffer << std::endl;
}
When I first made the program, I recall this working properly. Now though, I get an error via my own cout:
INVALID WAV FILE: RIFFýfK
Does anyone have any idea as to what has gone wrong? Perhaps a way to make fstream more consistent?
You're reading 4 characters but not adding a zero terminator, furthermore your comparison is wrong since you're not comparing strings equality, you should rather do:
char buffer[5];
std::ifstream in(fn, std::ios::binary);
in.read(buffer, 4);
buffer[4] = '\0'; // Add a zero-terminator at the end
if (strcmp(buffer,"RIFF")) { // If buffer isn't {'R','I','F','F','\0'}..
std::cout << "INVALID WAV FILE: " << buffer << std::endl;
}
I have been having a very hard time writing to a binary file and reading back. I am basically writing records of this format
1234|ABCD|efgh|IJKL|ABC
Before writing this record, I would write the length of this entire record ( using string.size()) and then I write the record to the binary file using ofstream as follows:
int size;
ofstream studentfile;
studentfile.open( filename.c_str(),ios::out|ios::binary );
studentfile.write((char*)&size,sizeof(int));
studentfile.write(data.c_str(),(data.size()*(sizeof(char))));
cout << "Added " << data << " to " << filename << endl;
studentfile.close();
And I read this data at some other place
ifstream ifile11;
int x;
std::string y;
ifile11.open("student.db", ios::in |ios::binary);
ifile11.read((char*)&x,sizeof(int));
ifile11.read((char*)&y,x);
cout << "X " << x << " Y " << y << endl;
first I read the length of the record into the variable x, and then read the record into string y. The problem is, the output shows x as being '0' and 'y' is empty.
I am not able figure this out. Someone who can look into this problem and provide some insight will be thanked very much.
Thank you
You can't read a string that way, as a std::string is really only a pointer and a size member. (Try doing std::string s; sizeof(s), the size will be constant no matter what you set the string to.)
Instead read it into a temporary buffer, and then convert that buffer into a string:
int length;
ifile11.read(reinterpret_cast<char*>(&length), sizeof(length));
char* temp_buffer = new char[length];
ifile11.read(temp_buffer, length);
std::string str(temp_buffer, length);
delete [] temp_buffer;
I know I am answering my own question, but I strictly feel this information is going to help everyone. For most part, Joachim's answer is correct and works. However, there are two main issues behind my problem :
1. The Dev-C++ compiler was having a hard time reading binary files.
2. Not passing strings properly while writing to the binary file, and also reading from the file. For the reading part, Joachim's answer fixed it all.
The Dev-C++ IDE didn't help me. It wrongly read data from the binary file, and it did it without me even making use of a temp_buffer. Visual C++ 2010 Express has correctly identified this error, and threw run-time exceptions and kept me from being misled.
As soon as I took all my code into a new VC++ project, it appropriately provided me with error messages, so that I could fix it all.
So, please do not use Dev-C++ unless you want to run into real troubles like thiis. Also, when trying to read strings, Joachim's answer would be the ideal way.
I know this title might sound confusing. I have a simple question which I haven't been able to solve yet.
Lets imagine I have a file, opening it with an Hex Editor shows it has two characters inside, say 0x1B and 0x00 (obviously unprintable). I'd like to take that as 1B00 in HEX, which is 6912 in DEC, as opposed to directly converting the characters which would be wrong and is what all other questions I saw asked. Well, thats the task I want to do here. Seems simple, but everything I've tried just does it wrong! Even though I am obviously opening the file in binary mode.
I have only managed to read the characters individually, and mess around a bit, but never do what I actually want, which is as simple as taking those 2 hex characters, interpreting them as an Hex Number, and then convert it to Decimal.
Sorry for any unclear idea, Im not a native speaker. Any help will be appreciated, Im sure you'll think this was quite a noobish question :P
EDIT: Sorry, apparently I didn't explain myself properly. I know this might seem abstract, but it is a really concrete little thing which I have struggled to get solved, yet I haven't been able. Maybe I can ask it another way:
How can I scan a character in binary mode, lets say 0x1B, and convert that to actual 1B characters. Just that.
Sounds like you want to read the file as raw data, and then display it on the screen in decimal? Super easy!
int main() {
std::ifstream myfile("filename.data", std::ofstream::binary);
uint16_t number;
char* buffer = (char*)(&number);
while(myfile.read(buffer, sizeof(number))) {
std::cout << number << ' ';
}
}
The reason it's so easy is that there's no hexidecimal involved. The file is saved as a series of bytes, each byte holds one of 256 values. They aren't hex, they're just a series of values. If you read two bytes into the uint16_t, that is the easiest way to interpret two bytes as a single unsigned 2 byte value. And streaming out a uint16_t will, by default, display that value in decimal. There's no hexidecimal involved. The hexidecimal you saw in the hex editor was because a hex editor interprets the bytes as hex values.
If all you want to do is print a number in hexadecimal form, use std::hex
int i = 0x1B;
std::cout << std::hex << i << std::endl;
std::ifstream infile("test.bin", std::ofstream::binary);
while (true)
{
char c1 = ifs.get();
if (!infile.good())
{
break;
}
char c2 = ifs.get();
if (!infile.good())
{
break;
}
int num = (int)c1 |((int)c2 << 8);
// if you need the oppisite order then
// int num = (int)c2 &((int)c1 << 8);
cout << num;
}
I have code that is writing a vector of size greater than 10million to a text file. I used clock() to time the writefile function and its the slowest part of my program. Is there a better way to write to file than my below method?
void writefile(vector<fields>& fieldsvec, ofstream& sigfile, ofstream& noisefile)
/* Writes clean and noise data to respective files
*
* fieldsvec: vector of clean data
* noisevec: vector of noise data
* sigfile: file to store clean data
* noisefile: file to store noise data
*/
{
for(unsigned int i=0; i<fieldsvec.size(); i++)
{
if(fieldsvec[i].nflag==false)
{
sigfile << fieldsvec[i].timestamp << ";" << fieldsvec[i].price << ";" << fieldsvec[i].units;
sigfile << endl;
}
else
{
noisefile << fieldsvec[i].timestamp << ";" << fieldsvec[i].price << ";" << fieldsvec[i].units;
noisefile << endl;
}
}
}
where my struct is:
struct fields
// Stores a parsed line of a file
{
public:
string timestamp;
float price;
float units;
bool nflag; //flag if noise (TRUE=NOISE)
};
I suggest getting rid of the endl. This effectively flushes the buffer every time and thus greatly increases the number of syscalls.
Writing '\n' instead of endl should be a very good improvement.
And by the way, the code can be simplified:
ofstream& files[2] = { sigfile, noisefile };
for(unsigned int i=0; i<fieldsvec.size(); i++)
files[fieldsvec[i].nflag] << fieldsvec[i].timestamp << ';' << fieldsvec[i].price << ";\n";
You could write your file in binary format instead of text format to increase the writing speed, as suggested in the first answer of this SO question:
file.open(filename.c_str(), ios_base::binary);
...
// The following writes a vector into a file in binary format
vector<double> v;
const char* pointer = reinterpret_cast<const char*>(&v[0]);
size_t bytes = v.size() * sizeof(v[0]);
file.write(pointer, bytes);
From the same link, the OP reported:
replacing std::endl with \n increased his code speed by 1%
concatenating all the content to be written in a stream and writing everything in the file at the end increased the code speed by 7%
the change of text format to binary format increased his code speed by 90%.
A significant speed-killer is that you are converting your numbers to text.
As for the raw file output, the buffering on an ofstream is supposed to be pretty efficient by default.
You should pass your array as a const reference. That might not be a big deal, but it does allow certain compiler optimizations.
If you think the stream is messing things up because of repeated writes, you could try creating a string with sprintf of snprintf and write it once. Only do this if your timestamp is a known size. Of course, that would make extra copying because the string must be then put in the output buffer. Experiment.
Otherwise, it's going to start getting dirty. When you need to tweak out the performance of files, you need to start tailoring the buffers to your application. That tends to get down to using no buffering or cache, sector-aligning your own buffer, and writing large chunks.