I'm using a measurement device which sends (binary) float values using a tcp socket with up to 70 kHz.
My goal is to read these values as fast as possible and use them in other parts of my program.
Till now I'm able to extract value by value using a QTcpSocket and QDataStream:
First I create the socket and connect the stream to it
mysock = new QTcpSocket(this);
mysock->connectToHost(ip, port);
QDataStream stream(mysock);
stream.setByteOrder(QDataStream::LittleEndian);
stream.setFloatingPointPrecision(QDataStream::SinglePrecision);
Then I read from the socket and write the stream data to my float value
while(true) //only for test purpose (dont stop reading)
if (mysock->waitForReadyRead())
{
while (mysock->bytesAvailable() >= 6)
{
QByteArray a = mysock->read(6); //each value sent is 6 bytes long
stream.skipRawData(2); //first 2 bytes don't belong to the number
float result;
stream >> result;
//qDebug()<<result;
}
}
When I measure the iteration frequency of the while(true) loop I'm able to achieve about 30 kHz.
Reading multiple values per read I can reach up to 70 Khz. (Not taking other calculations into account which might slow me down)
My questions are:
If I read multiple values at once, how do I extract these values from the QDataStream? I need a 6 bytes spacing with only 4 bytes containing the value.
Answer: In my case there is 2 bytes (trash) followed by a known number of values, for example 4 bytes for a float, 4 bytes for another float, 2 bytes for an uint16.
stream >> trashuint16 >> resultfloat1 >> resultfloat2 >> resultuint16
Expands 1: I can configure my device to send different values of different type (int, float) which need to be written to different variables.
Answer: Same.
Is there a more efficient way to read many values from a QTcpSocket?
Answer: Anwered in the comments.
Update (to answer some questions):
Max rate in Bytes: 70 kHz x 6 Byte (for one value) = 420 kB/s (Doesnt seem that much :))
Update 2
New Question: When i start a transaction (using stream.startTransaction) I would like to know whats inside that stream in binary code.
I dont understand how QDataStream::startTransaction works. How many bytes will be read? what happens with the data I dont extract using >>?
I've tried the following:
if (mysock->waitForReadyRead())
{
stream.startTransaction();
char *c = new char[40];
stream.readRawData(c, 40); //I want to know whats really inside
QByteArray a(c);
qDebug() << a <<stream.status();
if (!stream.commitTransaction())
break;
}
Doing this again and again, I'll sometimes get status = -1 (read too much) and sometimes not. How do I get the "size" of the stream?
Your code has couple mistakes.
You are doing direct reading from socket when in the same time you are using QDataStream. This can break stuff.
Also your code is assuming that your application will receive data in same chunks as it was sent by other end. You do not have such warranty! It may happen that you will receive chunk data which are ending in middle of your frame. It works just by pure luck or you are ignoring some bugs of your application.
This should go like this:
while(true)
if (mysock->waitForReadyRead()) // IMO doing such loop is terrible approach
// but this is Out of the scope of question, so ignoring that
{
while (true)
{
stream.startTransaction();
float result;
qint32 somedata
stream >> somedata >> result; // I do not know binary format your application is using
if (!in.commitTransaction())
break;
AddDataToModel(result, somedata);
}
}
Edit:
From comment:
Please correct me if I'm wrong, but if I want 2 bytes to be discarded I need to do "stream >> someint(2 byte) >> somefloat(4 byte)"? How can I handle many values in stream?
qint16 toBeDiscarded;
float value;
// note stream.setFloatingPointPrecision(QDataStream::SinglePrecision);
// is needed to read float as 32 bit floating point number
stream >> toBeDiscarded >> value;
ProcessValue(value);
Related
name of this topic is probably incorrect, but I haven't idea how to name this issue.
Something about background, I am programming one game, which 3d surface is divided to chunks. I thought out a saving mechanism, in which all chunk objects with their properties are saved in compressed form to unordered map, which is then serialized to file, so parts of world can be loaded and saved effectively regarding current needs.
Of course, when loading, file is loaded, deserialized to unordered map and strings are converted to chunks objects in real time.
That is a plann, but with hard problems with realization.
I tried all possible searches, but without result. during my play with tests, I wrote a small test script like this:
#include <iostream>
#include <fstream>
#include <sstream>
int main()
{
std::ifstream reader("output.dat", std::ios::binary);
std::string data;
reader>>data;
reader.close();
std::cout<<data.size()<<std::endl;
std::stringstream ss;
ss.str(data);
unsigned char id_prefix=0, zone_prefix=1;
while (ss.peek()!=EOF)
{
unsigned char type;
ss>>type;
if (type==id_prefix)
{
unsigned char tempx, tempy, tempz;
unsigned short tempid;
if (!(ss>>tempx)) std::cout<<"reading of x failed."<<std::endl;
if (!(ss>>tempy)) std::cout<<"Reading of y failed"<<std::endl;
if (!(ss>>tempz)) std::cout<<"Reading of z failed."<<std::endl;
if (!(ss>>tempid)) std::cout<<"Reading of id failed, position is "+std::to_string(ss.tellg())+", values are "+std::to_string(type)+" "+std::to_string(tempx)+" "+std::to_string(tempy)+" "+std::to_string(tempz)<<std::endl;
std::cout<<(int)tempx<<" "<<(int)tempy<<" "<<(int)tempz<<" "<<(int)tempid<<std::endl;
}
else if (type==zone_prefix)
{
unsigned char tempx, tempy, tempz;
unsigned int tempzone;
ss>>tempx;
ss>>tempy;
ss>>tempz;
ss>>tempzone;
std::cout<<(int)tempx<<" "<<(int)tempy<<" "<<(int)tempz<<" "<<(int)tempzone<<std::endl;
}
}
}
Output.dat is a file with one experimental decompressed chunk to reproduce parsing process in the game.
You can download it from:
https://www.dropbox.com/s/mljsb0t6gvfedc5/output.dat?dl=1
if you want, it have about 160 kb in size. And here is a first problem.
It is probably only my stupidity, but I thought that when I use std::ios::binary to open ifstream, and then extract its content to string, it will load whole file, but it loaded only first 46 bytes.
That is first problem, next in the game, I used other system to load data which worked, but then stringstream processing as can be seen in lower part of code failed too around this position.
I guess there are problems also with data, as you can see, format is uchar type (indicates whether following bytes refer to id or zone), coordinates (each as uchar), and ushort in case of id, uint in case of zone.
But when I looked into the file with my own created binary editor, it showed id as one byte only, not two as I expected from short value. Saving was done also with stringstream, in form:
unsigned short tempid=3; //example value
ss<
and in result file this was represented as a 51 (in one byte), what is ascii code for 3, so I am little confused, or little more than little.
Can you please help me with this? I am using Mingw g++ 4.9.3 on win7 64-bit.
Thanks much!
Edit from 1.1.2017
Now whole file is read in stringstream, but extraction of values still fails.
When >> extraction reads to the next whitespace, how is it with extraction to unsigned short for example?
I was playing with code bit, trying to change for example unsigned short tempid to unsigned char tempid.
And output does not make sense to me.
In short version, bytes like:
0;1;0;0;51
were read as type 0, x 1, y 0, z 0 and id 3 what is correct, even I don't understand why 51 is here instead of a 3.
Writing to the stream before seemed as:
unsigned short idtowrite=3;
ss<<idtowrite;
But when I changed unsigned short tempid to unsigned char tempid, it read it as type 0, x 1, y 0, z 0 and id 51, what is not correct, but I expect it from writed file.
I wouldn't solve it if it read correctly through full stream, but for some reason until 0;8;0;0;51 all is correct, and from 0;9;0;0;51, which is next to it fails, with x readed as 0, y as 0 and z as 51 and EOF is set.
I am thinking if reading haven't missed a byte, but I don't see a reason to do it.
Can you please recommend me some effective and working way how to store values in stringstream?
Thanks in advance!
std::ios::binary only has the effect of suppressing end-of-line conversion (so that e.g. \r\n in file is not converted to just \n in memory). It is certainly correct to supply this when dealing with binary files.
However, >> is still a formatted input function, which skips leading whitespace, terminates at whitespace and so on.
If you want to actually read the file as binary data, you must use the read function on the stream object.
I have 640*480 numbers. I need to write them into a file. I will need to read them later. What is the best solution? Numbers are between 0 - 255.
For me the best solution is to write them binary(8 bits). I wrote the numbers into txt file and now it looks like 1011111010111110 ..... So there are no questions where the number starts and ends.
How am I supposed to read them from the file?
Using c++
It's not good idea to write bit values like 1 and 0 to text file. The file size will bigger in 8 times. 1 byte = 8 bits. You have to store bytes, 0-255 - is byte. So your file will have size 640*480 bytes instead of 640*480*8. Every symbol in text file has size of 1 byte minimum. If you want to get bits, use binary operators of programming language that you use. To read bytes much easier. Use binary file for saving your data.
Presumably you have some sort of data structure representing your image, which somewhere inside holds the actual data:
class pixmap
{
public:
// stuff...
private:
std::unique_ptr<std::uint8_t[]> data;
};
So you can add a new constructor which takes a filename and reads bytes from that file:
pixmap(const std::string& filename)
{
constexpr int SIZE = 640 * 480;
// Open an input file stream and set it to throw exceptions:
std::ifstream file;
file.exceptions(std::ios_base::badbit | std::ios_base::failbit);
file.open(filename.c_str());
// Create a unique ptr to hold the data: this will be cleaned up
// automatically if file reading throws
std::unique_ptr<std::uint8_t[]> temp(new std::uint8_t[SIZE]);
// Read SIZE bytes from the file
file.read(reinterpret_cast<char*>(temp.get()), SIZE);
// If we get to here, the read worked, so we move the temp data we've just read
// into where we'd like it
data = std::move(temp); // or std::swap(data, temp) if you prefer
}
I realise I've assumed some implementation details here (you might not be using a std::unique_ptr to store the underlying image data, though you probably should be) but hopefully this is enough to get you started.
You can print the number between 0-255 as the char value in the file.
See the below code. in this example I am printing integer 70 as char.
So this result in print as 'F' on the console.
Similarly you can read it as char and then convert this char to integer.
#include <stdio.h>
int main()
{
int i = 70;
char dig = (char)i;
printf("%c", dig);
return 0;
}
This way you can restrict the file size.
Does anyone know how to read in a file with raw encoding? So stumped.... I am trying to read in floats or doubles (I think). I have been stuck on this for a few weeks. Thank you!
File that I am trying to read from:
http://www.sci.utah.edu/~gk/DTI-data/gk2/gk2-rcc-mask.raw
Description of raw encoding:
hello://teem.sourceforge.net/nrrd/format.html#encoding (change hello to http to go to page)
- "raw" - The data appears on disk exactly the same as in memory, in terms of byte values and byte ordering. Produced by write() and fwrite(), suitable for read() or fread().
Info of file:
http://www.sci.utah.edu/~gk/DTI-data/gk2/gk2-rcc-mask.nhdr - I think the only things that matter here are the big endian (still trying to understand what that means from google) and raw encoding.
My current approach, uncertain if it's correct:
//Function ripped off from example of c++ ifstream::read reference page
void scantensor(string filename){
ifstream tdata(filename, ifstream::binary); // not sure if I should put ifstream::binary here
// other things I tried
// ifstream tdata(filename) ifstream tdata(filename, ios::in)
if(tdata){
tdata.seekg(0, tdata.end);
int length = tdata.tellg();
tdata.seekg(0, tdata.beg);
char* buffer = new char[length];
tdata.read(buffer, length);
tdata.close();
double* d;
d = (double*) buffer;
} else cerr << "failed" << endl;
}
/* P.S. I attempted to print the first 100 elements of the array.
Then I print 100 other elements at some arbitrary array indices (i.e. 9,900 - 10,000). I actually kept increasing the number of 0's until I ran out of bound at 100,000,000 (I don't think that's how it works lol but I was just playing around to see what happens)
Here's the part that makes me suspicious: so the ifstream different has different constructors like the ones I tried above.
the first 100 values are always the same.
if I use ifstream::binary, then I get some values for the 100 arbitrary printing
if I use the other two options, then I get -6.27744e+066 for all 100 of them
So for now I am going to assume that ifstream::binary is the correct one. The thing is, I am not sure if the file I provided is how binary files actually look like. I am also unsure if these are the actual numbers that I am supposed to read in or just casting gone wrong. I do realize that my casting from char* to double* can be unsafe, and I got that from one of the threads.
*/
I really appreciate it!
Edit 1: Right now the data being read in using the above method is apparently "incorrect" since in paraview the values are:
Dxx,Dxy,Dxz,Dyy,Dyz,Dzz
[0, 1], [-15.4006, 13.2248], [-5.32436, 5.39517], [-5.32915, 5.96026], [-17.87, 19.0954], [-6.02961, 5.24771], [-13.9861, 14.0524]
It's a 3 x 3 symmetric matrix, so 7 distinct values, 7 ranges of values.
The floats that I am currently parsing from the file right now are very large (i.e. -4.68855e-229, -1.32351e+120).
Perhaps somebody knows how to extract the floats from Paraview?
Since you want to work with doubles, I recommend to read the data from file as buffer of doubles:
const long machineMemory = 0x40000000; // 1 GB
FILE* file = fopen("c:\\data.bin", "rb");
if (file)
{
int size = machineMemory / sizeof(double);
if (size > 0)
{
double* data = new double[size];
int read(0);
while (read = fread(data, sizeof(double), size, file))
{
// Process data here (read = number of doubles)
}
delete [] data;
}
fclose(file);
}
I am taking input from a file in binary mode using C++; I read the data into unsigned ints, process them, and write them to another file. The problem is that sometimes, at the end of the file, there might be a little bit of data left that isn't large enough to fit into an int; in this case, I want to pad the end of the file with 0s and record how much padding was needed, until the data is large enough to fill an unsigned int.
Here is how I am reading from the file:
std::ifstream fin;
fin.open('filename.whatever', std::ios::in | std::ios::binary);
if(fin) {
unsigned int m;
while(fin >> m) {
//processing the data and writing to another file here
}
//TODO: read the remaining data and pad it here prior to processing
} else {
//output to error stream and exit with failure condition
}
The TODO in the code is where I'm having trouble. After the file input finishes and the loop exits, I need to read in the remaining data at the end of the file that was too small to fill an unsigned int. I need to then pad the end of that data with 0's in binary, recording enough about how much padding was done to be able to un-pad the data in the future.
How is this done, and is this already done automatically by C++?
NOTE: I cannot read the data into anything but an unsigned int, as I am processing the data as if it were an unsigned integer for encryption purposes.
EDIT: It was suggested that I simply read what remains into an array of chars. Am I correct in assuming that this will read in ALL remaining data from the file? It is important to note that I want this to work on any file that C++ can open for input and/or output in binary mode. Thanks for pointing out that I failed to include the detail of opening the file in binary mode.
EDIT: The files my code operates on are not created by anything I have written; they could be audio, video, or text. My goal is to make my code format-agnostic, so I can make no assumptions about the amount of data within a file.
EDIT: ok, so based on constructive comments, this is something of the approach I am seeing, documented in comments where the operations would take place:
std::ifstream fin;
fin.open('filename.whatever', std::ios::in | std::ios::binary);
if(fin) {
unsigned int m;
while(fin >> m) {
//processing the data and writing to another file here
}
//1: declare Char array
//2: fill it with what remains in the file
//3: fill the rest of it until it's the same size as an unsigned int
} else {
//output to error stream and exit with failure condition
}
The question, at this point, is this: is this truly format-agnostic? In other words, are bytes used to measure file size as discrete units, or can a file be, say, 11.25 bytes in size? I should know this, I know, but I've got to ask it anyway.
Are bytes used to measure file size as discrete units, or can a file be, say, 11.25 bytes in size?
No data type can be less than a byte, and your file is represented as an array of char meaning each character is one byte. Thus it is impossible to not get a whole number measure in bytes.
Here is step one, two, and three as per your post:
while (fin >> m)
{
// ...
}
std::ostringstream buffer;
buffer << fin.rdbuf();
std::string contents = buffer.str();
// fill with 0s
std::fill(contents.begin(), contents.end(), '0');
suppose I send a big buffer to ostream::write, but only the beginning part of it is actually successfully written, and the rest is not written
int main()
{
std::vector<char> buf(64 * 1000 * 1000, 'a'); // 64 mbytes of data
std::ofstream file("out.txt");
file.write(&buf[0], buf.size()); // try to write 64 mbytes
if(file.bad()) {
// but suppose only 10 megabyte were available on disk
// how many were actually written to file???
}
return 0;
}
what ostream function can tell me how many bytes were actually written?
You can use .tellp() to know the output position in the stream to compute the number of bytes written as:
size_t before = file.tellp(); //current pos
if(file.write(&buf[0], buf.size())) //enter the if-block if write fails.
{
//compute the difference
size_t numberOfBytesWritten = file.tellp() - before;
}
Note that there is no guarantee that numberOfBytesWritten is really the number of bytes written to the file, but it should work for most cases, since we don't have any reliable way to get the actual number of bytes written to the file.
I don't see any equivalent to gcount(). Writing directly to the streambuf (with sputn()) would give you an indication, but there is a fundamental problem in your request: write are buffered and failure detection can be delayed to the effective writing (flush or close) and there is no way to get access to what the OS really wrote.