Passing QByteArray containing binary data to function by value - c++

So I'm working on a large program that manipulates binary data stored in QByteArrays. Today, I realized that when I pass a QByteArray to a function by value, e.g.
void myFunc(QByteArray data)
{
// Do stuff
}
that if my QByteArray has a 0x0 somewhere in the middle of the binary stream, like 0xA5 0x20 0x04 0x00 0x52. The 0x52 gets chopped off when the QByteArray is passed into the function.
I imagine that I could simply pass the QByteArray by value into the function, but there are some cases where I need a copy of the array (for data persistence or adding to a queue, etc) and unfortunately, I have many, many lines of code where such functions are passing around copied QByteArrays.
So I'm wondering if there is any way to overload the assignment operator (or use some other method) to copy the data between two QByteArrays without treating the 0x00 items as null-terminated string characters. I'm trying to find the simplest approach to this problem without having to completely gut my program and redo all the functions. Is there a better alternative datatype I can use (where i can just do a quick Find/replace, or some function I can wrap them in?
UPDATE: Added stripped down example to demo problem
int main(int argc, char *argv[])
{
QApplication a(argc, argv);
QByteArray data = QByteArrayLiteral("\xA5\x20\x04\x00\x52");
doStuff(data);
return a.exec();
}
void doStuff(QByteArray theData)
{
// theData will contain only 0xA5 0x20 0x04 0x00 if you look from in here
}
UPDATE 2:
OK, apparently, after testing the above example. it actually DOES work. However, the real scenario I'm working with in a thread and I need to test more. Will update once I narrow the problem down more...

I dont think it has anything todo with passing the QByteArray by value but more like how you initialize your arrays. Consider this example:
QByteArray data("\xA5\x20\x04\x00\x52");
qDebug( data.toHex() );
QByteArray data2;
data2.append( static_cast<char>( 0xA5 ) );
data2.append( static_cast<char>( 0x20 ) );
data2.append( static_cast<char>( 0x04 ) );
data2.append( static_cast<char>( 0x00 ) );
data2.append( static_cast<char>( 0x52 ) );
qDebug( data2.toHex() );
The output is:
a52004
a520040052
Initializing it with a string it chops of at the 0-termination. This also holds true for QByteArray::toStdString().
EDIT:
As Frank pointed out:
QByteArray data("\xA5\x20\x04\x00\x52", 5);
qDebug( data.toHex() );
will indeed put out what is expected:
a520040052

Related

Calculate SHA1 hash like Git with Qt C++

I'd like to hash a file in the same way that git hash-object does, so I can compare it to an existing hash, but using Qt and C++.
The answers to this question show how to get the same hash, but none of the examples use C++.
So far this is what we've tried:
QString fileName = entry.toObject().value( "name" ).toString();
QByteArray shaJson = entry.toObject().value( "sha" ).toString().toUtf8();
QByteArray shaFile;
QFile f( QString( "%1/%2" ).arg( QCoreApplication::applicationDirPath() ).arg( fileName ) );
if( f.open(QFile::ReadOnly ) )
{
QCryptographicHash hash(QCryptographicHash::Sha1);
hash.addData( QString( "blob " ).toUtf8() ); // start with the string "blob "
hash.addData( QString( "%1" ).arg( f.size() ).toUtf8() ); // add size in bytes of the content
hash.addData( QString( "\0" ).toUtf8() ); // null byte
hash.addData( f.readAll() ); // actual file content
shaFile = hash.result().toHex();
if( shaFile != shaJson ){
}
}
How to implement this hashing method with Qt?
Edit:
Here's an example hash output:
ccbf4f0a52fd5ac59e18448ebadf2ef37c62f54f
Computed with git hash-object from this file:
https://raw.githubusercontent.com/ilia3101/MLV-App/master/pixel_maps/80000301_1808x1007.fpm
So that's the hash we also like to compute with Qt.
The problem is that on the one hand, QString ignores \0 as termination string, on the other hand, QByteArray always appends extra \0. From Qt's docs:
Using QByteArray is much more convenient than using const char *.
Behind the scenes, it always ensures that the data is followed by a
\0 terminator, and uses implicit sharing (copy-on-write) to reduce
memory usage and avoid needless copying of data.
https://doc.qt.io/qt-5/qbytearray.html
So, every addData in your case is adding extra \0 to the data that is to be hashed. Some workaround might be the following code:
QFile file(path);
if( file.open(QFile::ReadOnly ) )
{
QCryptographicHash hash(QCryptographicHash::Sha1);
QByteArray header = QString("blob %1").arg(file.size()).toUtf8();
hash.addData(header.data(), header.size() + 1);
hash.addData(file.readAll());
shaFile = hash.result().toHex();
qDebug() << shaFile;
}
The data() of QByteArray is returning a pointer to the data stored in the byte array. The pointer can be used to access and modify the bytes that compose the array. The data is '\0'-terminated, i.e. the number of bytes in the returned character string is size() + 1 for the '\0' terminator. Therefore, we do not need add explicitly \0, QByteArray is doing that for us. We need to add +1 to the size since QByteArray returns size of an array as it would be no \0 character.
The code above generated ccbf4f0a52fd5ac59e18448ebadf2ef37c62f54f for your file, so I guess it is a correct hash.

How do I properly store raw byte data in XML and recover it?

Using C++ and Qt, I need to store some raw byte data (an unsigned char array) in a QDomElement (XML node), and then recover it later so that I can compare it to the raw data that is written directly to a different binary file. During testing, I noticed my solution works ~85% of the time, but comparing the recovered data and the raw data read from file seems to fail occasionally. The code snippets below illustrate the Qt methods I am currently using. I have very little knowledge of different character encodings and what I need to look out for in that regard, so I am assuming my mistake has something to do with that.
Storing the raw data in XML:
QDomElement myElement;
unsigned char rawData[ DATA_LEN ];
foo( rawData ); // upon return, rawData now contains the data I want to store in XML
QByteArray dataByteArray( reinterpret_cast< char * >( rawData ) );
QString dataStr( dataByteArray.toBase64() );
QByteArray excluded = " /():|+,.=[]_^{}";
myElement.setAttribute( "Data", QUrl::toPercentEncoding( dataStr, excluded ) );
Recovering the data from XML and comparing to raw data read from binary file (the memcmp() occasionally fails):
unsigned char recoveredData[ DATA_LEN ];
QString dataStr = QUrl::fromPercentEncoding( stringFromXmlNode.toUtf8() );
QByteArray dataByteArray = QByteArray::fromBase64( dataStr.toAscii() );
memcpy( recoveredData, reinterpret_cast< unsigned char * >( dataByteArray.data() ), DATA_LEN );
unsigned char dataFromFile[ DATA_LEN ];
fread( dataFromFile, 1, DATA_LEN, filePtr );
if( 0 != memcmp( dataFromFile, recoveredData, DATA_LEN ) )
{
return false;
}
I am restricted to Qt 4.8, so please refrain from any Qt5-specific solutions if possible, thanks!
You state the bytes are random, so they can contain 0 bytes. Byte value 0 is string terminator in C-style strings. This line in your code initializes QByteArray from such string:
QByteArray dataByteArray( reinterpret_cast< char * >( rawData ) );
Solution is to also pass length of rawData and use this constructor.
You want to use an XML CDATA section.
Look at QDomCDATASection
https://doc.qt.io/archives/qt-4.8/qdomcdatasection.html
Why don't you use QDataStream, which can define the byte order (important if exchanging data across different platforms) and the versioning?
From the Qt 4.8 documentation page:
A data stream cooperates closely with a QIODevice. A QIODevice represents an input/output medium one can read data from and write data to. The QFile class is an example of an I/O device.
Example (write binary data to a stream):
QFile file("file.dat");
file.open(QIODevice::WriteOnly);
QDataStream out(&file); // we will serialize the data into the file
out << QString("the answer is"); // serialize a string
out << (qint32)42; // serialize an integer
Example (read binary data from a stream):
QFile file("file.dat");
file.open(QIODevice::ReadOnly);
QDataStream in(&file); // read the data serialized from the file
QString str;
qint32 a;
in >> str >> a; // extract "the answer is" and 42
Each item written to the stream is written in a predefined binary
format that varies depending on the item's type. Supported Qt types
include QBrush, QColor, QDateTime, QFont, QPixmap, QString, QVariant
and many others. For the complete list of all Qt types supporting data
streaming see Serializing Qt Data Types.
You can read/write your XML Data with QDataStream and import them in the QDomDocument structure with the QDomDocument function setContent() and toByteArray().

QSerialPort reads hex values sepereatly

I'm using Qt's QSerialPort library to communicate with RS232. I connected ReadyRead signal to my readData() slot;
connect(comms,SIGNAL(readyRead()),this,SLOT(readData()));
When i send a string like "Hello World!" I can read all of data with comms.readAll() and comms.bytesAvailable() returns 12.
But when i send "Hello World!\n\r" it reads "Hello World!" and "\n\r" parts sepereatly and comms.bytesAvailable() returns 12 first, then 2.
And it's getting worse when i send hex bytes like (with no spaces)
0x0F 0x00 0x43 0x11 0x00 0x04 0x11 0x00 0x02 0x70
It reads values correctly but 1 or 2 bytes at a time. I tried waitForRead() but that doesn't help.
How can i read all incoming bytes at a time even it's not standart letter?
Try reading from the port while bytes are available:
if (f_port->bytesAvailable()) { // If there are bytes available
QByteArray f_data; // data container
f_data.clear();
if (f_port->open(QIODevice::ReadWrite)) { // Try to open the port
while(f_port->bytesAvailable()) { // Reading loop
f_data.append(f_port->readAll());
}
f_port->flush();
f_port->close();
}
qDebug() << f_data; // Check the result
}
Unfortunately you cannot be sure to have read all data.
You have to collect incoming data in some intermediate buffer and analyse it for commands complying to your protocol definition. That is, is must meet certain requirements like fixed length or particular starting byte (0x02 for instance) or ending byte (\r comes to mind) or a combination of those.
One way to do it is accumulating a buffer with the bytes you obtain.
Then verify if it's a correct command(that is up to you decide what is correct) and trigger the command you wanna do.
Also you should have a timer to remove trash from the buffer.
Let's see with an small pseudocode
static QByteArrray s_vBuffer;
readData()
{
s_vBuffer.append(....);
bool bValidCommand=VerifyCommand(s_vBuffer);
if(bValidCommand)
{
QByteArray vCommand=ExtractCommand(s_vBuffer);//also removing the part of the command
ExecuteCommand(vCommand);
}
else
{
//if timeout clear s_vBuffer
}
}
Other techniques involves checksums , CRC etc at the end of your command.etc

Append quint16/unsigned short to QByteArray quickly

In my project I'm working with QByteArrays appending data to them as the program goes. Most of the time, a simple quint8 gets appended just fine using QByteArray::append(). But when a quint16 gets appended, only 1 byte gets appended instead of 2.
QByteArray ba = QByteArray::fromHex("010203");
quint number(300);//300 in hex is 012c
ba.append(number);//What should be appended instead of just number?
//the current incorrect result is
ba.toHex() == "0102032c"
//the desired result is
ba.toHex() == "010203012c"
I've already tried this, but it just inserts the value as a string (4 bytes):
ba.append(QByteArray::number(number, 16));
What should I append to the QByteArray so both bytes of "number" get appended instead of just one byte? Also, the fastest method possible is preferred since this program needs to have great performance times. So absolutely no converting to QStrings.
Thanks for your time.
On its own, QByteArray only supports appending bytes; to append a big-endian representation of fixed-size integer types you can build your own operator<< (or what you prefer) overloads using the appropriate bit shifts:
QByteArray &operator<<(QByteArray &l, quint8 r)
{
l.append(r);
return l;
}
QByteArray &operator<<(QByteArray &l, quint16 r)
{
return l<<quint8(r>>8)<<quint8(r);
}
QByteArray &operator<<(QByteArray &l, quint32 r)
{
return l<<quint16(r>>16)<<quint16(r);
}
This allows you to write code like:
QByteArray b;
b<<quint16(300); // appends 0x01 0x2c
b<<quint8(4); // appends 0x04
b<<quint16(4); // appends 0x00 0x04
b<<quint32(123456); // appends 0x00 0x01 0xe2 0x40
b<<quint8(1)<<quin16(2)<<quint32(3); // appends 0x01 0x00 0x02 0x00 0x00 0x00 0x03
You should probably avoid writing
QByteArray b;
b<<1;
because in theory the output depends on the size of the current platform integer (although AFAIK on all platforms supported by Qt int is 32 bit).

Reading binary files

I've to read data from binary file.
This binary data format is:
0x00 0x00 0x01 - is delimiter
after this delimiter there is raw data byte array.
So, to sum up, my binary file looks like:
0x00 0x00 0x01 (here is raw data byte)
0x00 0x00 0x01 (here is another block
of raw data bytes) 0x00 0x00 0x01 ....
So i've wrote such code to parse my file (I'm not very familiar with C)
ifstream inp("myfile.bin",ios::binary);
char b1, b2, b3;
while (!inp.eof())
{
inp.read(&b1,sizeof(b1));
inp.read(&b2,sizeof(b2));
inp.read(&b3,sizeof(b3));
//finding first delimiter (data starts from delimiter)
while (!((0==b1)&&(0==b2)&&(1==b3)))
{
b1=b2;
b2=b3;
if (inp.eof())
break;
inp.read(&b3,sizeof(b3));
}
if (inp.eof())
break;
char* raw=new char[65535];
int rawSize=0;
inp.read(&b1,sizeof(b1));
inp.read(&b2,sizeof(b2));
inp.read(&b3,sizeof(b3));
raw[rawSize++]=b1;
raw[rawSize++]=b2;
if (inp.eof())
break;
//reading raw data until delimiter is found
while (!((0==b1)&&(0==b2)&&(1==b3)))
{
raw[rawSize++]=b3;
b1=b2;
b2=b3;
if (inp.eof())
break;
inp.read(&b3,sizeof(b3));
}
rawSize-=2; //because of two bytes of delimiter (0x00 0x00) would be added to raw
//Do something with raw data
if (inp.eof())
break;
inp.putback(1);
inp.putback(0);
inp.putback(0);
delete []raw;
}
But sometimes this code falls into infinite loop.
Could you advice me something?
Thanks
I think the problem there is that putback fails. As far as i recall, putback is guaranteed to work only once; second invocation will fail if the internal read buffer is aligned (that is, very rarely; seems like your situation).
To fix, get rid of putback. First of all, move the loop commented as "finding first delimiter" out of the outer while loop: the comment suggests that this code should only run once. After you do it, pay attention that at the beginning of the outer while loop, the sequence 0x00 0x00 0x01 has just been found, so the code doesn't have to use putback and look for it again.
You're using feof() wrong, it's only valid after a read has been attempted and failed.
How do you know that your magic byte sequence 0 0 1 doesn't appear inside the data? If the data is just a "binary array" that doesn't sound like it provides much of a guarantee ...