So I have this data packet that I want to send it to my device using TCP/IP protocol. My array is:
unsigned char array1[] = {'0x00', '0x84', '0x00', '0x00', '0x00', '0x06', '0x54', '0x01', '0x00', '0x01', '0x00', '0x03'};
I want this to convert into a string. How do I do it?
Right now I am just manually writing down the decimal equivalent:
unsigned char array1[] = {0,132,0,0,0,6,84,5,0,2,255,0};
and converting it into string:
std::string data ( array1, array1 + sizeof array1 / sizeof array1[0] );
However, I wonder can I use my hex packet just like a string directly?
string x= "00 84 00 00 00 06 54 05 00 02 FF 00";
Also is there a way I can design my message header which is the first 7 bytes that dont change? What changes is the rest of the part?
The following code should do what you need.
std::string s { "\x00\x01\x02\x03\x04", 5 };
Use the std::string constructor that also takes the length aka number of bytes.
Related
I have written a small application which works at some point with binary data. In unit tests, I compare this data with the expected one. When an error occurs, I want the test to display the hexadecimal output such as:
Failure
Expected: string_to_hex(expected, 11)
Which is: "01 43 02 01 00 65 6E 74 FA 3E 17"
To be equal to: string_to_hex(writeBuffer, 11)
Which is: "01 43 02 01 00 00 00 00 98 37 DB"
In order to display that (and to compare binary data in the first place), I used the code from Stack Overflow, slightly modifying it for my needs:
std::string string_to_hex(const std::string& input, size_t len)
{
static const char* const lut = "0123456789ABCDEF";
std::string output;
output.reserve(2 * len);
for (size_t i = 0; i < len; ++i)
{
const unsigned char c = input[i];
output.push_back(lut[c >> 4]);
output.push_back(lut[c & 15]);
}
return output;
}
When checking for memory leaks with valgrind, I fould a lot of errors such as this one:
Use of uninitialised value of size 8
at 0x11E75A: string_to_hex(std::__cxx11::basic_string, std::allocator > const&, unsigned long)
I'm not sure to understand it. First, everything seems initialized, including, I'm mistaken, output. Moreover, there is no mention of size 8 in the code; the value of len varies from test to test, while valgrind reports the same size 8 every time.
How should I fix this error?
So this is one of the cases where passing a pointer to char that points to buffer filled with arbitrary binary data into evil implicit constructor of std::string class was causing string to be truncated to first \0. Straightforward approach would be to pass a raw pointer but a better solution is to start using array_view span or similar utility classes that will provide index validation at least in debug build for both input and lut.
I have a 320Mb binary file (data.dat), containing 32e7 lines of hex numbers:
1312cf60 d9 ff e0 ff 05 00 f0 ff 22 00 2f 00 fe ff 33 00 |........"./...3.|
1312cf70 00 00 00 00 f4 ff 1d 00 3d 00 6d 00 53 00 db ff |........=.m.S...|
1312cf80 b7 ff b0 ff 1e 00 0c 00 67 00 d1 ff be ff f8 ff |........g.......|
1312cf90 0b 00 6b 00 38 00 f3 ff cf ff cb ff e4 ff 4b 00 |..k.8.........K.|
....
Original numbers were:
(16,-144)
(-80,-64)
(-80,16)
(16,48)
(96,95)
(111,-32)
(64,-96)
(64,-16)
(31,-48)
(-96,-48)
(-32,79)
(16,48)
(-80,80)
(-48,128)
...
I have a matlab code which can read them as real numbers and convert them to complex numbers:
nsamps = (256*1024);
for i = 1:305
nstart = 1 + (i - 1) * nsamps ;
fid = fopen('data.dat');
fseek(fid,4 * nstart ,'bof');
y = fread(fid,[2,nsamps],'short');
fclose(fid);
x = complex(y(1,:),y(2,:));
I am using C++ and trying to get data as a vector<complex<float>>:
std::ifstream in('data.dat', std::ios_base::in | std::ios_base::binary);
fseek(infile1, 4*nstart, SEEK_SET);
vector<complex<float> > sx;
in.read(reinterpret_cast<char*>(&sx), sizeof(int));
and very confuse to get complex data using C++. Can anyone give me a help?
Theory
I'll try to explain some points using the issues in your code as examples.
Let's start from the end of the code. You try to read a number, which is stored as a four-byte single-precision floating point number, but you use sizeof(int) as a size argument. While on modern x86 platforms with modern compilers sizeof(int) tends to be equal to sizeof(float), it's not guaranteed. sizeof(int) is compiler dependent, so please use sizeof(float) instead.
In the matlab code you read 2*nsamps numbers, while in C++ code only four bytes (one number) is being read. Something like sizeof(float) * 2 * nsamps would be closer to matlab code.
Next, std::complex is a complicated class, which (in general) may have any implementation-defined internal representation. But luckily, here we read that
For any object z of type complex<T>, reinterpret_cast<T(&)[2]>(z)[0]
is the real part of z and reinterpret_cast<T(&)[2]>(z)[1] is the
imaginary part of z.
For any pointer to an element of an array of complex<T> named p and
any valid array index i, reinterpret_cast<T*>(p)[2*i] is the real part
of the complex number p[i], and reinterpret_cast<T*>(p)[2*i + 1] is
the imaginary part of the complex number p[i].
so we can just cast an std::complex to char type and read binary data there. But std::vector is a class template with it's implementation-defined internal representation as well! It means, that we can't just reinterpret_cast<char*>(&sx) and write binary data to the pointer, as it points to the beginning of the vector object, which is unlikely to be the beginning of the vector data. Modern C++ way to get the beginning of the data is to call sx.data(). Pre-C++11 way is to take an address of the first element: &sx[0]. Overwriting the object from the beginning will result in segfault almost always.
OK, now we have the beginning of the data buffer which is able to receive binary representation of complex numbers. But when you declared vector<complex<float> > sx;, it got zero size, and as you are not pushing or emplacing it's elements, the vector will not "know" that it should resize. Segfault again. So just call resize:
sx.resize(number_of_complex_numbers_to_store);
or use an appropriate constructor:
vector<complex<float> > sx(number_of_complex_numbers_to_store);
Before writing data to the vector. Note that these methods operate with "high-level" concept of number of stored elements, not number of bytes to store.
Putting it all together, the last two lines of your code should look like:
vector<complex<float> > sx(nsamps);
in.read(reinterpret_cast<char*>(sx.data()), 2 * nsamps * sizeof(float));
Minimal example
If you continue having troubles, try a simpler sandbox code first.
For example, let's write six floats to a binary file:
std::ofstream ofs("file.dat", std::ios::binary | std::ios::out);
float foo[] = {1,2,3,4,5,6};
ofs.write(reinterpret_cast<char*>(foo), 6*sizeof(float));
ofs.close();
then read them to a vector of complex:
std::ifstream ifs("file.dat", std::ios::binary | std::ios::in);
std::vector<std::complex<float>> v(3);
ifs.read(reinterpret_cast<char*>(v.data()), 6*sizeof(float));
ifs.close();
and, finally, print them:
std::cout << v[0] << " " << v[1] << " " << v[2] << std::endl;
The program prints:
(1,2) (3,4) (5,6)
so this approach works fine.
Binary files
Here is the remark about binary files which I initially posted as a comment.
Binary files haven't got the concept of "lines". The number of "lines" in binary file completely depends on the size of the window you are viewing it in. Think of binary files as of a magnetic tape, where each discrete position of the head is able to read only one byte. Interpretation of those bytes is up to you.
If everything should work fine, but you get weird numbers, check the displacement in fseek call. A mistake by a number of bytes yields random-looking values instead of the floats you wish to get.
Surely, you might just read a vector (or an array) of floats, observing the above considerations, and then convert them to complex numbers in a loop. Also, it's a good way to debug your fseek call to make sure that you start reading from the right place.
My task is read file contents and place them into vector array of std::strings, then output from vector to another file.
I managed to solve it for file containing only normal text, but failed with, lets say .bmp
Here is .bmp file header:
BM)( 6 ( — к Р((
hex shows these are NULL bytes, not whitespaces
42 3d 06 29 28 00 00 00 00 00 36 00 00 00 ...
As a result , executing the following code
char *buffer = new char [50];
ifstream ifs( "file.bmp" , std::ifstream::binary );
ifs >> std::noskipws;
ifs.read( buffer, 50 );
std::string abc(buffer); //output it to other file next
i end up with buffer and abc strings equal to "BM)(6(—кР(( ..." with all nullbytes skipped.
ifs.read() was great until now. So whats the most comfortable way to read/write N exact bytes from a file ?
(I did a search but had difficulties with results)
That was lame, but somebody might google this someday so i ll write an answer.
If while debugging you would hover mouse over buffer after calling ifs.read() you would see "BM)(6(—кР(( ..." , but the null bytes are actually there. But (in MSVS) you couldnt see actual bytes buffer has unless you redefine buffer as
char buffer[50];
Next, when making std::string out of buffer, doing
string abc(buffer); // would not copy null bytes to std string,
So do this instead:
string abc(buffer, 50); //copies all bytes.
Now i have the real file contents in the string and if i output abc to other file i will have same data.
P.S. when checking results use hex editor to see real bytes, dont trust Notepad.
I encountered an odd problem when exporting float values to a file. I would expect every float to be of the same length (obviously), but my programme sometimes exports it a 32 bit number and sometimes as a 40 bit number.
A minimal working example of a programme that still shows this behaviour is:
#include <stdio.h>
const char* fileName = "C:/Users/Path/To/TestFile.txt";
float array [5];
int main(int argc, char* argv [])
{
float temp1 = 1.63006e-33f;
float temp2 = 1.55949e-32f;
array[0] = temp1;
array[1] = temp2;
array[2] = temp1;
array[3] = temp2;
array[4] = temp2;
FILE* outputFile;
if (!fopen_s(&outputFile, fileName, "w"))
{
fwrite(array, 5 * sizeof(float), 1, outputFile);
fclose(outputFile);
}
return true;
}
I would expect the output file to contain exactly 20 (5 times 4) bytes, each four of which represent a float. However, I get this:
8b 6b 07 09 // this is indeed 1.63006e-33f
5b f2 a1 0d 0a // I don't know what this is but it's a byte too long
8b 6b 07 09
5b f2 a1 0d 0a
5b f2 a1 0d 0a
So the float temp2 takes 5 bytes instead of four, and the total length of he file is 23. How is this possible?! The number aren't so small that they are subnormal numbers, and I can't think of any other reason why there would be a difference in size.
I am using the MSVC 2010 compiler on a 64-bit Windows 7 system.
Note: I already asked a very similar question here, but when I realised the problem was more general, I decided to repost it in a more concise way.
QDataStream uses sometimes 32 bit and sometimes 40 bit floats
The problem is that on Windows, you have to differentiate between text and binary files. You have the file opened as text, which means 0d (carriage-return) is inserted before every 0a (line-feed) written. Open the file like this:
if (!fopen_s(&outputFile, fileName, "wb"))
The rest as before, and it should work.
You're not writing text; you're writing binary data... However, your file is open for writing text ("w") instead of writing binary ("wb"). Hence, fwrite() is translating '\n' to "\r\n".
Change this:
if (!fopen_s(&outputFile, fileName, "w"))
To this:
if (!fopen_s(&outputFile, fileName, "wb"))
In "wb", the b stands for binary mode.
This HMACSHA1 code below works for converting "Password" and "Message" to AFF791FA574D564C83F6456CC198CBD316949DC9 as evidence by http://buchananweb.co.uk/security01.aspx.
My question is, Is it possible to have:
BYTE HMAC[] = {0x50,0x61,0x73,0x73,0x77,0x6F,0x72,0x64};
BYTE data2[] = {0x4D,0x65,0x73,0x73,0x61,0x67,0x65};
And still get the same value: AFF791FA574D564C83F6456CC198CBD316949DC9.
For example, if I was on a server and received the packet:
[HEADER] 08 50 61 73 73 77 6F 72 64 00
[HEADER] 07 4D 65 73 73 61 67 65 00
And I rip 50 61 73 73 77 6F 72 64 & 4D 65 73 73 61 67 65 from the packet and used this for my HMACSHA1. How would I go about doing that to get the correct HMACSHA1 value?
BYTE HMAC[] = "Password";
BYTE data2[] = "Message";
//BYTE HMAC[] = {0x50,0x61,0x73,0x73,0x77,0x6F,0x72,0x64};
//BYTE data2[] = {0x4D,0x65,0x73,0x73,0x61,0x67,0x65};
HMAC_CTX ctx;
result = (unsigned char*) malloc(sizeof(char) * result_len);
ENGINE_load_builtin_engines();
ENGINE_register_all_complete();
HMAC_CTX_init(&ctx);
HMAC_Init_ex(&ctx, HMAC, strlen((const char*)HMAC), EVP_sha1(), NULL);
HMAC_Update(&ctx, data2, strlen((const char*)(data2)));
HMAC_Final(&ctx, result, &result_len);
HMAC_CTX_cleanup(&ctx);
std::cout << "\n\n";
for(int i=0;i<result_len;i++)
std::cout << setfill('0') << setw(2) << hex << (int)result[i];
int asd;
std::cin >> asd;
// AFF791FA574D564C83F6456CC198CBD316949DC9
EDIT:
It works by doing this:
BYTE HMAC[] = {0x50,0x61,0x73,0x73,0x77,0x6F,0x72,0x64, 0x00};
BYTE data2[] = {0x4D,0x65,0x73,0x73,0x61,0x67,0x65, 0x00};
By adding 0x00, at the end. But, my question is more towards ripping it from data, and using it... would it still be fine?
The issue is the relation ship between arrays, strings, and the null char.
When you declare "Password", the compiler logically treats the string literal as a nine byte array, {0x50,0x61,0x73,0x73,0x77,0x6F,0x72,0x64, 0x00}. When you call strlen, it will count the number of bytes until it encounters the first 0x00. strlen("Password") will return 8 even though there are technically nine characters in the array of characters.
So when you declare an array of 8 bytes as follows without a trailing null byte:
BYTE HMAC[] = {0x50,0x61,0x73,0x73,0x77,0x6F,0x72,0x64};
The problem is that "strlen(HMAC)" will count at least 8 bytes, and keep counting while traversing undefined memory until it finally (if ever) hits a byte that is zero. At best, you might get lucky because the stack memory always has a zero byte padding your array declaration. More likely it will return a value much larger than 8. Maybe it will crash.
So when you parse the HMAC and MESSAGE field from your protocol packet, you count the number of bytes actually parsed (not including the terminating null). And pass that count into the hmac functions to indicate the size of your data.
I don't know your protocol code, but I hope you aren't using strlen to parse the packet to figure out where the string inside the packet ends. A clever attacker could inject a packet with no null terminator and cause your code do bad things. I hope you are parsing securely and carefully. Typical protocol code doesn't include the null terminating byte in the strings packed inside. Usually the "length" is encoded as an integer field followed by the string bytes. Makes it easier to parse and determine if the length would exceed the packet size read in.