I have a vector which holds byte data (chars) received from a socket. This data holds different datatypes i want to extract. E.g. the first 8 elements (8 Bytes) of the vector are an uint64_t. Now I want to convert these first 8 Bytes to a single uint64.
A workaround I've found is:
// recv_buffer is the vector containing the received Bytes
std::vector<uint64_t> frame_number(recv_buffer.begin(), recv_buffer.begin() + sizeof(uint64_t));
uint64_t frame_num = frame.number.at(0);
Is there a way to extract the data without creating a new vector?
This is an effective method:
C/C++:
uint64_t hexToUint64(char *data, int32_t offset){
uint64_t num = 0;
for (int32_t i = offset; i < offset + 8; i++) {
num = (num << 8) + (data[i] & 0xFF);
}
return num;
}
Java:
long hexToUint64(byte[] data, int offset){
return
((long)data[offset++] << 56 & 0xFF00000000000000L) |
((long)data[offset++] << 48 & 0xFF000000000000L) |
((long)data[offset++] << 40 & 0xFF0000000000L) |
((long)data[offset++] << 32 & 0xFF00000000L) |
((long)data[offset++] << 24 & 0xFF000000L) |
((long)data[offset++] << 16 & 0xFF0000L) |
((long)data[offset++] << 8 & 0xFF00L) |
((long)data[offset++] & 0xFFL);
}
JavaScript:
function hexToUint64(data, offset) {
let num = 0;
let multiple = 0x100000000000000;
for (let i = offset; i < offset + 8; i++ , multiple /= 0x100) {
num += (data[i] & 0xFF) * multiple;
}
return num;
}
One normally uses memcpy or similar to a properly aligned structure, and then ntohl to convert a number from network byte order to computer byte order. ntohl is not part of the C++ specification, but exists in Linux and Windows and others regardless.
uint64_t frame_num;
std::copy(recv_buffer.begin(), recv_buffer.begin() + sizeof(uint64_t), static_cast<char*>(&fame_num);
//or memcpy(&frame_num, recv_buffer.data(), sizeof(frame_num));
frame_num = ntohl(ntohl);
It is tempting to do this for a struct that represents an entire network header, but since C++ compilers can inject padding bytes into structs, and it's undefined to write to the padding, it's better to do this one primitive at a time.
You could perform the conversion byte by byte like this:
int main()
{
unsigned char bytesArray[8];
bytesArray[0] = 0x05;
bytesArray[1] = 0x00;
bytesArray[2] = 0x00;
bytesArray[3] = 0x00;
bytesArray[4] = 0x00;
bytesArray[5] = 0x00;
bytesArray[6] = 0x00;
bytesArray[7] = 0x00;
uint64_t intVal = 0;
intVal = (intVal << 8) + bytesArray[7];
intVal = (intVal << 8) + bytesArray[6];
intVal = (intVal << 8) + bytesArray[5];
intVal = (intVal << 8) + bytesArray[4];
intVal = (intVal << 8) + bytesArray[3];
intVal = (intVal << 8) + bytesArray[2];
intVal = (intVal << 8) + bytesArray[1];
intVal = (intVal << 8) + bytesArray[0];
cout<<intVal;
return 0;
}
I suggest doing the following:
uint64_t frame_num = *((uint64_t*)recv_buffer.data());
You should of course first verify that the amount of data you have in recv_buffer is at least sizeof(frame_num) bytes.
Related
I am trying to implement a four byte value (most significant data first) to compute the total length of data. I found a code snippet to compute this but I didn't get a 4 byte data in the output. Instead I only got a 2 byte value.
char bytesLen[4] ;
unsigned int blockSize = 535;
bytesLen[0] = (blockSize & 0xFF);
bytesLen[1] = (blockSize >> 8) & 0xFF;
bytesLen[2] = (blockSize >> 16) & 0xFF;
bytesLen[3] = (blockSize >> 24) & 0xFF;
std::cout << "bytesLen: " << bytesLen << '\n';
Did I missed something in my code?
No, you didn't. You're outputting the array as a C string, which is null terminated. The third byte is nul so only two characters will be shown.
This is not a rational way to output binary values.
Also you're saving least significant byte first, not most significant. For most significant you have to reverse the order of the bytes.
This shows how to do the same thing without shift operators and bitmasks.
#include <iostream>
#include <iomanip>
// C++11
#include <cstdint>
int main(void)
{
// with union, the memory allocation is shared
union {
uint8_t bytes[4];
uint32_t n;
} length;
// see htonl if needs to be in network byte order
// or ntohl if from network byte order to host
length.n = 535;
std::cout << std::hex;
for(int i=0; i<4; i++) {
std::cout << (unsigned int)length.bytes[i] << " ";
}
std::cout << std::endl;
return 0;
}
If you want ms byte first, then you've reversed the order of the bytes.
You get incorrect output because you treat everything as a C string even though it is not. Get rid of the char type and fix the printing.
In C++, it would be like this:
#include <iostream>
#include <cstdint>
int main()
{
uint8_t bytesLen[sizeof(uint32_t)];
uint32_t blockSize = 535;
bytesLen[3] = (blockSize >> 0) & 0xFF;
bytesLen[2] = (blockSize >> 8) & 0xFF;
bytesLen[1] = (blockSize >> 16) & 0xFF;
bytesLen[0] = (blockSize >> 24) & 0xFF;
bool removeZeroes = true;
std::cout << "bytesLen: 0x";
for(size_t i=0; i<sizeof(bytesLen); i++)
{
if(bytesLen[i] != 0)
{
removeZeroes = false;
}
if(!removeZeroes)
{
std::cout << std::hex << (int)bytesLen[i];
}
}
std::cout << std::endl;
return 0;
}
Here's the fixed code [untested]. Note this won't compile as is. You'll need to reorder it slightly, but it should help:
unsigned char bytesLen[4] ;
unsigned int blockSize = 535;
// little endian
#if 0
bytesLen[0] = (blockSize & 0xFF);
bytesLen[1] = (blockSize >> 8) & 0xFF;
bytesLen[2] = (blockSize >> 16) & 0xFF;
bytesLen[3] = (blockSize >> 24) & 0xFF;
// big endian
#else
bytesLen[3] = (blockSize & 0xFF);
bytesLen[2] = (blockSize >> 8) & 0xFF;
bytesLen[1] = (blockSize >> 16) & 0xFF;
bytesLen[0] = (blockSize >> 24) & 0xFF;
#endif
char tmp[9];
char *
pretty_print(char *dst,unsigned char *src)
{
char *hex = "0123456789ABCDEF";
char *bp = dst;
int chr;
for (int idx = 0; idx <= 3; ++idx) {
chr = src[idx];
*bp++ = hex[(chr >> 4) & 0x0F];
*bp++ = hex[(chr >> 0) & 0x0F];
}
*bp = 0;
return dst;
}
std::cout << "bytesLen: " << pretty_print(tmp,bytesLen) << '\n';
UPDATE:
Based upon your followup question, to concatenate binary data, we can not use string-like functions such as sprintf [because the binary data may have 0x00 inside, which would stop the string transfer short]. Also, if the binary data had no 0x00 in it, the string functions would run beyond the end of the array(s) looking for it, and bad things would happen. The string functions also assume signed char data and when dealing with raw binary, we want to use unsigned char.
Here's something to try:
unsigned char finalData[1000]; // size is just example
unsigned char bytesLen[4];
unsigned char blockContent[300];
unsigned char *dst;
dst = finalData;
memcpy(dst,bytesLen,sizeof(bytesLen));
dst += sizeof(bytesLen);
memcpy(dst,blockContent,sizeof(blockContent));
dst += sizeof(blockContent);
// append more if needed in similar way ...
Note: The above presupposes that blockContent is of fixed size. If it were to have a variable number of bytes, we'd need to replace sizeof(blockContent) with (e.g.) bclen where that is the number of bytes in blockContent
I've been going back trying to find a segmentation error in my program. Very often when the program crashes it is at this point.
unsigned long data = octets[g];
So I have tracked this buffer as being created in the main loop with a fixed defined size. However since it's defined in a if statement in main does it need to be allocated with "new"? Basically after receiving from a TCP socket the char buffer is copied to an unsigned char buffer to check for certain binary data types. So only if data arrives is this called into existance.
INT8U byteArray[BUFFERSIZE];
This buffer is then passed for message ID and crc checking. Is not doing a "new" type allocation the issue because it is in the main loop? I thought it would go out of scope at the end of the "if new data is received" statement.
long calc_crc24q(byte* octets, int start, int last) //start is first byte, last is MSbyte of CRC
{
long crc = CRC24SEED;
for(int g = start; g < last; g++) //should xor from preamble to the end of data
{
unsigned long data = octets[g]; //fault occurs here often
crc = crc ^ data << 16;
for (int i = 0; i < 8; i++)
{
crc <<= 1;
if (crc & 0x1000000)
crc = crc ^ CRC24POLY;
}
}
return crc & 0x00ffffff; //returns an int value with high byte 00 then data in lower 3 bytes
}
//---------------------------------------------
Here is the message id
unsigned int id_message(INT8U* buffer, unsigned int posStart, unsigned int numberbytes, unsigned int& messageLength)
{
unsigned int messID = 0;
unsigned int posEnd;
unsigned int noBytes = 0;
if(buffer[posStart] == Preamble)
{
unsigned int dataLength = (((0x0000 | buffer[posStart+1]) << 8) | buffer[posStart+2]); //0x byte1 byte2
messID = ((0x0000 | (buffer[posStart+3] << 4)) | ((buffer[posStart+4] >> 4) & 0x0F)); //byte1 shift 4 bits add upper 4 bits of byte 2
noBytes = dataLength + 6;
//numberbytes = noBytes;
posEnd = posStart + noBytes - 1;
if(calc_crc24q( buffer, posStart, posEnd-2) != (((0x00000000 | buffer[posEnd-2]) << 16) | ((0x00000000 | buffer[posEnd-1]) << 8) | (0x00000000 | buffer[posEnd])) )
{
cout << "CRC error" << endl;
return 0;
}
//return message type extracted from data segment
messageLength = posStart + noBytes;
return messID;
}
return 255; //unknown type
}
I've made simply program to swap endian in PCM audio (2 channels, 48kHz, 24 bit), but only one channel is swapped correctly, second one is still little Endian (i've checked generated output in CoolEdit 2000). Could anybody give me some guidance what's wrong in my code?
inline int endian_swap(unsigned int x)
{
unsigned char c1, c2, c3, c4;
c1 = x & 255;
c2 = (x >> 8) & 255;
c3 = (x >> 16) & 255;
c4 = (x >> 24) & 255;
return ((int)c1 << 24) + ((int)c2 << 16) + ((int)c3 << 8) + c4;
}
int main()
{
FILE *fpIn, *fpOut;
short x;
fpIn = fopen("audio.pcm", "rb");
fpOut = fopen("out.pcm", "wb");
int test = sizeof(short);
int count = 0;
int swaped = 0;
while( fread(&x, sizeof(int), 1, fpIn) == 1 )
{
swaped = endian_swap(x);
fwrite(&swaped, sizeof(int), 1, fpOut);
}
system("pause");
return 0;
}
Best regards!
You are reading in the file one int at a time. But an int is probably either 16-bit or 32-bit. You say you have 24-bit audio.
You should modify your code to read three char at a time, into a char [3] array. You will then too modify your swap_endian function to operate on a char [3] (this is easy; just swap the contents of the first and last elements of the array!)
You declared short x. Try declaring unsigned int x.
BYTE * srcData;
BYTE * pData;
int i,j;
int srcPadding;
//some variable initialization
for (int r = 0;r < h;r++,srcData+= srcPadding)
{
for (int col = 0;col < w;col++,pData += 4,srcData += 3)
{
memcpy(pData,srcData,3);
}
}
I've tried loop unrolling, but it helps little.
int segs = w / 4;
int remain = w - segs * 4;
for (int r = 0;r < h;r++,srcData+= srcPadding)
{
int idx = 0;
for (idx = 0;idx < segs;idx++,pData += 16,srcData += 12)
{
memcpy(pData,srcData,3);
*(pData + 3) = 0xFF;
memcpy(pData + 4,srcData + 3,3);
*(pData + 7) = 0xFF;
memcpy(pData + 8,srcData + 6,3);
*(pData + 11) = 0xFF;
memcpy(pData + 12,srcData + 9,3);
*(pData + 15) = 0xFF;
}
for (idx = 0;idx < remain;idx++,pData += 4,srcData += 3)
{
memcpy(pData,srcData,3);
*(pData + 3) = 0xFF;
}
}
Depending on your compiler, you may not want memcpy at all for such a small copy. Here is a variant version for the body of your unrolled loop; see if it's faster:
uint32_t in0 = *(uint32_t*)(srcData);
uint32_t in1 = *(uint32_t*)(srcData + 4);
uint32_t in2 = *(uint32_t*)(srcData + 8);
uint32_t out0 = UINT32_C(0xFF000000) | (in0 & UINT32_C(0x00FFFFFF));
uint32_t out1 = UINT32_C(0xFF000000) | (in0 >> 24) | ((in1 & 0xFFFF) << 8);
uint32_t out2 = UINT32_C(0xFF000000) | (in1 >> 16) | ((in2 & 0xFF) << 16);
uint32_t out3 = UINT32_C(0xFF000000) | (in2 >> 8);
*(uint32_t*)(pData) = out0;
*(uint32_t*)(pData + 4) = out1;
*(uint32_t*)(pData + 8) = out2;
*(uint32_t*)(pData + 12) = out3;
You should also declare srcData and pData as BYTE * restrict pointers so the compiler will know they don't alias.
I don't see much that you're doing that isn't necessary. You could change the post-increments to pre-increments (idx++ to ++idx, for instance), but that won't have a measurable effect.
Additionally, you could use std::copy instead of memcpy. std::copy has more information available to it and in theory can pick the most efficient way to copy things. Unfortunately I don't believe that many STL implementations actually take advantage of the extra information.
The only thing that I expect would make a difference is that there's no reason to wait for one memcpy to finish before starting the next. You could use OpenMP or Intel Threading Building Blocks (or a thread queue of some kind) to parallelize the loops.
Don't call memcpy, just do the copy by hand. The function call overhead isn't worth it unless you can copy more than 3 bytes at a time.
As far as this particular loop goes, you may want to look at a technique called Duff's device, which is a loop-unrolling technique that takes advantage of the switch construct.
Maybe changing to a while loop instead of nested for loops:
BYTE *src = srcData;
BYTE *dest = pData;
int maxsrc = h*(w*3+srcPadding);
int offset = 0;
int maxoffset = w*3;
while (src+offset < maxsrc) {
*dest++ = *(src+offset++);
*dest++ = *(src+offset++);
*dest++ = *(src+offset++);
dest++;
if (offset > maxoffset) {
src += srcPadding;
offset = 0;
}
}
Im starting with an array of 100,000 bytes where only the lower 6 bits in each byte have useful data. I need to pack that data into an array of 75,000 bytes as fast as possible, preserving the order of the data.
unsigned int Joinbits(unsigned int in) {}
// 00111111 00111111 00111111 00111111
// 000000 001111 111122 222222
void pack6(
register unsigned char o,
register unsigned char const *i,
unsigned char const *end
)
{
while(i!=end)
{
*o++ = *i << 2u | *(i+1) >> 4u; ++i;
*o++ = (*i & 0xfu) << 4u | *(i+1) >> 2u; ++i;
*o++ = (*i & 0xfcu) << 6u | *(i+1) ; i+=2;
}
}
Will fail if input length is not divisible by 4. Assumes high 2 bits of input are zero.
Completely portable. Reads 4 input bytes 6 times, so 50% inefficiency on reads, however the processor cache and compiler optimiser may help. Attempting to use a variable to save the read may be counter-productive, only an actual measurement can tell.
for(int pos=0; pos<100000; pos+=4)
{
*(int*)out = (in[0] & 0x3F) | ((in[1] & 0x3F)<<6) | ((in[2] & 0x3F)<<12) | ((in[3] & 0x3F)<<18);
in += 4;
out += 3;
}
This is C, I don't know C++. And is probably filled with bugs, and is by no means the fastest way, it probably isn't even fast. But I wanted to just have a go, because it seemed like a fun challenge to learn something, so please hit me with what I did wrong! :D
unsigned char unpacked[100000];
unsigned int packed[75000 / 4];
for (int i = 0; i < (100000 / 6); i += 6) {
unsigned int fourBytes = unpacked[i];
fourBytes += unpacked[i + 1] << 6;
fourBytes += unpacked[i + 2] << 12;
fourBytes += unpacked[i + 3] << 18;
fourBytes += unpacked[i + 4] << 24;
fourBytes += unpacked[i + 5] << 30;
unsigned short twoBytes = unpacked[i + 5] >> 2;
twoBytes += unpacked[i + 6] << 4
twoBytes += unpacked[i + 7] << 10;
packed[i] = fourBytes;
packed[i + 4] = twoBytes;
}