I'm using Microsoft's cpprest sdk to read binary data over the internet.
My variable stream below is of type concurrency::streams::istream. I'm trying to read a million rows of type struct row and process them. I see that I don't get all the bytes I request. I suspect there is a good way of coding this but I haven't been able to figure it out. I also suspect that my casting to extract a row from the buffer is not the right way to do things. Any help would be appreciated.
struct row {
unsigned long long tag_id : 32, day : 32;
unsigned long long time;
double value;
};
size_t row_count = 1000000;
concurrency::streams::container_buffer<vector<uint8_t>> buffer;
size_t bytes_requested = sizeof(row) * row_count;
size_t bytes_received = stream.read(buffer, bytes_requested).get();
// bytes_received does not always match bytes requested
for (size_t i = 0; i < row_count; ++i) {
row &r = *(row *) &buffer.collection()[i * sizeof(row)];
// do something with row here
}
Related
I am working on a project, it reads data(24bits per sample) from binary files, decompress it to 32bits and convert it signed int.
I have this function for the decompress and convert tasks, and it causes wasting a lot of time in the project for huge data.
signed int convertedSample(QByteArray samp){
QString sample = QString(samp.toHex() + "00");
bool ok;
signed int decimalSample = sample.toInt(&ok, 16);
return decimalSample;
}
To read data from files and store it, I do:
while(!file.atEnd()){
QByteArray chunkSamples = file.read(chunkSize);// chunkSize here is the size of 1 million of samples
for (int i = 0; i < chunkSamples.size() ; i++){
QByteArray samp = chunkSamples.mid(i * 3, 3); // 24bits
buffer.append(convertedSample(samp)); // buffer is a QVector of int
}
Any help to make it faster ?
Thanks to the guys in the comments, I've made some improvements over time. I avoided using strings for the conversion because it seems that using strings is always slow and replaced it with using bitwise shift and OR to combine the individual bytes into int.
I replaced this code:
QString sample = QString(samp.toHex() + "00");
bool ok;
int decimalSample = sample.toInt(&ok, 16);
with :
int decimalSample = ((samp[0] << 24) | (samp[1] << 16) | (samp[2] << 8));
Hi i am trying to write a TCP connection in poco. the client sends a packet with this fields :
packetSize : int
date : int
ID : int
so the first 4 bytes contains the packet size. in the receive side i have this code :
int packetSize = 0;
char *bufferHeader = new char[4];
// receive 4 bytes that contains packetsize here
socket().receiveBytes(bufferHeader, sizeof(bufferHeader), MSG_WAITALL);
Poco::MemoryInputStream *inStreamHeader = new Poco::MemoryInputStream(bufferHeader, sizeof(bufferHeader));
Poco::BinaryReader *BinaryReaderHeader = new Poco::BinaryReader(*inStreamHeader);
(*BinaryReaderHeader) >> packetSize; // now we have the full packet size
Now I am trying to store all remaining incoming bytes into one array for future binary reading :
int ID = 0;
int date = 0;
int availableBytes = 0;
int readedBytes = 0;
char *body = new char[packetSize - 4];
do
{
char *bytes = new char[packetSize - 4];
availableBytes = socket().receiveBytes(bytes, sizeof(bytes), MSG_WAITALL);
if (availableBytes == 0)
break;
memcpy(body + readedBytes, bytes, availableBytes);
readedBytes += availableBytes;
} while (availableBytes > 0);
Poco::MemoryInputStream *inStream = new Poco::MemoryInputStream(body, sizeof(body));
Poco::BinaryReader *BinaryReader = new Poco::BinaryReader(*inStream);
(*BinaryReader) >> date;
(*BinaryReader) >> ID;
cout << "date :" << date << endl;
cout << "ID :" << ID << endl;
the problem is the byte block of body is not storing the remaining bytes , it has always only the first 4 bytes (date). so in the out put the date is correct but the ID is not as expected. I tried to Stream it without Block copy and manually receive the each field without loop, it was just fine and had expected data. but when i try to store the incoming bytes into one array and then pass that array to a memorystream to read it, i have only the first block correct and expected!!
I really need to store all incoming bytes into one array and then read whole that array, how should i change my code?
thanks alot
I see two errors in your code. Or, more precisely, an error you do twice.
You confuse sizeof of char[] with sizeof of char *; the first is the number of characters in the array, the second is the size of the pointer: typically 4 or 8 bytes, depending on the memory model.
So, when you write
availableBytes = socket().receiveBytes(bytes, sizeof(bytes), MSG_WAITALL);
you are asking for 4 (I suppose) bytes. This is not serious as you continue to ask other bytes until the message is finished.
The real problem is the following instruction
Poco::MemoryInputStream *inStream = new Poco::MemoryInputStream(body, sizeof(body));
where you transfer only sizeof(char *) bytes in inStream
You should substitute sizeof(body) and sizeof(bytes) with packetSize - 4.
P.s.: sorry for my bad english
Edit: I've seen another error. In this instruction
char *bytes = new char[packetSize - 4];
you allocate packetSize - 4 chars. This memory in never deleted and in allocated in the do ... while() cycle.
You can allocate bytes outside of the cycle (togheter with body).
Edit 2016.03.17
Proposed solution (caution: non tested)
size_t toRead = packetSize - 4U;
size_t totRead = 0U;
size_t nowRead;
char * body = new char[toRead];
do
{
nowRead += socket().receiveBytes(body+totRead, toRead-totRead,
MSG_WAITALL);
if ( 0 == nowRead )
throw std::runtime_error("shutdown from receiveBytes()");
totRead += nowRead;
} while ( totRead < toRead );
Poco::MemoryInputStream *inStream = new Poco::MemoryInputStream(body,
toRead);
delete[] body;
body = NULL;
I'm having an issue reading in some bytes from a yuv file (it's 1280x720 if that matters) and was hoping someone could point out what I'm doing wrong. I'm getting different results using the read command and using an istream iterator . Here's some example code of what I'm trying to do:
void readBlock(std::ifstream& yuvFile, YUVBlock& destBlock, YUVConfig& config, const unsigned int x, const unsigned int y, const bool useAligned = false)
{
//Calculate luma offset
unsigned int YOffset = (useAligned ? config.m_alignedYFileOffset : config.m_YFileOffset) +
(destBlock.yY * (useAligned ? config.m_alignedYUVWidth : config.m_YUVWidth) + destBlock.yX);// *config.m_bitDepth;
//Copy Luma data
//yuvFile.seekg(YOffset, std::istream::beg);
for (unsigned int lumaY = 0; lumaY < destBlock.m_YHeight && ((lumaY + destBlock.yY) < config.m_YUVHeight); ++lumaY)
{
yuvFile.seekg(YOffset + ((useAligned ? config.m_alignedYUVWidth : config.m_YUVWidth)/* * config.m_bitDepth*/) * (lumaY), std::istream::beg);
int copySize = destBlock.m_YWidth;
if (destBlock.yX + copySize > config.m_YUVWidth)
{
copySize = config.m_YUVWidth - destBlock.yX;
}
if (destBlock.yX >= 1088 && destBlock.yY >= 704)
{
char* test = new char[9];
yuvFile.read(test, 9);
delete[] test;
yuvFile.seekg(YOffset + ((useAligned ? config.m_alignedYUVWidth : config.m_YUVWidth)/* * config.m_bitDepth*/) * (lumaY));
}
std::istream_iterator<uint8_t> start = std::istream_iterator<uint8_t>(yuvFile);
std::copy_n(start, copySize, std::back_inserter(destBlock.m_yData));
}
}
struct YUVBlock
{
std::vector<uint8_t> m_yData;
std::vector<uint8_t> m_uData;
std::vector<uint8_t> m_vData;
unsigned int m_YWidth;
unsigned int m_YHeight;
unsigned int m_UWidth;
unsigned int m_UHeight;
unsigned int m_VWidth;
unsigned int m_VHeight;
unsigned int yX;
unsigned int yY;
unsigned int uX;
unsigned int uY;
unsigned int vX;
unsigned int vY;
};
This error only seems to be happening at X =1088 and Y = 704 in the image. I'm expecting to see a byte value of 10 as the first byte I read back. When I use
yuvFile.read(test, 9);
I get 10 as my first byte. When I use the istream iterator:
std::istream_iterator<uint8_t> start = std::istream_iterator<uint8_t>(yuvFile);
std::copy_n(start, copySize, std::back_inserter(destBlock.m_yData));
The first byte I read is 17. 17 is the byte after 10 so it seems the istream iterator skips the first byte.
Any help would be appreciated
There is a major difference between istream::read and std::istream_iterator.
std::istream::read performs unformatted read.
std::istream_iterator performs formatted read.
From http://en.cppreference.com/w/cpp/iterator/istream_iterator
std::istream_iterator is a single-pass input iterator that reads successive objects of type T from the std::basic_istream object for which it was constructed, by calling the appropriate operator>>.
If your file was created using std::ostream::write or fwrite, you must use std::istream::read or fread to read the data.
If your file was created using any of the methods that create formatted output, such as std::ostream::operato<<(), fprintf, you have a chance to read the data using std::istream_iterator.
I am stuck since 2 days with a seemingly simple calculation.
But I just don't get it.
I am encoding an audio file with a compressing algorithm.
The entire audio file is separated into "chunks" of 960 bytes.
Each chunk is compressed to 60 bytes.
My uncompressed file is 1480320 bytes long.
My encoded file is 46320 bytes long.
Something seems to be wrong.
I tried to calculate the theoretic uncompressed file size from the file size of the encoded audio.
Here is how the file is encoded:
short *m_in;
short *m_out;
unsigned char *m_data;
unsigned char *m_fbytes;
int m_max_frame_size;
int m_frame_size;
int m_sampling_rate;
int m_max_payload_bytes;
int m_bitrate_bps;
int m_iByteLen1FrameEncoded;
int m_iByteLen1FrameDecoded;
m_sampling_rate=48000;
m_max_frame_size = 960*6;
m_max_payload_bytes=1500;
m_bitrate_bps= 24000;
m_iByteLen1FrameEncoded=60;
m_iByteLen1FrameDecoded=960;
m_in = (short*)malloc(m_max_frame_size*sizeof(short));
m_out = (short*)malloc(m_max_frame_size*sizeof(short));
m_data = (unsigned char*)calloc(m_max_payload_bytes,sizeof(char));
m_fbytes = (unsigned char*)malloc(m_iByteLen1FrameDecoded*sizeof(short));
FILE *fin= fopen(uPathInput.c_str(), "rb");
FILE *fout=fopen(uPathOutput.c_str(), "wb");
int curr_read=0;
int stop=0;
while (!stop)
{
int err;
err = fread(m_fbytes, sizeof(short), 960, fin);
curr_read = err;
for(int i=0;i<curr_read;i++)
{
opus_int32 s;
s=m_fbytes[2*i+1]<<8|m_fbytes[2*i];
s=((s&0xFFFF)^0x8000)-0x8000;
m_in[i]=s;
}
if (curr_read < 960)
{
for (int i=curr_read;i<960;i++)
{
m_in[i] = 0;
}
stop = 1;
}
//iLen will always return 60, so I guess the 960 bytes are compressed to 60 bytes, right?
int iLen = opus_encode(m_enc, m_in, m_iByteLen1FrameDecoded, m_data, m_max_payload_bytes);
if (fwrite(m_data, 1, iLen, fout) !=iLen)
{
fprintf(stderr, "Error writing.\n");
}
}
fclose(fin);
fclose(fout);
}
The compression ratio seems to be
960/60 = 16
So I calculated 46320 bytes * 16.
But that gets me to 741120 bytes.
And that doesn't fit. I expected it to be 1480320 bytes.
I am trying to find the error in my calculation, but I just don't manage.
Does anybody see where I went wrong?
Thank you very much for any help!
Right, to expand on my comments. The problem is found here:
fread(m_fbytes, sizeof(short), 960, fin);
You're reading 960 shorts, which should be 2 bytes wide, so you're really reading 1920 bytes. If opus_encode() returns the compressed size in bytes, that would make the compression ration 32 as Robert observed.
I'd also simplify the code processing the chunks:
size_t ITEM_SIZE = sizeof(short);
int ITEM_COUNT = 960;
// fread should first return a short item count, then zero
size_t shorts_read = 0;
while (shorts_read = fread(m_fbytes, ITEM_SIZE, ITEM_COUNT, fin)) {
size_t i = 0;
for (; i<read; i++) {
opus_int32 s;
// etc.
}
for (; i < ITEM_COUNT; i++) {
m_in[i] = 0;
}
// opus_encode() etc
}
You get rid of the useless stop flag and a level of nesting, and the construct is idiomatic for "read until you can't." (See this SO question.)
I retract what I mentioned about the code being hokey, I thought fread returns the bytes read, not the items read.
I'm tying to implement a basic audio delay - but all I'm getting is garbage, probably something very obvious - but I can't seem to spot it...
Audio is processed via buffers that are determined at runtime.
I think I'm doing something horribly wrong with the pointers, tried looking at some other code - but they all seem "incomplete" always something rudimentary is missing - probably what's miss in my code as well.
// Process audio
// 1
void Gain::subProcessSimpleDelay( int bufferOffset, int sampleFrames )
{
// Assign pointers to your in/output buffers.
// Each buffer is an array of float samples.
float* in1 = bufferOffset + pinInput1.getBuffer();
float* in2 = bufferOffset + pinInput2.getBuffer();
float* out1 = bufferOffset + pinOutput1.getBuffer();
// SampleFrames = how many samples to process (can vary).
// Repeat (loop) that many times
for( int s = sampleFrames; s > 0; --s )
{
// get the sample 'POINTED TO' by in1.
float input1 = *in1;
float feedback = *in2;
float output;
unsigned short int p, r;
unsigned short int len;
len = 600;
// check at delay length calculation
if (len > 65535)
len = 65535;
// otherwise, a length of 0 will output the input from
// 65536 samples ago
else if (len < 1)
len = 1;
r = p - len; // loop
output = buffer[r];
buffer[p] = input1 + output * feedback;
p++;
*out1 = output;
// store the result in the output buffer.
// increment the pointers (move to next sample in buffers).
in1++;
in2++;
out1++;
}
}
Could anyone tell me what's wrong?
You haven't initialized p. Other things to be careful of in this code:-
Are you sure that sampleFrames + bufferOffset is less than the size of your input and output buffers? You could really do with a way to check that.
It's not clear where buffer comes from, or what else might be writing to it. If it's garbage before your code runs, you're going to end up with garbage everywhere, because the first thing you do is read from it.
You don't say what types pinInput1.getBuffer() etc. return. If they return a char*, and you just know that it happens to point to an array of floats, you need to cast the result to float* before you do any pointer arithmetic, to make sure you're advancing to the next float in the array, not the next byte of the array.