I am sending the data from a file using the Libcurl post callback. The example here shows, how to send data as 1 byte per call from callback function.
I have changed the code so the file is read into chunks. That pretty much works fine.
the current sample code is:
if(sizeleft){
*( char *)ptr = readptr[0];
readptr++;
sizeleft--;
return 1;
}
This example sends the data as 1 byte.
but suppose i have to send it multiple bytes.I have tried to increment readptr by two each time and decreasing sizeleft by two and i return 2bytes at a time.
It didnt work like this and the data is corrupted.
I would appreciate if someone out there could help me out.
Thank you
It's difficult to tell from your question exactly what you're doing, exactly what you'd expect to happen and exactly what actually happens. However it looks like you're on the right track.
The documentation for CURLOPT_READFUNCTION states that size * nitems (a.k.a. size * nmemb in the example) is the upper limit of the number of bytes you may write into buffer, and the return value of your function is the actual number of bytes that you wrote. Returning zero means that you have written everything you wish to write, and your callback function won't be called any more.
If the value you return from your function does not equal the number of bytes that you actually wrote into the buffer, then you can expect corruption.
PS: Something like:
// copy as many bytes as we can, up to either:
// * The number of bytes we have remaining.
// or
// * The space available in ptr.
size_t maxBytes = size * nmemb;
size_t numBytes = std::min (maxBytes, sizeleft);
memcpy (ptr, readptr, numBytes);
readptr += numBytes;
sizeleft -= numBytes;
return numBytes;
Related
I am programming an ESP32 in the Arduino framework. For my application, I need to create a buffer which will store information from both the present and the last time it was accessed. Here is what I am attempting to do.
//first buffer
char buffer1[4];
//second buffer
char buffer2[8];
void setup {
//setup
}
//buffer1 values will change with each iteration of loop from external inputs
//buffer2 must store most recent values of buffer1 plus values of buffer1 from when loop last ran
for example:
**loop first iteration**
void loop {
buffer1[0] = {1};
buffer1[1] = {2};
buffer1[2] = {3};
buffer1[3] = {1};
saveold(); //this is the function I'm trying to implement to save values to buffer2 in an element-wise way
}
//value of buffer2 should now be: buffer2 = {1,2,3,1,0,0,0,0}
**loop second iteration**
void loop {
buffer1[0] = {2};
buffer1[1] = {3};
buffer1[2] = {4};
buffer1[3] = {2};
saveold();
}
//value of buffer2 should now be: buffer2 = {2,3,4,2,1,2,3,1}
From what I've been able to understand through searching online, the "saveold" function I'm trying to make
should implement some form of memmove for these array operations
I've tried to piece it together, but I always overwrite the value of buffer2 instead of somehow shifting new values in, while retaining the old ones
This is all I've got:
void saveold() {
memmove(&buffer2[0], &buffer1[0], (sizeof(buffer1[0]) * 4));
}
From my understanding, this copies buffer1 starting from index position 0 to buffer2, starting at index position 0, for 4 bytes (where 1 char = 1 byte).
Computer science is not my backround, so perhaps there is some fundamental solution or strategy that I am missing. Any pointers would be appreciated.
You have multiple options to implement saveold():
Solution 1
void saveold() {
// "shift" lower half into upper half, saving recent values (actually it's a copy)
buffer2[4] = buffer2[0];
buffer2[5] = buffer2[1];
buffer2[6] = buffer2[2];
buffer2[7] = buffer2[3];
// copy current values
buffer2[0] = buffer[0];
buffer2[1] = buffer[1];
buffer2[2] = buffer[2];
buffer2[3] = buffer[3];
}
Solution 2
void saveold() {
// "shift" lower half into upper half, saving recent values (actually it's a copy)
memcpy(buffer2 + 4, buffer2 + 0, 4 * sizeof buffer2[0]);
// copy current values
memcpy(buffer2 + 0, buffer1, 4 * sizeof buffer1[0]);
}
Some notes
There are even more ways to do it. Anyway, choose the one you understand best.
Be sure that buffer2 is exactly double size of buffer1.
memcpy() can be used safely if source and destination don't overlap. memmove() checks for overlaps and reacts accordingly.
&buffer1[0] is the same as buffer1 + 0. Feel free to use the expression you better understand.
sizeof is an operator, not a function. So sizeof buffer[0] evaluates to the size of buffer[0]. A common and most accepted expression to calculate the size of an array dimension is sizeof buffer1 / sizeof buffer1[0]. You only need parentheses if you evaluate the size of a data type, like sizeof (int).
Solution 3
The last note leads directly to this improvement of solution 1:
void saveold() {
// "shift" lower half into upper half, saving recent values
size_t size = sizeof buffer2 / sizeof buffer2[0];
for (int i = 0; i < size / 2; ++i) {
buffer2[size / 2 + i] = buffer2[i];
}
// copy current values
for (int i = 0; i < size / 2; ++i) {
buffer2[i] = buffer1[i];
}
}
To apply this knowledge to solution 2 is left as an exercise for you. ;-)
The correct way to do this is to use buffer pointers, not by doing hard-copy backups. Doing hardcopies with memcpy is particularly bad on slow legacy microcontrollers such as AVR. Not quite sure what MCU this ESP32 got, seems to be some oddball one from Tensilica. Anyway, this answer applies universally for any processor where you have more data than CPU data word length.
perhaps there is some fundamental solution or strategy that I am missing.
Indeed - it really sounds that what you are looking for is a ring buffer. That is, an array of fixed size which has a pointer to the beginning of the valid data, and another pointer at the end of the data. You move the pointers, not the data. This is much more efficient both in terms of execution speed and RAM usage, compared to making naive hardcopies with memcpy.
I have a function which opens a file from an SD card, uses the file size to set the size of a buffer, writes a block of information to that buffer, then does something with that information, as shown in this code:
char filename = "filename.txt";
uint16_t duration;
uint16_t pixel;
int q = 0;
int w = 0;
bool largefile;
File f;
int readuntil;
long large_buffer;
f = SD.open(filename);
if(f.size() > 3072) {
w = 3072;
} else {
w = f.size();
}
uint8_t buffer[w];
while(f.available()) {
f.read(buffer, sizeof(buffer));
while(q < sizeof(buffer)) {
doStuffWithInformation(buffer[q++]);
}
q=0;
}
f.close;
This works great with smaller file sizes, but anything over the hard limit buffer size of 3072 (which I arrived at empirically, its just the amount of memory that can be safely committed to this function), runs into a problem. Larger files read fine until they hit the last loop of while(f.available()), where they read the end of the file, but then continue reading the buffer, the tail end of which is filled with data from the last loop, that wasn't overwritten by the latest f.read(). How can I make sure that the last loop of the while(f.available()) function only works with the information that was written to the buffer during the current loop? My only idea right now is to solve for factors of the file size, and set the buffer size as the largest factor less than 3072, but this seems intensive to run every time this function is called. Is there an elegant solution staring me in the face?
Your program is not behaving correctly because f.read() is not guaranteed to read the whole buffer. Moreover, it is bound to happen when you read the last chunk of the file, unless the file size is a factor of buffer size (3072 in your case).
While Arduino specification (https://www.arduino.cc/en/Reference/FileRead) doesn't say so, SD.read function returns the number of bytes read. See code of the library here: https://github.com/arduino-libraries/SD/blob/master/src/utility/SdFile.cpp, int16_t SdFile::read(void* buf, uint16_t nbyte)
Knowing that, you should change your loop as following (while also rewriting it as a for loop for better readability and removing q definition above):
while(f.available()) {
uint16_t sz = f.read(buffer, sizeof(buffer));
for (uint16_t q = 0; q < sz; ++q) {
doStuffWithInformation(buffer[q]);
}
}
On a side note, now, when you have this logic in place, it would make sense for you to do away with variable length array and use a fixed buffer of size 512 - the standard sector size on the SD card. Most likely, it will yield the same performance in regards to read, and slightly better performance in regards to sizeof, which will becomes a compile-time constant rather than a run-time calculation. This also makes your program simpler. This makes for following code:
f = SD.open(filename);
...
uint8_t buffer[512];
I am trying to send (ideally) a 2d buffer from one process to another, over a Message Queue, but i am attempting to do it first with a 1d buffer.
The functions called to initialization the queue are the following:
HANDLE MsgQueueCommunicator::InitMessageQueue_data(bool IsRead,wchar16_t* wQueueName)
{
MSGQUEUEOPTIONS msgopts;
msgopts.dwSize = sizeof(MSGQUEUEOPTIONS);
msgopts.dwFlags = MSGQUEUE_ALLOW_BROKEN;//0;
msgopts.dwMaxMessages = 0;
msgopts.cbMaxMessage = sizeof(data[20]);
msgopts.bReadAccess = IsRead;
HANDLE hq = CreateMsgQueue(wQueueName, &msgopts);
if ( hq == NULL )
{
return NULL;
}
return hq;
}
Queue initialization in process 1:
HANDLE hMsgQueueR = MsgQueueCommunicator::getInstance()->InitMessageQueue_data(true, L"CommDataStreaming");
Queue initialization in process 2:
HANDLE s_hMsgQueue_Communication = MsgQueueCommunicator::getInstance()->InitMessageQueue_data(false,L"CommDataStreaming");
To write to the queue, i call the following functions:
BOOL MsgQueueCommunicator::Write_Array_To_Queue(HANDLE hq,double data[20])
{
return WriteMsgQueue(hq,(LPVOID)&data, sizeof(data),INFINITE,0);
}
MsgQueueCommunicator::getInstance()->Write_Array_To_Queue(s_hMsgQueue_Communication, usb_data);
Where usb_data is a 1d double array.
To read from the queue, i call the following functions:
BOOL MsgQueueCommunicator::Read_Array_From_Msg_Queue(HANDLE hq,double data[20])
{
DWORD dwBytesRead;
DWORD dwFlags;
return ReadMsgQueue(hq, (LPVOID)&data, sizeof(data), &dwBytesRead, INFINITE, &dwFlags);
}
MsgQueueCommunicator::getInstance()->Read_Array_From_Msg_Queue(hMsgQueueR, usb_data);
Where usb_data is again a 1d double array.
Now, when i check the values that are placed into usb_data[20] before it is written to the queue, i can see that they are non-zero integers. However, when i read the array from the queue and check its values, they are zero. Im not sure what is causing this issue. I've used message queues to send single values, strings, and structs, so i figured i would be able to follow the same procedure to send over an array, but this does not seem to be the case, unless i am overlooking something.
My question is, can i send arrays/buffers over a message queue, and if yes, have I set it up properly?
Note:This is being developed in a windows embedded compact 7 environment and VS2008.
There are several problems with the code provided.
1) Wrong parameter values - you do not need to take an address of the data buffer since the variable is already a pointer to the beginning of the memory that contains the elements. So change (LPVOID)&data to (LPVOID)data.
2) Wrong size - the sizeof operator will return 4 since that is the size of the pointer. In your case you would need to pass 160 as the size (20 * sizeof(double)).
As for variable size writes - this gets a bit more complicated since you need to know how much data to read at the other end. What you can do is use lets say first/first two/first four bytes of the buffer to contain size and then proceed with the data. Then you can have a function that accepts a double array of variable length and writes it. For example:
BOOL Write_Array_To_Queue(HANDLE hq,double data[], unsigned int count)
{
size_t buffer_size = sizeof(count) + count * sizeof(double);
BYTE* buffer = new BYTE[buffer_size];
memcpy(buffer, &count, sizeof(count));
memcpy(buffer + sizeof(count), &data, sizeof(double) * count);
return WriteMsgQueue(hq,(LPVOID)buffer, buffer_size,INFINITE,0);
}
or
BOOL Write_Array_To_Queue(HANDLE hq,double data[], unsigned int count)
{
return WriteMsgQueue(hq,(LPVOID)&count, sizeof(count),INFINITE,0) && WriteMsgQueue(hq,(LPVOID)data, sizeof(double) * count,INFINITE,0);
}
and then in the receiving side you would first read out an unsigned int and then read as much data as denoted by the read value.
I have a program that generates files containing random distributions of the character A - Z. I have written a method that reads these files (and counts each character) using fread with different buffer sizes in an attempt to determine the optimal block size for reads. Here is the method:
int get_histogram(FILE * fp, long *hist, int block_size, long *milliseconds, long *filelen)
{
char *buffer = new char[block_size];
bzero(buffer, block_size);
struct timeb t;
ftime(&t);
long start_in_ms = t.time * 1000 + t.millitm;
size_t bytes_read = 0;
while (!feof(fp))
{
bytes_read += fread(buffer, 1, block_size, fp);
if (ferror (fp))
{
return -1;
}
int i;
for (i = 0; i < block_size; i++)
{
int j;
for (j = 0; j < 26; j++)
{
if (buffer[i] == 'A' + j)
{
hist[j]++;
}
}
}
}
ftime(&t);
long end_in_ms = t.time * 1000 + t.millitm;
*milliseconds = end_in_ms - start_in_ms;
*filelen = bytes_read;
return 0;
}
However, when I plot bytes/second vs. block size (buffer size) using block sizes of 2 - 2^20, I get an optimal block size of 4 bytes -- which just can't be correct. Something must be wrong with my code but I can't find it.
Any advice is appreciated.
Regards.
EDIT:
The point of this exercise is to demonstrate the optimal buffer size by recording the read times (plus computation time) for different buffer sizes. The file pointer is opened and closed by the calling code.
There are many bugs in this code:
It uses new[], which is C++.
It doesn't free the allocated memory.
It always loops over block_size bytes of input, not bytes_read as returned by fread().
Also, the actual histogram code is rather inefficient, since it seems to loop over each character to determine which character it is.
UPDATE: Removed claim that using feof() before I/O is wrong, since that wasn't true. Thanks to Eric for pointing this out in a comment.
You're not stating what platform you're running this on, and what compile time parameters you use.
Of course, the fread() involves some overhead, leaving user mode and returning. On the other hand, instead of setting the hist[] information directly, you're looping through the alphabet. This is unnecessary and, without optimization, causes some overhead per byte.
I'd re-test this with hist[j-26]++ or something similar.
Typically, the best timing would be achieved if your buffer size equals the system's buffer size for the given media.
I have a byte array like this:
lzo_bytep out; // my byte array
size_t uncompressedImageSize = 921600;
out = (lzo_bytep) malloc((uncompressedImageSize +
uncompressedImageSize / 16 + 64 + 3));
wrkmem = (lzo_voidp) malloc(LZO1X_1_MEM_COMPRESS);
// Now the byte array has 802270 bytes
r = lzo1x_1_compress(imageData, uncompressedImageSize,
out, &out_len, wrkmem);
How can I split it into smaller parts under 65,535 bytes (the byte array is one large image which I want to sent over UDP which has upper limit 65,535 bytes) and then join those small chunks back into a continuous array?
The problem with doing this is that the UDP packets can arrive out or order, or be dropped. Use TCP for this; that's what it's for.
You don't have to "split" the array. You just have to point into different parts of it.
Assuming you're using a typical UDP write() function, it takes several arguments. One of them is a pointer to the buffer and the other is the length.
If you want to get the first 65535 bytes, your buffer is at wrkmem and the length is 65535.
For the second 65535 bytes, your buffer is at wrkmem + 65535 and your length is 65535.
The third 65535 bytes, your buffer is at wrkmem + 2 * 65535 and your length is 65535.
Get it?
(That said, the other posters are correct. You should be using TCP).
On the other side, when you want to re-join the array, you must allocate enough memory for the whole thing, then use a copy function like memcpy() to copy the arriving chunks into their correct position. Remember that UDP may not deliver the pieces in order and may not deliver all of them.
You might wish to try a message based middleware like ØMQ and feed the entire compressed image as one message and have the middleware run asynchronously and manage redelivery at the fastest speed possible. It provides a BSD socket compatible API and so can be easy to migrate code over and allows you to easily swap between various underlying transport protocols as required.
Other message systems are available.
void my_free (void *data, void *hint)
{
free (data);
}
/* ... */
size_t uncompressedImageSize = 921600, compressedImageSize = 0;
size_t out_len = (uncompressedImageSize + uncompressedImageSize / 16 + 64 + 3);
lzo_bytep out = (lzo_bytep)malloc (out_len);
lzo_voidp wkrmem = (lzo_voidp)malloc (LZO1X_1_MEM_COMPRESS);
zmq_msg_t msg;
rc = lzo1x_1_compress (imageData, uncompressedImageSize,
out, &compressedImageSize, wrkmem);
assert (compressedImageSize > 0);
rc = zmq_msg_init_data (&msg, out, compressedImageSize, my_free, NULL);
assert (rc == 0);
/* Send the message to the socket */
rc = zmq_send (socket, &msg, 0);
assert (rc == 0);