I am trying to implement a simple file transfer. Below here is two methods that i have been testing:
Method one: sending and receiving without splitting the file.
I hard coded the file size for easier testing.
sender:
send(sock,buffer,107,NULL); //sends a file with 107 size
receiver:
char * buffer = new char[107];
recv(sock_CONNECTION,buffer,107,0);
std::ofstream outfile (collector,std::ofstream::binary);
outfile.write (buffer,107);
The output is as expected, the file isn't corrupted because the .txt file that i sent contains the same content as the original.
Method two: sending and receiving by splitting the contents on receiver's side. 5 bytes each loop.
sender:
send(sock,buffer,107,NULL);
Receiver:
char * buffer = new char[107]; //total file buffer
char * ptr = new char[5]; //buffer
int var = 5;
int sizecpy = size; //orig size
while(size > var ){ //collect bytes
recv(sock_CONNECTION,ptr,5,0);
strcat(buffer,ptr); //concatenate
size= size-var; //decrease
std::cout<<"Transferring.."<<std::endl;
}
std::cout<<"did it reach here?"<<std::endl;
char*last = new char[size];
recv(sock_CONNECTION,last,2,0); //last two bytes
strcat(buffer,last);
std::ofstream outfile (collector,std::ofstream::binary);
outfile.write (buffer,107);
Output: The text file contains invalid characters especially at the beginning and the end.
Questions: How can i make method 2 work? The sizes are the same but they yield different results. the similarity of the original file and the new file on method 2 is about 98~99% while it's 100% on method one. What's the best method for transferring files?
What's the best method for transferring files?
Usually I'm not answering questions like What's the best method. But in this case it's obvious:
You sent the file size and a checksum in network byte order, when starting a transfer
Sent more header data (e.g filename) optionally
The client reads the file size and the checksum, and decodes it to host byte order
You sent the file's data in reasonably sized chunks (5 bytes isn't a reasonable size), chunks should match tcp/ip frames maximum available payload size
You receive chunk by chunk at the client side until the previously sent file size is matched
You calculate the checksum for the received data at the client side, and check if it matches the one that was received beforhand
Note: You don't need to combine all chunks in memory at the client side, but just append them to a file at a storage medium. Also the checksum (CRC) usually can be calculated from running through data chunks.
Disagree with Galik. Better not to use strcat, strncat, or anything but the intended output buffer.
TCP is knda fun. You never really know how much data you are going to get, but you will get it or an error.
This will read up to MAX bytes at a time. #define MAX to whatever you want.
std::unique_ptr<char[]> buffer (new char[size]);
int loc = 0; // where in buffer to write the next batch of data
int bytesread; //how much data was read? recv will return -1 on error
while(size > MAX)
{ //collect bytes
bytesread = recv(sock_CONNECTION,&buffer[loc],MAX,0);
if (bytesread < 0)
{
//handle error.
}
loc += bytesread;
size= size-bytesread; //decrease
std::cout<<"Transferring.."<<std::endl;
}
bytesread = recv(sock_CONNECTION,&buffer[loc],size,0);
if (bytesread < 0)
{
//handle error
}
std::ofstream outfile (collector,std::ofstream::binary);
outfile.write (buffer.get(),size);
Even more fun, write into the output buffer so you don't have to store the whole file. In this case MAX should be a bigger number.
std::ofstream outfile (collector,std::ofstream::binary);
char buffer[MAX];
int bytesread; //how much data was read? recv will return -1 on error
while(size)
{ //collect bytes
bytesread = recv(sock_CONNECTION,buffer,MAX>size?size:MAX,0);
// MAX>size?size:MAX is like a compact if-else: if (MAX>size){size}else{MAX}
if (bytesread < 0)
{
//handle error.
}
outfile.write (buffer,bytesread);
size -= bytesread; //decrease
std::cout<<"Transferring.."<<std::endl;
}
The initial problems I see are with std::strcat. You can't use it on an uninitialized buffer. Also you are not copying a null terminated c-string. You are copying a sized buffer. Better to use std::strncat for that:
char * buffer = new char[107]; //total file buffer
char * ptr = new char[5]; //buffer
int var = 5;
int sizecpy = size; //orig size
// initialize buffer
*buffer = '\0'; // add null terminator
while(size > var ){ //collect bytes
recv(sock_CONNECTION,ptr,5,0);
strncat(buffer, ptr, 5); // strncat only 5 chars
size= size-var; //decrease
std::cout<<"Transferring.."<<std::endl;
}
beyond that you should really as error checking so the sockets library can tell you if anything went wrong with the communication.
Related
Background:
I have a .wav file saved on an SD card. I would like to transfer that file to a server using my esp32. I am using node red to handle the server side activities.
Method Employed:
open the file in binary mode.
evaluate the size of the file
decide on a max upload size and allocate a buffer
Read the file and store to the buffer.
use http post to send data to the server.
if file is too large to send in a single buffer then divide the file up and send multiple http posts.
Problem:
I can successfully send text files. when I try to send .wav files the size of the sent wave file increases and the file is corrupted. Analyzing the file is difficult as its not all text, what I have done is open the file in notepad++ to see if I can spot anything. Everything should be the same in theory but several characters are coming up as blank squares in the transferred file and some are coming up as the exact same.
Analysis/Theory:
I am quite lost as to what the issue is. My leading theory is that a wave file is written in int16_t but in order to post the data it needs to be * uint8_t, maybe when the casting of the int16 to a uint8 data is lost, I looked at trying to change a int16_t into two int8_t bytes as done here https://stackoverflow.com/a/53374797/14050333 but had no luck, maybe I'm jumping to conclusions. Any help would be hugely appreciated!
Code:
Full code used to sell text files.
void loop()
{
WiFiClient client;
Serial.println("starting file upload");
IPAddress host(192, 168, 0, 37);
int port = 1880;
if (!client.connect(host, port))
{ // check connection to host if untrue internet connection could be down
Serial.println("couldn't connect to host");
}
HTTPClient http;
const char* serverName = "http://192.168.0.37:1880/sensor_file";
http.begin(client, serverName);
char *fname = "/sdcard/test_text.txt";
FILE *fp = fopen(fname, "rb"); // read in bytes
//get file size
fseek(fp, 0, SEEK_END); //send file pointer to end of file
int file_size = ftell(fp); //get end position of file
fseek(fp, 0, SEEK_SET); //send pointer back to start
int max_upload_size = 10; // array size, larger = less uploads but too large can cause memory issues
int num_of_uploads = file_size / max_upload_size; // figure out how many evenly sized upload chunks we need
int num_of_uploads_mod = file_size % max_upload_size; //find out size of remaining upload chunk if needed
int i;
//upload file in even chunks
if (num_of_uploads > 0)
{
char buff1[max_upload_size+1] = {}; // array to save file too. add 1 for end of array symbol '\n'
for (i = 0; i < num_of_uploads; i++)
{
fread(buff1, sizeof(buff1)-1, 1, fp); // -1 as don't want to count the '\n'
http.addHeader("File_name", "test file"); //header to say what the file name is
int httpResponseCode = http.POST((uint8_t *)buff1, sizeof(buff1)-1); //send data. Datatype is (uint8_t *)
}
}
//upload any remaining data
if (num_of_uploads_mod > 0)
{
int remainder = file_size - num_of_uploads * max_upload_size;
char buff2[remainder+1] = {};
fread(buff2, sizeof(buff2)-1, 1, fp); //read from file and store to buff2
http.addHeader("File_name", "test file");
int httpResponseCode = http.POST((uint8_t *)buff2, sizeof(buff2)-1); //send buff2 to server
}
http.end(); // Close connection
delay(10 * 1000);
}
Adjustments made for .wav files
int remainder = file_size - num_of_uploads * max_upload_size;
int16_t buff2[remainder+1] = {};
fread(buff2, sizeof(buff2)-1, 1, fp); //remainder
http.addHeader("File_name", "test file");
int httpResponseCode = http.POST((uint8_t *)buff2, sizeof(buff2)-1);
Its working!
There were 2 main issues with the code as outlined by heap underrun. The first issue is that I was reading in the wav file as int16_t the correct datatype to use was uint8_t.
Why are you using an array of int16_t-type elements as a buffer? You are reading a file in binary mode, so be it .wav, .jpg, .ttf, or anything else, it's just a sequence of bytes (uint8_t, not int16_t) anyway. Another thing, fread() expects the size of each object to read as the second parameter and the number of objects to read as the third parameter, so, in case of objects being bytes, first define buffer as uint8_t buff1[max_upload_size] = {}; (no need for +1/-1 games), and then fread(buff1, sizeof *buff1, sizeof buff1 / sizeof *buff1, fp);. The same for buff2. –
heap underrun
The second issue was that I did not include a header in the post stream specifying the content type. As it wasn't needed for the text file and when writing the file in node-red it lets you choose the encoding. I didn't think I would need it, however as it turns out I needed to add:
http.addHeader("Content-Type", "application/octet-stream");
Below is the working code for the file upload section:
if (num_of_uploads > 0)
{
uint8_t buff1[max_upload_size] = {};
for (i = 0; i < num_of_uploads; i++)
{
fread(buff1, sizeof *buff1, sizeof buff1 / sizeof *buff1, fp);
http.addHeader("File_name", "test file"); //header to say what the file name is
http.addHeader("Content-Type", "application/octet-stream");
int httpResponseCode = http.POST(buff1, sizeof(buff1));
}
}
if (num_of_uploads_mod > 0)
{
int remainder = file_size - num_of_uploads * max_upload_size;
uint8_t buff2[remainder] = {};
fread(buff2, sizeof *buff2, sizeof buff2 / sizeof *buff2, fp);
http.addHeader("File_name", "test file");
http.addHeader("Content-Type", "application/octet-stream");
int httpResponseCode = http.POST(buff2, sizeof(buff2));
}
On a slightly interesting side note out of curiosity I tried running the above code but with
uint16_t buff1[max_upload_size] = {};
and
http.POST((uint8_t) buff1, sizeof(buff2));
The file uploaded but the size was 2x what it should be, curiously however the file wasn't corrupted, and played the audio as it was recorded. Just thought that was interesting.
I'll close out this answer as the original question was successfully answered. Again thank you for the help, I've been at this literally weeks and you solved my problems in hours!
I am still learning Cpp, so please advise if I am misunderstanding here.
Using an ESP32, I am trying to read / write files to Flash / FFat. This is the method I have created which should read a file from flash and load it into PSRAM:
unsigned char* storage_read(char* path) {
File file = FFat.open(path);
if(!file) {
Serial.println("no file");
return 0x00;
}
int count = file.size();
unsigned char* buffer = (unsigned char*)ps_malloc(count);
Serial.printf("Bytes: %d\n", count);
Serial.printf("Count: %d\n", sizeof(buffer));
for (int i = 0; i < count; i++) {
buffer[i] = (unsigned char)file.read();
}
file.close();
return buffer;
}
The problem is that I get the contents of my b64 data file, with the addition of several extra bytes of data globbed on the end.
Calling the method with:
Serial.printf("Got: %s", storage_read("/frame/testframe-000.b64"));
I get the output:
Bytes: 684
Count: 4
Got: <myb64string> + <68B of garbage>
Why would sizeof not be returning the proper size?
What would be the proper way of loading this string into a buffer?
Why would sizeof not be returning the proper size?
That's because sizeof() has a very specific function (not very intuitive). It is used - compile time - to query the size of the data type passed to it. Calling sizeof(buffer) returns the size, in bytes, of the type of variable buffer. It's an unsigned char*, so a 4-byte memory address. So that's what you get.
What would be the proper way of loading this string into a buffer?
What I noticed is that you're expecting to load string data from your file, but you don't explicitly terminate it with a zero byte. As you probably know, all C strings must be terminated with a zero byte. Data that you load from the file most likely doesn't have one (unless you took extra care to add it while saving). So when you read a string from a file sized N bytes, allocate a buffer of N+1 bytes, load the file into it and terminate it with a zero. Something like this:
unsigned char* storage_read(char* path) {
File file = FFat.open(path);
if(!file) {
Serial.println("no file");
return 0x00;
}
int count = file.size();
unsigned char* buffer = (unsigned char*)ps_malloc(count + 1); //< Updated
Serial.printf("Bytes: %d\n", count);
Serial.printf("Count: %d\n", sizeof(buffer));
for (int i = 0; i < count; i++) {
buffer[i] = (unsigned char)file.read();
}
buffer[count] = 0; //< Added
file.close();
return buffer;
}
And since you're returning a heap-allocated buffer from your function, take extra care to remember to delete it in caller when finished. This line in your code will leak the memory:
Serial.printf("Got: %s", storage_read("/frame/testframe-000.b64"));
I'm writing a console application using c++ which using sockets and send an HTTP GET request to a server but the response is an html file bigger than 1000000 infact my buffer: char buffer[1000000]; is too small.
I need to receive bigger data from the server than the size of buffer.
I use this code but what is the way to receive a bigger response? I'm a beginner in this programming area so please help me with code and explenations thanks:
char buffer[1000000];
int nDataLength;
while ((nDataLength = recv(Socket, buffer, 1000000, 0)) > 0) {
int i = 0;
while (buffer[i] >= 32 || buffer[i] == '\n' || buffer[i] == '\r') {
myString += buffer[i];
i += 1;
}
}
cout << myString << "\n";
You need to use a smaller fixed length buffer when reading from the socket, and then append the received data to a dynamically growing buffer (like a std::string, or a file) on each loop iteration. recv() tells you how many bytes were actually received, do not access more than that many bytes when accessing the buffer.
char buffer[1024];
std::string myString;
int nDataLength;
while ((nDataLength = recv(Socket, buffer, sizeof(buffer), 0)) > 0) {
myString.append(buffer, nDataLength);
}
std::cout << myString << "\n";
recv return value is total size of receved data.
so you can know total data size, if your buffer is smaller than total data size there is 2 solutions. I guess...
1. allocate buffer on the heap. using like new, allcoc etc.
2. store received data to data structure(like circular queue, queue) while tatal data size is zero(recv function return)
I prefer to use 2nd solution.
Googling about recv function , socket programming sample codes.
That'll helpfull.
I am trying to send out Mat image by TCP. Firstly the Mat has been transferred into uchar and then into char format. The whole image in char format will be send out buffer by buffer whose size is 1024 byte. The following is my code.
Mat decodeImg = imdecode(Mat(bufferFrame), 1);
uchar *transferImg = decodeImg.data;
char* charImg = (char*) transferImg;
int length = strlen(charImg);
int offset = 0;
while (true)
{
bzero(bufferSend, BUFFER_SIZE);
if (offset + BUFFER_SIZE <= length)
{
for (int i = 0; i < BUFFER_SIZE; i++)
{
bufferSend[i] = charImg[i + offset];
}
// memcpy(charImg+offset, bufferSend,BUFFER_SIZE);
if (send(sockfd, bufferSend, sizeof(bufferSend), 0) < 0)
{
printf("Send FIle Failed,total length is%d,failed offset is%d\n",
length,
offset);
break;
}
}
else
{
for (int i = 0; i < length - offset; i++)
{
bufferSend[i] = charImg[i + offset];
}
if (send(sockfd, bufferSend, sizeof(bufferSend), 0) < 0)
{
printf("Send FIle Failed,total length is%d,failed offset is%d\n",
length,
offset);
break;
}
break;
}
offset += BUFFER_SIZE;
}
The output of the code shows : send file failed, total length is 251035, failed offset is 182272.
I am really appreciated on your help. Thank you in advance!
Pulling out the crystal ball here. This might be OP's problem, but if it isn't, this is certainly a problem that needs to be addressed.
Mat decodeImg = imdecode(Mat(bufferFrame), 1);
uchar *transferImg = decodeImg.data;
Get data. Not a bad idea if that's what you need to send.
char* charImg = (char*) transferImg;
Take the array of bytes from above and treat it as an array of characters.
int length = strlen(charImg);
And Boom. Matrix data is not ascii formated data, a string, so it should not be treated like a string.
strlen counts data until it reaches a null character, a character with the numerical value 0, which does not exist in the normal alpha numeric world and thus can be used as a canary value to signal the end of a string. The count is the number of characters before the first null character in the string.
In this case we don't have a string. We have a blob of binary numbers, any one of which could bee 0. There could be a null value anywhere. Could be right at the beginning. Could be a hundred bytes in. There might not be a null value in the until long after all of the valid image data has been read.
Anyway, strlen will almost certainly return the wrong value. Too few bytes and the receiver doesn't get all of the image data and I have no idea what it does. That code's not available to us. It probably gets upset and discards the result. Maybe it crashes. There's no way to know. If there is too much information, we also don't know what happens. Maybe it processes the file happily and ignores the extra crap that's sent. Maybe it crashes.
But what if it closes the TCP/IP connection when it has enough bytes? That leaves the sender trying to write a handful of unsent and unwanted bytes into a closed socket. send will fail and set the error code to socket closed.
Solution:
Get the right size of the data.
What I'm reading from the openCV documentation is inside a Mat is Mat::elemSize which will give you the size of each item in the matrix and Mat::size which returns a Size object containing the rows and columns. Multiply rows * columns * elemSize and you should have the number of bytes to send.
EDIT
This looks to be a better way to get the size.
I am trying to send wav files in C++ via TCP on Linux over a socket but I don't understand how a wav file can be read correctly.
My goal is to read the file on the client into a char array, send it with "write()" to the server, and the server should write the data into a local wav file again.
I read the .wav file like this:
////////////define socket - left out for simplicity
ifstream file ("audio.wav", ios::in|ios::binary|ios::ate); //open .wav file
char * buffer; //declare data buffer, should contain .wav data to write to socket
streampos filesize; //size of file
int n; //number of written bytes
//if file opened correctly, read content and write to socket
if (file.is_open()){
filesize = file.tellg();
buffer = new char [filesize];
file.seekg (0, ios::beg);
file.read (buffer, filesize);
file.close();
n = write(socket, buffer, sizeof(buffer));
}
On the server, this returns the array "RIFF" of length "4", so its part of the header of the wav file.
How can I read the whole .wav file content correctly for writing to the TCP socket?
Thanks.
That is simple: filesize is size of the file in bytes. However, sizeof(buffer) is only 4 on a 32-bit OS. Modify your code like this:
if(file.is_open()) {
filesize = file.tellg();
buffer = new char [filesize];
file.seekg (0, ios::beg);
file.read (buffer, filesize);
file.close();
n = write_all(socket, buffer, filesize); // use filesize here
delete[] buffer; // !!
}
To simplify processing on the other side, you may want to send filesize first to avoid parsing of the RIFF header to know how many bytes to accept. I would also suggest allocating a smaller buffer and reading several times to send the larger files over:
if(file.is_open()) {
filesize = file.tellg();
file.seekg(0, ios::beg);
uint32_t remains = filesize;
write(socket, &remains, sizeof(uint32_t));
// write 4B with size of the file (optional)
buffer = new char[(filesize > 4096)? 4096 : filesize];
// only up to 4k buffer to avoid running out of memory
n = 0;
while(remains > 0) {
int chunk = (remains > 4096)? 4096 : remains;
// decide how much to read in at one time (not more than size of the buffer)
file.read(buffer, chunk);
n += write_all(socket, buffer, chunk);
// read a chunk and write it to the socket
remains -= chunk;
// update number of bytes that remains to be transferred
}
// send the file several times
file.close();
delete[] buffer; // !!
}
You may notice the use of a helper function write_all. That is required, because the socket might get full and then write will not write all the data given to it. It could look like this:
size_t write_all(int socket, const char *buffer, size_t size)
{
size_t n = 0;
while(size > 0) {
size_t written = write(socket, buffer, size);
if(written == -1)
return written; // handle errors
n += written;
size -= written;
}
return n;
}
buffer is of type char*, so sizeof(buffer) is the size of an ordinary data pointer on your platform. So you're writing 4 or 8 bytes (assuming ordinary plateform) to your socket.
You need to put filesize in the write call instead of sizeof(buffer).
(sizeof is a compile-time construct. It's evaluated when your code is compiled. It can't return a size that's determined at runtime.)
In the line n = write(socket, buffer, sizeof(buffer)); the value of the last buffer becomes the size of a pointer on your platform. Since only four bytes are sent, I will assume that you are either on a 32-bit platform or compiling the application in 32-bit mode.
Replacing sizeof(buffer) will only be a partial solution since write() may not write all the data at once. You will need to check the value returned from write() and keep record of how many bytes have been written to the socket.
I used sndfile for handling wav files.